Solved snmpd Stops Responding

Does anyone in here run snmpd(8) from net-snmp on their FreeBSD servers? I ask, because snmpd(8) keeps getting to a point where it stops responding and has to be brutal killed and restarted. I get alarms in ZenOSS that snmpd(8) cannot be reached when this happens. This is not happening with snmpd(8) on my Linux servers, just FreeBSD unfortunately. It is also random. There is not a specific length of time when this happens.

# uname -r
11.0-RELEASE-p9


# /usr/local/sbin/snmpd -v

NET-SNMP version: 5.7.3
Web: http://www.net-snmp.org/
Email: net-snmp-coders@lists.sourceforge.net


For the record this is happening on 10.3-RELEASE installs as well. If anyone here knows a way to keep this daemon responding to snmp queries under FreeBSD instead of falling over randomly, hints and tips for that would be greatly appreciated.
 
I use Observium to monitor about 50 devices 6 of which are FreeBSD file servers and Jail hosts. Observium uses Net-SNMP (via crom) to poll the devices and recover info to the RRD database files. I have never had any issues with SNMP clients but the general rule of the thumb is that on BSD machine one should use native daemons (bsnmpd on FreeBSD and snmpd on OpenBSD). OpenBSD guys are very adamant about it. FreeBSD not so much. FreeNAS guys wanted to move to Net-SNMP the last time I was following their development closely (around 9.3 release) even that on 9.2.xxx bsnmpd worked just fine. FreeNAS last release almost brought the house down (people are massively going back to 9.3 and 9.10) so I would take their advise with a pinch of salt. I, but more importantly many other more knowledgeable people had look at the Net-SNMP code and it is horrible. I would like to see FreeBSD guys fixing and improving their own version of bsnmpd.
 
Ah, okay. I will test out bsnmpd(1) on one of my FreeBSD servers. Thank you Oko.

A quick look through the handbook topics does not give me a clue about where to find information on setting up bsnmpd(1). Any direction for where to find detailed documentation for bsnmpd(1) would be appreciated. In the meantime, I will be searching the WWW for information.
 
A handbook is not regularly updated and it is trying to cover too much and too many releases. Thanks God they have up to date Desktop section :) Anyhow.
You have an example of snmpd.conf file in /etc directory. If you installed Net-SNMP than those guys do have a man pages for snmpd.conf file. The example of their file is in /usr/local/etc/ and reading through it could be useful. Yes documentation is lacking including man pages and bsnmpd is not completed.
 
I see. So I am going to presume it will use my existing 2c version /etc/snmpd.conf from net-snmp's install. Is that a fair assumption?

Thanks again. :)
 
Hrm, no, that was not a fair assumption. It appears to not be loading my /etc/snmpd.conf file and when I copy that to /etc/snmpd.config then bsnmpd(1) fails to start. How is the format different from what net-snmp's snmpd(8) uses? I need to see an example working config file without all the comments.

From /var/log/messages:
Code:
Jun  7 10:14:41 ossnfssrv-0 snmpd[85597]: '=' expected, got 'M-^C'
Jun  7 10:14:41 ossnfssrv-0 snmpd[85597]:   in file /etc/snmpd.config line 1
Jun  7 10:14:41 ossnfssrv-0 snmpd[85597]: error in config file
 
Last edited by a moderator:
Here is the one I use on my file servers


Code:
host := backup1.int.autonlab.org
contact := "(redacted)"
location := "Rack A1-2B, Wean Hall 3611 [40.4424925,-79.9425528]"
system := 1 # FreeBSD 10.1
description := "Hardware: Intel(R) Xeon(R) CPU E5620  @ 2.40GHz (2400.13-MHz K8-class CPU) Software: FreeBSD 10.3-RELEASE"
read := "public"

# Configuration
%snmpd
begemotSnmpdDebugDumpPdus = 2
begemotSnmpdDebugSyslogPri = 7

# Set the read and write communities.
begemotSnmpdCommunityString.0.1 = "public"
begemotSnmpdCommunityDisable = 1

# open standard SNMP ports
begemotSnmpdPortStatus.0.0.0.0.161 = 1

# open a unix domain socket
begemotSnmpdLocalPortStatus."/var/run/snmpd.sock" = 1
begemotSnmpdLocalPortType."/var/run/snmpd.sock" = 4

sysContact = $(contact)
sysLocation = $(location)
sysObjectId = 1.3.6.1.4.1.12325.1.1.2.1.$(system)
sysDescr = $(description)

begemotSnmpdModulePath."mibII" = "/usr/lib/snmp_mibII.so"
begemotSnmpdModulePath."hostres" = "/usr/lib/snmp_hostres.so"
 
Last edited:
This got me further. However, as this is a NFS server with shares on ZFS I need to monitor disk space on the ZFS shares in our company ZenOSS installation. I do not see any of my disk space being reported. Not even the UFS boot partition. Looking through the documentation I can find is not giving me any clues so far. Where do I start troubleshooting this to get my disk space reports?

Also, for some reason I am getting an alarm on sendmail(1). We are running postfix(1) to get our "root mail spam" with sendmail(1) disabled. Why is this looking for and reporting sendmail(1)? Where do I set it to monitor postfix(1) instead?

Thanks again.

(Added) Weird, for some reason the net-snmp snmpd(8) allows ZenOSS to monitor ZFS volumes. But when polling the server with bsnmpd(1) running I am seeing a bunch of lines similar to this in the output:

INFO zen.ZenModeler: Skipping /usr2, type: zfs, dev: usr2 (other) as it matches zFileSystemMapIgnoreTypes.

Looks like that may be a ZenOSS problem instead of a bsnmpd(1) problem. I'll explore that.
 
This appears to be the problem where bsnmpd(1) is reporting the storage types:

Code:
HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrFSBerkeleyFFS
HOST-RESOURCES-MIB::hrStorageType.4 = OID: HOST-RESOURCES-TYPES::hrFSOther
HOST-RESOURCES-MIB::hrStorageType.5 = OID: HOST-RESOURCES-TYPES::hrFSUnknown
HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-TYPES::hrFSOther
HOST-RESOURCES-MIB::hrStorageType.7 = OID: HOST-RESOURCES-TYPES::hrFSOther
HOST-RESOURCES-MIB::hrStorageType.8 = OID: HOST-RESOURCES-TYPES::hrFSOther
HOST-RESOURCES-MIB::hrStorageType.9 = OID: HOST-RESOURCES-TYPES::hrFSOther

The hrStorageType values on these are not recognized and/or are ignored in ZenOSS. They need to be HOST-RESOURCES-TYPES::hrStorageFixedDisk. What do I put in /etc/snmpd.config to get bsnmpd(1) to change those values to hrStorageFixedDisk?
 
Last edited by a moderator:
There is certainly a dearth of documentation for bsnmpd(1). Very discouraging so far. I have blown most of my day just trying to get the monitoring and reporting I need on one server. It is not looking good for further deployment at this point. :(
 
I am waiting 11.1 release to start moving my 10.3 machines to that branch. I have just checked on one of my servers and appears that MIBs files are stored in two different locations

Code:
root@backup1:~ # find /usr -name "*MIB*"
/usr/share/snmp/mibs/FOKUS-MIB.txt
/usr/share/snmp/mibs/FREEBSD-MIB.txt
/usr/share/snmp/mibs/BEGEMOT-IP-MIB.txt
/usr/share/snmp/mibs/BEGEMOT-ATM-FREEBSD-MIB.txt
/usr/share/snmp/mibs/BEGEMOT-BRIDGE-MIB.txt
/usr/share/snmp/mibs/BEGEMOT-HOSTRES-MIB.txt
/usr/share/snmp/mibs/BEGEMOT-MIB2-MIB.txt
/usr/share/snmp/mibs/BRIDGE-MIB.txt
/usr/share/snmp/mibs/BEGEMOT-WIRELESS-MIB.txt
/usr/share/snmp/mibs/BEGEMOT-HAST-MIB.txt
/usr/share/snmp/mibs/BEGEMOT-PF-MIB.txt
/usr/share/snmp/mibs/BEGEMOT-MIB.txt
/usr/share/snmp/mibs/RSTP-MIB.txt


but I also see lots of MIBs files in /usr/local/share

Code:
/usr/local/share/snmp/mibs/SNMP-USM-AES-MIB.txt
/usr/local/share/snmp/mibs/NET-SNMP-MIB.txt
/usr/local/share/snmp/mibs/UCD-SNMP-MIB.txt
/usr/local/share/snmp/mibs/DISMAN-SCRIPT-MIB.txt
/usr/local/share/snmp/mibs/NET-SNMP-AGENT-MIB.txt
/usr/local/share/snmp/mibs/SNMP-FRAMEWORK-MIB.txt
/usr/local/share/snmp/mibs/IF-INVERTED-STACK-MIB.txt
/usr/local/share/snmp/mibs/INET-ADDRESS-MIB.txt
/usr/local/share/snmp/mibs/NET-SNMP-VACM-MIB.txt
/usr/local/share/snmp/mibs/UCD-DEMO-MIB.txt
/usr/local/share/snmp/mibs/IPV6-FLOW-LABEL-MIB.txt
/usr/local/share/snmp/mibs/IP-MIB.txt
/usr/local/share/snmp/mibs/DISMAN-EVENT-MIB.txt
/usr/local/share/snmp/mibs/UCD-DISKIO-MIB.txt
/usr/local/share/snmp/mibs/BRIDGE-MIB.txt
/usr/local/share/snmp/mibs/SNMP-NOTIFICATION-MIB.txt
/usr/local/share/snmp/mibs/SMUX-MIB.txt
/usr/local/share/snmp/mibs/UDP-MIB.txt
/usr/local/share/snmp/mibs/IANA-LANGUAGE-MIB.txt
/usr/local/share/snmp/mibs/HOST-RESOURCES-MIB.txt
/usr/local/share/snmp/mibs/SCTP-MIB.txt
/usr/local/share/snmp/mibs/DISMAN-SCHEDULE-MIB.txt
/usr/local/share/snmp/mibs/NET-SNMP-PASS-MIB.txt
/usr/local/share/snmp/mibs/SNMP-TLS-TM-MIB.txt
/usr/local/share/snmp/mibs/SNMP-COMMUNITY-MIB.txt
/usr/local/share/snmp/mibs/NET-SNMP-EXTEND-MIB.txt
/usr/local/share/snmp/mibs/NET-SNMP-EXAMPLES-MIB.txt
/usr/local/share/snmp/mibs/SNMP-VIEW-BASED-ACM-MIB.txt
/usr/local/share/snmp/mibs/TCP-MIB.txt
/usr/local/share/snmp/mibs/EtherLike-MIB.txt
/usr/local/share/snmp/mibs/SNMP-TSM-MIB.txt
/usr/local/share/snmp/mibs/IPV6-ICMP-MIB.txt
/usr/local/share/snmp/mibs/SNMP-USM-DH-OBJECTS-MIB.txt
/usr/local/share/snmp/mibs/IP-FORWARD-MIB.txt
/usr/local/share/snmp/mibs/SNMP-USER-BASED-SM-MIB.txt
/usr/local/share/snmp/mibs/RFC1213-MIB.txt
/usr/local/share/snmp/mibs/IPV6-MIB.txt
/usr/local/share/snmp/mibs/MTA-MIB.txt
/usr/local/share/snmp/mibs/TUNNEL-MIB.txt
/usr/local/share/snmp/mibs/NETWORK-SERVICES-MIB.txt
/usr/local/share/snmp/mibs/SNMPv2-MIB.txt
/usr/local/share/snmp/mibs/RMON-MIB.txt
/usr/local/share/snmp/mibs/AGENTX-MIB.txt
/usr/local/share/snmp/mibs/IANA-ADDRESS-FAMILY-NUMBERS-MIB.txt
/usr/local/share/snmp/mibs/IPV6-TCP-MIB.txt
/usr/local/share/snmp/mibs/SNMP-PROXY-MIB.txt
/usr/local/share/snmp/mibs/SNMP-MPD-MIB.txt
/usr/local/share/snmp/mibs/TRANSPORT-ADDRESS-MIB.txt
/usr/local/share/snmp/mibs/IPV6-UDP-MIB.txt
/usr/local/share/snmp/mibs/UCD-DLMOD-MIB.txt
/usr/local/share/snmp/mibs/SNMP-TARGET-MIB.txt
/usr/local/share/snmp/mibs/IANAifType-MIB.txt
/usr/local/share/snmp/mibs/IF-MIB.txt
/usr/local/share/snmp/mibs/IANA-RTPROTO-MIB.txt
/usr/local/share/snmp/mibs/UCD-IPFWACC-MIB.txt
/usr/local/share/snmp/mibs/NOTIFICATION-LOG-MIB.txt


and even at the third location

Code:
/usr/local/share/smi/mibs/ietf/SLAPM-MIB
/usr/local/share/smi/mibs/ietf/T11-FC-VIRTUAL-FABRIC-MIB
/usr/local/share/smi/mibs/ietf/SNMP-TARGET-MIB
/usr/local/share/smi/mibs/ietf/SNMP-MPD-MIB
/usr/local/share/smi/mibs/ietf/UDP-MIB
/usr/local/share/smi/mibs/ietf/DOT12-RPTR-MIB
/usr/local/share/smi/mibs/ietf/HPR-IP-MIB
/usr/local/share/smi/mibs/ietf/DOT3-EPON-MIB
/usr/local/share/smi/mibs/ietf/PKTC-IETF-MTA-MIB
/usr/local/share/smi/mibs/ietf/L2TP-MIB
/usr/local/share/smi/mibs/ietf/SFLOW-MIB
/usr/local/share/smi/mibs/ietf/PerfHist-TC-MIB
/usr/local/share/smi/mibs/ietf/IPS-AUTH-MIB
/usr/local/share/smi/mibs/ietf/IPV6-TCP-MIB
/usr/local/share/smi/mibs/ietf/SONET-MIB
/usr/local/share/smi/mibs/ietf/APM-MIB
/usr/local/share/smi/mibs/ietf/RSTP-MIB
/usr/local/share/smi/mibs/ietf/DOCS-IETF-BPI2-MIB
/usr/local/share/smi/mibs/iana/IANA-RTPROTO-MIB
/usr/local/share/smi/mibs/iana/IANA-FINISHER-MIB
/usr/local/share/smi/mibs/iana/IANA-IPPM-METRICS-REGISTRY-MIB
/usr/local/share/smi/mibs/iana/IANA-MAU-MIB
/usr/local/share/smi/mibs/iana/IANA-PRINTER-MIB
/usr/local/share/smi/mibs/iana/IANAifType-MIB
/usr/local/share/smi/mibs/iana/IANA-GMPLS-TC-MIB
/usr/local/share/smi/mibs/iana/IANA-MALLOC-MIB
/usr/local/share/smi/mibs/iana/IANA-CHARSET-MIB
/usr/local/share/smi/mibs/iana/IANATn3270eTC-MIB
/usr/local/share/smi/mibs/iana/IANA-LANGUAGE-MIB
/usr/local/share/smi/mibs/iana/IANA-ITU-ALARM-TC-MIB
/usr/local/share/smi/mibs/iana/IANA-ADDRESS-FAMILY-NUMBERS-MIB
/usr/local/share/smi/mibs/tubs/TUBS-IBR-PING-MIB
/usr/local/share/smi/mibs/tubs/TUBS-IBR-TEST-MIB
/usr/local/share/smi/mibs/tubs/TUBS-IBR-TNM-MIB
/usr/local/share/smi/mibs/tubs/TUBS-IBR-XEN-MIB
/usr/local/share/smi/mibs/tubs/POLICY-DEVICE-AUX-MIB
/usr/local/share/smi/mibs/tubs/POLICY-DEVICE-AUX-MIB-orig
/usr/local/share/smi/mibs/tubs/TUBS-IBR-LINUX-MIB
/usr/local/share/smi/mibs/tubs/TUBS-IBR-LINUX-NETFILTER-MIB
/usr/local/share/smi/mibs/tubs/TUBS-IBR-NFS-MIB
/usr/local/share/smi/mibs/tubs/TUBS-IBR-PROC-MIB

IIRC Net-SNMP uses its own MIBs files. I am almost 99% sure that /usr/share MIBs file are the one used by BSNMPd while the one is /usr/loca/share are the one installed by Net-SNMP. I forgot why I installed Net-SNMP on this server. Anyhow what I am trying to say is that Zenoos is probably not able to display statistics from native MIBs files and is tuned to work with Net-SNMP. You have to get to the bottom of this issue. One day is a joke. You should not be giving up after one day.
 
Also, for some reason I am getting an alarm on sendmail(1). We are running postfix(1) to get our "root mail spam" with sendmail(1) disabled. Why is this looking for and reporting sendmail(1)? Where do I set it to monitor postfix(1) instead?

Sendmail used to be the standard UNIX mail servers and most mail servers and applications have interface which emulates/expects Sendmail. Personally on FreeBSD I always disable Sendmail and use DragonFly tiny mail server
mail/dma to get alerts (S.M.A.R.T. and sysutils/monit) Again, you really have to be pesistent and try to understand what actually happens instead of assuming that things work this or that way.
 
... You should not be giving up after one day.
Heh, I understand that. ;) However, I was doing this on a server that I need to get into production on a time limit. I will have to spin up a new VM just for playing with this to learn it. I will revisit this thread when I get that VM up and running.
 
I see. So I am going to presume it will use my existing 2c version /etc/snmpd.conf from net-snmp's install. Is that a fair assumption?

Thanks again. :)

No, you can't share config files between bsnmpd and snmpd, they are very different.

You'll also need to install a port in order to make SNMP usable with bsnmpd: sysutils/bsnmp-ucd

Then enable a few modules in the default /etc/snmpd.config:
Code:
# Load MIB-2 module
begemotSnmpdModulePath."mibII"    = "/usr/lib/snmp_mibII.so"

# Host resources module
#  This requires the mibII module.
begemotSnmpdModulePath."hostres" = "/usr/lib/snmp_hostres.so"

# UCD-SNMP MIB
begemotSnmpdModulePath."ucd" = "/usr/local/lib/snmp_ucd.so"

With those three modules enabled, restart bsnmpd, and your monitoring software will be able to access the things it needs.
 
...
Then enable a few modules in the default /etc/snmpd.config:
Code:
# Load MIB-2 module
begemotSnmpdModulePath."mibII"    = "/usr/lib/snmp_mibII.so"

# Host resources module
#  This requires the mibII module.
begemotSnmpdModulePath."hostres" = "/usr/lib/snmp_hostres.so"

# UCD-SNMP MIB
begemotSnmpdModulePath."ucd" = "/usr/local/lib/snmp_ucd.so"
With those three modules enabled, restart bsnmpd, and your monitoring software will be able to access the things it needs.

Timely advice, as I just got my test VM spun up. Thanks! :)
 
Yeah, I made sure those were available on the system and placed the lines in the config before starting bsnmpd(1). It is still not showing what I need.

I have uploaded a couple of screen captures from ZenOSS. One of a FreeBSD server in production that is using net-snmp and one of the test VM running bsnmpd(1). I need everything that is shown in the server running net-snmp to be shown with bsnmpd(1) as well. Specifically the section under "Components".
 

Attachments

  • zenoss_with_snmpd_2017-06-08_12-43-42.png
    zenoss_with_snmpd_2017-06-08_12-43-42.png
    22.6 KB · Views: 530
  • zenoss_with_bsnmpd_2017-06-08_12-45-15.png
    zenoss_with_bsnmpd_2017-06-08_12-45-15.png
    20.7 KB · Views: 487
Here is a copy of my current configuration file on this test VM:

Code:
host := "ossomzbsnmpdtestvm-1"
location := "VM at AOC"
contact := "(redacted)"
system := 1 # FreeBSD
description := "VMware Virtual Machine running FreeBSD 11.0-RELEASE"
read := "(redacted)"

# Configuration
%snmpd
begemotSnmpdDebugDumpPdus = 2
begemotSnmpdDebugSyslogPri = 7

# Set the read and write communities.
begemotSnmpdCommunityString.0.1 = "(redacted)"
begemotSnmpdCommunityDisable = 1

# open standard SNMP ports
begemotSnmpdPortStatus.0.0.0.0.161 = 1

# open a unix domain socket
begemotSnmpdLocalPortStatus."/var/run/snmpd.sock" = 1
begemotSnmpdLocalPortType."/var/run/snmpd.sock" = 4

sysContact = $(contact)
sysLocation = $(location)
sysDescr = $(description)

# Load MIB-2 module
begemotSnmpdModulePath."mibII" = "/usr/lib/snmp_mibII.so"

# Host resources module
#  This requires the mibII module.
begemotSnmpdModulePath."hostres" = "/usr/lib/snmp_hostres.so"

# UCD-SNMP MIB
begemotSnmpdModulePath."ucd" = "/usr/local/lib/snmp_ucd.so"

Is there a way to customize how bsnmpd(1) reports a MIB type? Such as what I showed yesterday with the storage?
 
Last edited by a moderator:
You'll have to adjust your ZenOSS configuration to look in different spots.
This appears to be the problem where bsnmpd(1) is reporting the storage types:

HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrFSBerkeleyFFS
HOST-RESOURCES-MIB::hrStorageType.4 = OID: HOST-RESOURCES-TYPES::hrFSOther
HOST-RESOURCES-MIB::hrStorageType.5 = OID: HOST-RESOURCES-TYPES::hrFSUnknown
HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-TYPES::hrFSOther
HOST-RESOURCES-MIB::hrStorageType.7 = OID: HOST-RESOURCES-TYPES::hrFSOther
HOST-RESOURCES-MIB::hrStorageType.8 = OID: HOST-RESOURCES-TYPES::hrFSOther
HOST-RESOURCES-MIB::hrStorageType.9 = OID: HOST-RESOURCES-TYPES::hrFSOther


The hrStorageType values on these are not recognized and/or are ignored in ZenOSS. They need to be HOST-RESOURCES-TYPES::hrStorageFixedDisk. What do I put in /etc/snmpd.config to get bsnmpd(1) to change those values to hrStorageFixedDisk?

You don't. You update your monitoring software to use the correct MIBs that are returned from the device being queried. :)
 
So, you are saying a File System type of "Other" is a correct MIB? Frankly, I am likely not going to be able to get that allowed as a valid file system type on our monitoring system as I am not in control of the ZenOSS install. Meaning I am not the administrator who would make those changes. I can only add or remove my servers.

I would have to have a much more valid reason than "I want to use bsnmpd(1) instead of net-snmp.", since net-snmp does work, but hangs periodically which can be worked around. I suspect that file system type of "Other" is ignored for a reason.

I will keep digging at this, but I am losing hope that I can make this work in our environment if there is no way to customize MIBs. :(
 
Yep, I was afraid I was not going to get this into our ZenOSS here with bsnmpd(1) since it required getting the ZenOSS admin to customize just for bsnmpd(1). Not going to happen. So, I wrote a script to check for responses from net-snmp's snmpd(8) and kill/restart it when it is not responding.

Code:
#!/bin/sh
#
# Script to test for a response from snmpd to see if snmpd is
# "navel gazing" and not responding. Kill and restart snmpd if
# no response from the server within about 15 seconds.
#
# Written for FreeBSD with net-snmp.
# Adjust paths for other Unix or Linux.
#
# By Gene Alexander <ealexander@gci.com>. This script is
# free to use at your discretion. No warranty is implied.

myhostname=$(/bin/hostname)
mylogger="/usr/bin/logger"
mypkill="/bin/pkill"
myservice="/usr/sbin/service"
mycheck=$(/usr/local/bin/snmpget -t 5 -r 3 -c REPLACEME -v 2c localhost sysUpTimeInstance)
myresult=$?

case $myresult in
        0)
                exit
        ;;
        1)
                printf "Host: $myhostname\n"
                printf "No response from snmpd on $(date).\n"
                printf "Killing and restarting snmpd.\n"
                $mylogger "Killing and restarting snmpd via cron monitor script."
                printf "\n"
                $mypkill -9 snmpd
                $myservice snmpd start
        ;;
        *)
                printf "Host: $myhostname\n"
                printf "Error code from snmpget: $myresult\n"
                printf "Please check the server.\n"
        ;;
esac
Unfortunately a brutal kill is the only thing that works. :(

I call this script "checksnmp" and have it in /root/bin/ with perms 0750 on my servers. It runs from cron with this line:
Code:
0,15,30,45 * * * * /root/bin/checksnmp
 
Last edited:
Back
Top