Handling SNMP Traps with Hobbit

   [1]Hobbit   is   a   fast   replacement   for   the   [2]Big   Brother
   monitoring/alert  system.  There  were  some ways to handle SNMP traps
   with  Big Brother, but none that I could find that worked with Hobbit.
   (I hope I haven't reinvented the wheel....) I'm running all of this on
   the  same  computer.  Most  all  of the worries I had about handling a
   large number of traps went away when a misconfigured device flooded my
   SNMP  trap  server  at  a  rate of 30 messages per minute over several
   days.  I  discovered  it  when  the  log  files were getting unusually
   large...

   I got the idea after reading an article on integrating SNMP traps with
   Nagios in [3]Sys Admin. I just adapted it to Hobbit.

   What it does not do:
    1. Does  not:  Forward  SNMP  traps to an external Network Management
       System
    2. Does not: Poll devices using SNMP

   Here's what I used:
    1. [4]Net-SNMP - SNMP tools (snmptrapd,...)
    2. [5]SNMPTT - SNMP trap translator
    3. [6]SEC - Simple Event Correlator
    4. Various MIB files

  How it works

     * Snmptrapd is configured to feed SNMPTT the OID and hostname of the
       sending SNMP agent (I haven't tried working with SNMP proxies...).
     * SNMPTT  then  translates  the OID into a text message based on the
       MIB   description   for   that   OID   and  logs  the  message  in
       /var/log/messages  (or  where  ever  you  put  it).  You  can also
       configure  SNMPTT  to  log  to  a  MySQL  database simultaneously.
       There's a PHP script listed below that you can use to query it.
     * SEC  monitors  the /var/log/messages file for entries from SNMPTT.
       Since  some  equipment  can  send  the same trap multiple times in
       quick  sucession,  SEC  is configured to ignore duplicate messages
       for a second or two.
     * SEC  then  launches  a  wrapper script that sends Hobbit a message
       using Hobbit's BB client program.
     * A  script  is  run  by  Hobbit every 5 minutes to prevent any trap
       message  columns  from  turning  purple.  (I  don't want my screen
       turning  purple  if I don't get a trap inside of 30min or whatever
       the no response timeout period is for Hobbit.)
     _________________________________________________________________

   I  won't go into detail how to configure and install each package; so,
   definitely  read  the documentation for each one. You'll probably find
   other uses for them. Be sure to adjust the path names as needed.

  Net SNMP

   Just  compile and install this package. You might have it depending on
   your distribution.

   RC script: [7]rc.snmptrapd

   Logrotate configuration:
/var/log/snmp/snmptrapd.log {
    missingok
    postrotate
        /etc/rc.d/rc.snmptrapd restart
    endscript
}

   /etc/snmp/snmptrapd.conf :
traphandle default /usr/local/sbin/snmptthandler

  Get your MIB files

   This will depend on what you have, but here's some:
     * American Power Conversion -
       [8]ftp://ftp.apc.com/apc/public/software/pnetmib/mib/
     * Cisco - [9]ftp://ftp.cisco.com/pub/mibs/v1/
     * MIB Librarys:
          + [10]at ByteSphere They have MIBS for most vendors
          + [11]SNMP4tPC They have MIBS for most vendors
     * Your  device may have a MIB file on it too. It's usually buried in
       the web management interface somewhere.

  SNMPTT

   After  installing  the  package,  you'll  need to convert MIB files to
   snmptt.conf  files. The SNMPTT manual mentions using the snmptranslate
   command  from the Net-SNMP package to verify that a given MIB file can
   be parsed correctly.

    Examples:

   The  --net_snmp_perl  option  creates  long  description sections with
   detail  about  all  of  the  possibile variables for a given OID trap.
   Using it is optional.

   American Power Conversion:
$ export MIBDIRS=/usr/local/share/snmp/mibs
$ /usr/local/sbin/snmpttconvertmib --format_desc=6 --net_snmp_perl --in=powerne
t368.mib --out=snmptt.conf.apc 2>>errors

   Cisco:
$ mkdir v1
$ tar -C v1 -zxvf v1.tar.gz
$ export MIBDIRS=/usr/local/share/snmp/mibs:$PWD/v1
$ for i in v1/*my
> do
> /usr/local/sbin/snmpttconvertmib --format_desc=6 --net_snmp_perl --in=$i --ou
t=snmptt.conf.cisco 2>>errors
> done

   Don't be supprised to get warning messages about "can't translate/find
   OID" in the "errors" file. It's telling you that some of the MIB files
   require  other  MIB  files  that  you  don't  have. You can ignore the
   warnings,  track down all of the required MIBs (by reading the problem
   MIB file), or only translate the MIBs that pertain to your equipment.

   I've  noticed  that  the  APC  and  Dell  MIB  files  have  a SEVERITY
   definition  in  them.  SNMPTT  uses that to establish the severity for
   each  trap  (Normal|INFORMATIONAL|SEVERE|WARNING|...).  However,  I've
   noticed  that  Cisco  and Canoga Perkins don't have those definitions;
   so,  every  trap event is considered Normal. You'll need to change the
   severity for the various traps as desired in the snmptt.conf file.

   Copy your various snmptt.conf.* files to /etc/snmp.

   RC script: [12]rc.snmptt

   Logrotate configuration:
/var/log/snmp/snmpttunknown.log {
    missingok
    postrotate
        /etc/rc.d/rc.snmptt restart
    endscript
}
/var/log/snmp/snmptt.log {
    missingok
    postrotate
        /etc/rc.d/rc.snmptt restart
    endscript
}


   The  default  settings  for the /etc/snmp/snmptt.ini file are fine. If
   you  run snmptt as non-root, make sure it has permission to read/write
   log    files    where    indicated.    I've   listed   what   I   use:
   [13]/etc/snmp/snmptt.ini.   I   used   the   [14]SNMPTT   MySQL  setup
   instructions to create the MySQL databse.

  SEC - Simple Event Correlator

   RC script: [15]rc.sec

   Logrotate configuration:
/var/log/sec.log {
    missingok
    postrotate
        /etc/rc.d/rc.sec restart
    endscript
}

   /etc/sec.conf :
# sec.conf file
#
# notes:
#  http://simple-evcorr.sourceforge.net/
#  http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
#
# trap.sh time_stamp OID severity hostname message

# Snmptrap event translated by snmptraptt and logged in /var/log/messages
# $1 - Time stamp
# $2 - OID
# $3 - Severity Status
# $4 - Hostname (or IP)
# $5 - Message
type=SingleWithSuppress
ptype=RegExp
pattern=(\w+\s+\d+\s\d+:\d+:\d+)\s\w+ snmptt\[\d+\]: ([\d\.]*) (Normal|INFORMAT
IONAL|MINOR|WARNING|SEVERE|MAJOR|CRITICAL)\s\"Status Events\"\s([\w\.\-]*)\s\-\
s(.*)
desc=snmptrap received from $4
action=shellcmd /home/hobbit/server/ext/trap.sh "$1" $2 $3 $4 "$5"
window=2


# Unknown Snmptrap event logged in /var/log/snmp/snmpttunknown.log
# $1 - Time stamp
# $2 - OID
# $3 - Hostname (or IP)
type=SingleWithSuppress
ptype=RegExp
pattern=(\w+\s+\w+\s+\d+\s\d+:\d+:\d+\s+\d+):\sUnknown trap \(([\d\.]*)\) recei
ved from ([\w\.\-]*) at:
desc=unknown snmptrap received from $3
action=shellcmd /home/hobbit/server/ext/trap.sh "$1" $2 Unknown $3 "Unknown tra
p"
window=2

Trap script

   I  made  the  [16]trap.pl  script so it can be used by both SEC (which
   processes  the trap) as well as Hobbit (which runs it to keep the trap
   column from going purple).

   You'll  need  to  rename  the file to trap.pl since the file is called
   trap.txt so your web browser can view it.

   Copy  trap.pl  to /home/hobbit/server/ext and make it executable. Make
   sure the path to perl is correct in trap.pl.

   You need to install Text::Autoformat from CPAN for trap.pl.

  SEC Client

   The  wrapper  script,  trap.sh,  is  called by SEC when it processes a
   trap.  It uses Hobbit's bbcmd in order to setup the proper environment
   variables.

   trap.sh:
#!/bin/sh
# $1 - Time stamp
# $2 - OID
# $3 - Severity Status
# $4 - Hostname (or IP)
# $5 - Message

TRAPCLIENT="/home/hobbit/server/ext/trap.pl"
HOBBITCFG="/home/hobbit/server/etc/hobbitserver.cfg"

/home/hobbit/server/bin/bbcmd --env=$HOBBITCFG $TRAPCLIENT -C -s $3 -t $4 -d "$
1" -m "$5 ($2)"

  Hobbit External program

   The   trap.pl   script  is  run  from  Hobbit  every  5  minutes  from
   hobbitlaunch.cfg.
[trap]
        ENVFILE /home/hobbit/server/etc/hobbitserver.cfg
        NEEDS hobbitd
        CMD $BBHOME/ext/trap.pl -S
        LOGFILE $BBSERVERLOGS/trap.log
        INTERVAL 5m

Trap History

   If  you  setup  SNMPTT  to  log to MySQL, you can use the [17]trap.php
   script to query the SNMPTT log database on your MySQL server.
   Copy the trap.php script to the hobbit/www directory.

Example Cisco Switch Configuration

snmp-server enable traps config
snmp-server enable traps copy-config
snmp-server enable traps syslog
snmp-server enable traps entity
snmp-server enable traps flash insertion removal
snmp-server enable traps bridge
snmp-server enable traps stpx
snmp-server enable traps rtr
snmp-server enable traps c2900
snmp-server enable traps vtp
snmp-server enable traps vlancreate
snmp-server enable traps vlandelete
snmp-server enable traps port-security
snmp-server enable traps MAC-Notification
snmp-server enable traps envmon fan shutdown supply temperature status
snmp-server enable traps hsrp
snmp-server enable traps cluster
snmp-server enable traps vlan-membership
snmp-server host 10.1.1.1 PUBLIC

   Don't  forget, the Cisco MIBs don't have a SEVERITY set for any traps.
   You'll need to update the snmptt.conf file for your Cisco equipment to
   change the SEVERITY status for the OIDs you find interesting.
     _________________________________________________________________

   Last updated: 5/12/2006
   andy.farrior (@) victoriacollege.edu

References

   1. http://hobbitmon.sourceforge.net/
   2. http://www.bb4.org/
   3. http://www.samag.com/documents/s=9559/sam0503g/0503g.htm
   4. http://net-snmp.sourceforge.net/
   5. http://snmptt.sourceforge.net/
   6. http://simple-evcorr.sourceforge.net/
   7. file://localhost/var/www/htdocs/rc.snmptrapd
   8. ftp://ftp.apc.com/apc/public/software/pnetmib/mib/
   9. ftp://ftp.cisco.com/pub/mibs/v1/
  10. http://www.oidview.com/mibs/
  11. http://www.wtcs.org/snmp4tpc/mibs.htm
  12. file://localhost/var/www/htdocs/rc.snmptt
  13. file://localhost/var/www/htdocs/snmptt-ini.txt
  14. http://snmptt.sourceforge.net/docs/snmptt.shtml#LoggingDatabase-MySQL
  15. file://localhost/var/www/htdocs/rc.sec
  16. file://localhost/var/www/htdocs/trap-pl.txt
  17. file://localhost/var/www/htdocs/trap-php.txt
