Syslog parser

B

bbzz

I'm not sure this is the right place.

I have a syslog server which collects all syslog messages from literally dozens of Cisco and Juniper routers, in one file (this is for another topic) which rotates each day.

What I need is a script which will parse this huge file (about 300 MB uncompressed), and just give some basic info such as source/destination IP address, ports, type of message, etc. One of the problems is that you pretty much get a new type of message every day.

Is there a script/software which can do this, and can be fed new types of messages so that it knows how to handle them?

Someone suggested sawmill for this, but it's not doing what I need it to do.
 
What you're looking for is sometimes called a "SIEM", i.e. Security Information and Event Management. There are many commercial products that claim to be SIEMs. There is a couple of opensource ones. The one I could find in FreeBSD Ports collection is security/ossec-hids-server. According the project website, some of the Cisco and Juniper products are supported
Devices support via Syslog
  • Cisco IOS routers (all versions)
  • Juniper Netscreen (all versions)
Click to expand...
I do not have any personal experience with OSSEC, however, so I can not comment on how it works.
 
security/logcheck has been around for ages, though it is normally used to parse syslog output in real time and trigger alerts based on what it encounters. There are probably summarizing and reporting options in in too. Too long ago to remember all it did. It's simple, CLI-based.
 
Actually security/logcheck does not work real time but is triggered by a crontab entry. I think its several rule files make it a very suitable candidate because it's relatively easy to add a new rule. However, because it's main purpose is basically to report suspicious entries it might require some (massive?) tweaking to change the e-mail which it sends out (because this is basically a rough summary of suspicious rules).

Quite frankly I'd take a completely different route: look into sysutils/syslog-ng or maybe even sysutils/rsyslog7 for the main syslog server. These allow you to store your log entries just about everywhere, including an SQL back-end.

You might want to start looking at sysutils/rsyslog7 which can also use a back-end SQL server but seems to be more focussed on security and flexibility whereas sysutils/syslog-ng is a bit more global (for example; rsyslog7 will provide direct support for common backends such as MySQL and PostgreSQL but syslog-ng basically relies on databases/libdbi to handle the SQL support).

Once you have all your log entries in a database it should become much easier to extract information from it. You'd still need a program of some sort, but even a simple shell script should be able to extract information based on specific filters.
 
You must log in or register to reply here.
Back
Top