Syslog vs Cisco

Ophiuchus

Ophiuchus

Hi everyone,

I have just deployed a new syslog server for our Cisco switches, routers and access points (more than 250 devices) and I am facing a weird problem. The syslogd seems to be working but with flaws. Here is the situation:

  • I logged into one of our cisco switches and changed its logging ip to the syslog server.
  • Logged into the switch and simply did something that produces a syslog connection (conf t; <enter>; <enter>; end)
  • Everything worked fine and I had that log entry in my syslog server
  • After that I changed my syslog server ip to its new location (all devices are configured to send their logs to this new IP)
  • syslog log file had a few log entries and than almost nothing (explained below)

Additional information:
  • tshark interface dump shows about 5 to 10 entries per second and the log file grows about 1 entry per 30 seconds
  • I have disabled ipf (and even rebooted), and nothing has changed
  • The server in question is a virtual machine with adequate resources
  • Our previous syslog server (ubuntu) had exactly the same problems with syslogd and I installed and used syslog-ng (without any problems)
  • I can paste my configuration files if desired

So, I know that I just can install and use syslog-ng instead of syslogd for this case and I am almost sure that it will work. Well, the truth is that I am a real bull headed person who really wants to understand "what is going on". Any help will be truly appreciated. Thanks in advance...

About me:
  • I am totally new to FreeBSD world (6 months) and have almost no experience on any Linux distros. According to my decent experience with FreeBSD, I can only say that FreeBSD changed my whole IT vision and understaning (You were living in a dream world "Oke").
 
Hopefully not obvious, but are you sure the missed entries aren't just a case of syslogd consolidating duplicates? Perhaps you can try launch syslogd with the "-c" parameter:

Code: 
-c      Disable the compression of repeated instances of the same line
        into a single line of the form ``last message repeated N times''
        when the output is a pipe to another program.  If specified
        twice, disable this compression in all cases.
 
Nah, I am sure that they are no duplicates. There are about 100 access points in our domain and their syslog level are set to debug (due to another mysterious problem). What I mean is that those APs should be creating hell of a syslog transaction.

Anyway, I'll try that tomorrow morning and post the result.
 
Ophiuchus said:
There are about 100 access points in our domain and their syslog level are set to debug (due to another mysterious problem).
Click to expand...
I don't know about your APs but on Cisco devices turning on debugging can seriously impact the performance of the device. You can even get it to crash (just try debug all).

Also note that syslog is UDP. That means there's no guarantee packets will actually arrive at their destination.
 
Yes, I know that but syslog-ng on an ubuntu server (with the same resources) handles all of these syslog messages so I believe that a FreeBSD server also can.

Anyway, I am a little bit frustrated. I'll try setting up syslog-ng on the new server.
 
I would suggest installing sysutils/syslog-ng anyway. It's a lot easier to configure and has a lot more functionality.

You can probably copy the configuration of the 'old' Ubuntu server and use it on FreeBSD without any issues. The only thing that might need changing are the paths you used.

It should also be possible to run both. The 'standard' syslog can be used just for the local machine. Syslog-ng can be configured to only listen on the network.
 
Installed and configured syslog-ng. I just faced one simple problem and the problem is solved after adding
Code: 
use_dns(no);
use_fqdn(no);
options to the syslog-ng.conf file.

Can it be that syslog (not ng) is trying to resolve the name of the syslog clients before writing the syslog entries?

I shall check this one when I have more time.
 
Ophiuchus said:
can it be, that syslog (not ng) is trying to solve the name of the syslog clients before writing the syslog entries?
Click to expand...
It does. From syslogd(8):
Code: 
     -n      Disable dns query for every request.
Which implies it's turned on by default.
 
I have disabled DNS queries in syslog.conf and things got a lot better. but, I still found some missing syslog entries. I had to switch back to syslog-ng setup and the logs are again back to normal.
 
We use syslog-ng to aggregate syslogs from about 15000 routers, switches and an assortment of other network devices. Granted, it runs on Solaris but it's an old machine so you shouldn't have any problems with it.
 
That's a real statistic. We've got much fewer devices and I am also planning to get logs from our servers. That will be in some distant future so no problems at this point.

May I ask, what is your level of logging (info, warning, devel, critical... etc)?
 
Ophiuchus said:
May I ask, what is your level of logging (info, warning, devel, critical... etc)?
Click to expand...
I'll have to look it up but I think it's just the standard setting in this regard.

We mainly have it to aggregate all the logs. But we do have some interesting scripts. For example there's a script plugged into syslog-ng, its sole purpose is to look for configuration writes. As soon as it detects one it'll flag the device in our database. Another, continuously running, script picks this up, grabs the config from the device and stores it. Then another script comes along and checks the configs for compliance. Things like AAA settings, ACLs, IOS versions, etc. Any anomalies are detected and flagged so our quality assurance teams can fix it and make it comply with company policy.
 
I have checked previous days log file and saw that the log file contains about 400 000 lines of log. Not much of a disk space, of course.

I liked the script you have mentioned in your previous post. May be, I should try writing a similar one (exercise / increase my experience). As far as I know, our LMS server (old Ciscoworks) does exactly the same thing as this script.
 
You must log in or register to reply here.
Back
Top