Solved Debugging newsyslog?

`Orum

`Orum

I upgraded one of our servers to 12.1 earlier this month, and ever since then I've noticed a disturbing trend in CPU usage:
nsl.png

When investigating this, I noticed it was basically all due to newsyslog gobbling up CPU time, and seemingly more and more as time went on. I tried killing all the running processes (where you see a dip on the 27th), hoping it was just that one was hung up causing others to hang as well, but when they restart from cron, the problem continues, as you can see.

I've run newsyslog -nv, which shows the following:
Code: 
Processing /etc/newsyslog.conf
Found: <include> /etc/newsyslog.conf.d/*
Found: <include> /usr/local/etc/newsyslog.conf.d/*
Processing /etc/newsyslog.conf.d/ftp.conf
Processing /etc/newsyslog.conf.d/opensm.conf
Processing /etc/newsyslog.conf.d/pf.conf
Processing /etc/newsyslog.conf.d/sendmail.conf
Processing /usr/local/etc/newsyslog.conf.d/net-snmp.conf
...and then appears to just hang idefinitely. net-snmp.conf by default tries to rotate /var/log/snmpd.log, but as that doesn't exist on my system, I've commented it out, so the file is currently nothing but comments. Any idea why it hangs and then gobbles up CPU, or where I should go in terms of debugging from here?
 
Okay, after working away at this, I gave up and used truss, even though that feels like pounding a nail in with a wrecking ball. It seems newsyslog was spending a ton of time trying to rotate samba logs, so I went to investigate why. It turns out there were an absurd amount of files in /var/log/samba4/, and they were there due to a (much earlier) misconfiguration of newsyslog by me. As best as I can tell, here's the timeline:
  1. Some time while the server was running 11.x, I changed newsyslog to rotate Samba's log files with the mask log.*.
  2. The log files build up, as newsyslog renames the old logs to a new file that still matches that same mask, e.g. log.smbd becomes log.smbd.0. But, both of these match log.*. Thus, files replicate exponentially due to my misconfiguration, doubling every time samba's log files are rotated (i.e. weekly).
  3. At some point later, newsyslog stops working (non-zero signal), but only after it rotates other logs. I suspect the whole samba mess I created was causing it to crash, but not hang.
  4. I upgrade to 12.1, and newsyslog no longer crashes when it hits the samba log directory, and instead hangs and grabs CPU.
So, lesson learned: don't use wildcards at the end of any file mask in newsyslog.conf.
 
You must log in or register to reply here.
Back
Top