Log swapping processes

[mfeldheim]

We had some weird attack causing some of our front machines to run into swap and even run out of swap. Now the logfiles only show "out of swap" errors, which doesnt really help to find out what process exactly forced the system into crash - and ideas how I could improve logging to tackle / maybe even kill swapping processes?

# tail -n 2 /var/log/messages
Sep 28 09:06:43 gateway-001 kernel: swap zone exhausted, increase kern.maxswzone
Sep 28 09:06:43 gateway-001 last message repeated 192 times

# sysctl -a | grep "kern.maxswzone"
kern.maxswzone: 33554432

# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/ad4s1b      16777216        0 16777216     0%

 

[AndyUKG]

Try changing the log settings in /etc/newsyslog.conf so that you keep more historical info?

But if your problem is simply running out of memory you may well not see much in the logs apart (apart from the fact the system is out of memory).

You mention its a "front machine", if it's a web server try tuning apache or whatever so that it prevents so many connections that you run out of memory and dies might be the first thing to consider/check,

Andy.

 

[phoenix]

Keep top running in a terminal, and watch the memory columns to see what's using all the RAM?

 

[fronclynne]

Boy, I'd like to say that ps(1) could do something, but I'm not sure. It does too much for my tiny brain.

 

[Galactic_Dominator]

I think you're looking for sysutils/fuser. procstat/pstat, and maybe even fstat may have access to the info as well, but it's harder to parse.