How to find out which process generates high disk usage

T

tty23

Hi,

I have troubles with high disk usage on my FreeBSD system.

The system disk is used for 100% (according to gstat) for most
of the time. This is very annoying because the systems becomes
unresponsive.

It seems that this has something to do with sendmail. When I kill
all sendmail stuff and the periodic scripts the high disk usage
is gone.

These are some of the processes:
Code: 
root    85293  0,0  0,0  8340   308  ??  IJ    3:02am   0:00,00 /bin/sh /etc/periodic/daily/440.status-mailq
smmsp   85294  0,0  0,0 36724  1628  ??  DJ    3:02am   1:15,95 mailq -Ac (sendmail)
root    85295  0,0  0,0  5828   280  ??  IJ    3:02am   0:00,00 tee /dev/stderr
root    85296  0,0  0,0  9120   268  ??  IJ    3:02am   0:00,00 egrep -v (mqueue is empty|Total requests)
root    85297  0,0  0,0  2764   200  ??  IJ    3:02am   0:00,00 wc -l

The funny thing is, I deleted
Code: 
/etc/periodic/daily/440.status-mailq
, still it is running.

Is there any way to figure out which process generates all the disk access? This would
be a great help to find out what is going on.

Thanks

Johannes
 
That helps :)

I guess I have to look out for high VCSW numbers.

The offending seem to be sendmail. Any ideas on what
could be the reason for sendmail to hijack my disk?

One other process, that looks strange is yarrow. Any
ideas on what this is?
 
tty23 said:
The offending seem to be sendmail. Any ideas on what
could be the reason for sendmail to hijack my disk?
Click to expand...
A spammer using your system as a relay?
 
SirDice said:
A spammer using your system as a relay?
Click to expand...

Mhh, that is unlikely, it runs in my private network :)

Here are the processes, that cause the high disk usage:
Code: 
smmsp    1979  0,6  0,4 53108 15300  ??  DsJ  11:34am   1:20,31 sendmail: ./p77HRgWZ011358 from queue (sendmail)
smmsp    1980  0,6  0,4 53108 15344  ??  DsJ  11:34am   1:21,13 sendmail: ./p77HeFeh011358 from queue (sendmail)

I think these are the result mails from the "periodic daily" runs on my system. I guess for
some reason the mails are stuck in the queue.
What I do not understand:
  1. Why are the mails not delivered?
  2. Why do those undelivered mails saturate my OS disk with requests?

I guessed this entry from the maillog may be the cause:
Code: 
Aug 10 06:35:26 frege sendmail[70019]: gethostbyaddr(IPv6:fc00:1::1) failed: 1

But I added an entry for this address to the hosts file, and still, the sendmail process
is stuck.

The output of [CMD=""]service sendmail status[/CMD]:
Code: 
Cannot 'status' sendmail. Set sendmail_enable to YES in /etc/rc.conf or use 'onestatus' instead of 'status'.
sendmail_submit is running as pid 22345.
sendmail_clientmqueue is running as pid 22347.

Any ideas?

I am a bit desperate :\
 
What is inside these emails? Especially if you look in the qf* file, you should see the headers which may contain information about delivery problems. The df* file contains the body.
 
DutchDaemon said:
What is inside these emails? Especially if you look in the qf* file, you should see the headers which may contain information about delivery problems. The df* file contains the body.
Click to expand...

How to I figure out whats in the mails? Where should I look for the qf* and The df* files?
 
Either in /var/spool/mqueue/ or in /var/spool/clientmqueue/. You can open these files with a regular text viewer like less.
 
I just noticed that the mails are not always the same, as I thought, but changing every second or so:
Code: 
smmsp    1979  0,6  0,3 53108 11720  ??  DsJ  11:34am   1:57,74 sendmail: ./p6INdwu5042044 from queue (sendmail)
smmsp    1980  0,6  0,3 53108 11656  ??  DsJ  11:34am   1:58,85 sendmail: ./p77KeG6X011358 from queue (sendmail)
Code: 
smmsp    1979  0,8  0,3 53108 10968  ??  DsJ  11:34am   1:58,29 sendmail: ./p6LL4XO3056500 from queue (sendmail)
smmsp    1980  0,8  0,3 53108 10920  ??  DsJ  11:34am   1:59,39 sendmail: ./p6MIbNog013507 from queue (sendmail)

and so on.

I just rechecked, netstat -a shows not strange connections. Guess there is no spammer on my system.
However, I noticed the DsJ in the output of ps aux. So this is inside a jail.

The man page of ps tells me to look at /proc/⟨pid⟩/status for details.
Yay, this is inside my "git" jail.

For all my jails, I have this in my /etc/rc.conf
Code: 
# Prevent sendmail to try to connect to localhost
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

I thought, this would prevent sendmail from trying to send mails.

I guess this is wrong.

There are a bazillion mails in the /var/spool/clientmqueue on "git", even so many
that running ls -l does not finish within a few minutes.

EDIT1:
There are 850324 mails :O.
Note to self: find works much better in this case than ls.

This is what the mails contain:
Code: 
This is a MIME-encapsulated message

--p6N1xVuZ008652.1311433795/git.xxx.xxx

The original message was received at Tue, 12 Jul 2011 18:17:37 +0200 (CEST)
from localhost
with id p69F3qBC087141

   ----- The following addresses had permanent fatal errors -----
postmaster
    (expanded from: postmaster)

   ----- Transcript of session follows -----
postmaster... Deferred: Connection refused by [127.0.0.1]
Message could not be delivered for 5 days
Message will be deleted from queue

--p6N1xVuZ008652.1311433795/git.xxx.xxx
Content-Type: message/delivery-status

The mail goes on and on, its about 56kBytes.

Any ideas how to stop that?

Why does the sendmail process even try to send mails, I thought
my config would prohibit this?
 
Perhaps enabling outbound email to forward these nightly emails to an external address may help. See below for some insights. Note that a lot of programs and scripts are hardcoded to offer email to a local submission queue whether you like it or not.

http://lists.freebsd.org/pipermail/freebsd-questions/2006-April/120156.html
http://markmail.org/message/d2kspvd...ONE+page:1+mid:buyslvmzprdbjmjw+state:results

You can also alias all accounts mentioned in /etc/mail/aliases to /dev/null and run newaliases.
 
DutchDaemon said:
Perhaps enabling outbound email to forward these nightly emails to an external address may help. See below for some insights. Note that a lot of programs and scripts are hardcoded to offer email to a local submission queue whether you like it or not.
Click to expand...

Thanks for the hint, I will do that.
 
Note: I'm not entirely sure if this works perfectly inside a jail, but on a system that only has to send out these system emails, you don't need any sendmail lines in /etc/rc.conf. None. The ones that are present in /etc/defaults/rc.conf produce exactly that result (i.e. Sendmail processing mails offered on localhost and sending them out). So putting in an alias to an external address (or /dev/null) should take care of the problem.
 
You must log in or register to reply here.
Back
Top