Machine with 8G RAM, plenty free, using swap?

F

frijsdijk

Hi,

I have a machine that's newly installed with 8.1-RELEASE-p2, AMD64 arch.

The machine has 8GM RAM and 8GB swap as well. It's running apache22 (prefork), php 5.2 and mysql 5.1.

Every other day I notice the machine is using almost all of its swap, while there is loads of RAM free. An example:

Code: 
last pid: 83767;  load averages:  0.81,  1.05,  0.76
93 processes:  1 running, 92 sleeping
CPU:     % user,     % nice,     % system,     % interrupt,     % idle
Mem: 5873M Active, 518M Inact, 1301M Wired, 202M Cache, 827M Buf, 6332K Free
Swap: 8192M Total, 7276M Used, 916M Free, 88% Inuse

  PID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
64890 mysql       21  44    0  3185M 94384K sbwait  0  15:05  8.69% mysqld
82755 www          1  47    0  2418M  1104M swread  7   1:19  5.37% httpd
83692 www          1  49    0   228M 70284K swread 11   0:01  5.37% httpd
82688 www          1  46    0  2422M  1287M swread 12   1:11  4.79% httpd
83695 www          1  48    0   228M 69492K sbwait  3   0:01  2.88% httpd
82754 www          1  45    0  2436M  1095M swread  4   1:28  0.59% httpd
 1027 root         4  44    0  5764K   384K rpcsvc  4  11:22  0.00% nfsd
82675 www          1  44    0  2436M   807M swread  2   1:49  0.00% httpd
82715 www          1  44    0  2436M  1043M swread  6   1:25  0.00% httpd
82683 www          1  44    0  2436M   836M swread  0   1:05  0.00% httpd
82717 www          1  76    0   158M     0K accept  8   0:51  0.00% <httpd>
 1376 root         1  44    0 12056K  1092K select  0   0:34  0.00% sendmail
 1178 root         1  44    0 37128K  1452K select 13   0:24  0.00% perl
  855 root         1  44    0  6984K   612K select  0   0:22  0.00% syslogd
 1388 root         1  44    0  7912K   360K nanslp  8   0:07  0.00% cron
 1370 root         1  44    0 26132K   680K select  4   0:05  0.00% sshd

Anyone any ideas what can cause this? Surely if I restart Apache and MySQL, all is normal again. I know that it's not a bad practice to restart Apache at least once a day, but MySQL should be able to run for longer periods of time, at least, that's my experience.

There are some custom things on top of the GENERIC kernel, I'm not sure if they have any negative effect here. They are:

Code: 
options         QUOTA
options         SHMMAXPGS=4096
options         SHMSEG=256
options         SEMMSL=1536
options         SEMMNI=256
options         SEMMNS=512
options         SEMMNU=512
options         SEMMAP=512
options         SEMOPM=2560
options         SEMUME=256
options         ACCEPT_FILTER_HTTP
options         PMAP_SHPGPERPROC=1000

Any ideas?
 
Short: There is no problem at all.

Long:
The swap exists for one reason: push non-used pages. So if apache/mysql have non-used pages (for example mysql indexes not used at all) -- why OS should hold it in RAM ? It can swap out part of data and use this memory for buffering/caching/etc. This will give system benefit, while not used pages are going swap.
Same for free memory - if memory is consuming power, why OS should use about 5% of it? Better if system use everything he has, so it can serve things faster.

There is no bad when "88% Inuse". It is really bad when you constanly see swap in/swap out -- this means you really low on memory
 
It looks to me like you need to tune up your database, if that is possible, and adjust your my.cnf values better. Your system is using most of its RAM as I can see by your statistics.
Code: 
Mem: [B]5873M[/B] Active, 518M Inact, 1301M Wired, 202M Cache, 827M Buf, [B]6332K[/B] Free
Swap: 8192M Total, 7276M Used, 916M Free, 88% Inuse
Also, mysql seems to consume half of your memory while apache the other half.
Code: 
  PID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
64890 mysql       21  44    0  3185M 94384K sbwait  0  15:05  8.69% mysqld
82755 www          1  47    0  2418M  1104M swread  7   1:19  5.37% httpd
83692 www          1  49    0   228M 70284K swread 11   0:01  5.37% httpd
82688 www          1  46    0  2422M  1287M swread 12   1:11  4.79% httpd
83695 www          1  48    0   228M 69492K sbwait  3   0:01  2.88% httpd
82754 www          1  45    0  2436M  1095M swread  4   1:28  0.59% httpd
I don't think that swapping is that bad and some times it can be the cause of bad indexing tables like mentioned. Yet, it can be annoying for the performance if it gets that high. I wonder how many hits you get on that web server because your CPU values look high enough.
If you can't tune the values of mysql then you might consider separating the database from the webserver.
 
Thanks all for replies so far. I can not really agree with the OS just swapping out to ram and that that's a good thing. This is the only machine where I see this happening - that it's actually using a substantial amount of swap (88% in this case) while there is close to the same percentage of RAM actually free. Swap is used, AFAIK, at the last moment when RAM is running out. Using swap isn't cheap, it's expensive, because it slows down the OS. I've never seen a server using this much swap when there is so much RAM available.

I think the solution would be:
- periodically restart apache (isn't happening right now, good for log rotation as well :-)
- downtune MySQL.

But still. Why use all that swap when all that RAM is free??
 
Wait up. Have I been looking with my nose?

Code: 
Mem: 5873M Active, 518M Inact, 1301M Wired, 202M Cache, 827M Buf, 6332K Free
Swap: 8192M Total, 7276M Used, 916M Free, 88% Inuse

I have been reading that as 6.3GB RAM free.

Ok, feel stupid now! :-)
 
More interesting question is
Code: 
82755 www          1  47    0  2418M  1104M swread  7   1:19  5.37% httpd
What httpd does so it can eat 2G? o_O
Sure, it can be non-real use (e.g. echo from copy-on-write), but its something funny.. What service this server do btw ?
 
frijsdijk said:
Wait up. Have I been looking with my nose?

Code: 
Mem: 5873M Active, 518M Inact, 1301M Wired, 202M Cache, 827M Buf, 6332K Free
Swap: 8192M Total, 7276M Used, 916M Free, 88% Inuse

I have been reading that as 6.3GB RAM free.

Ok, feel stupid now! :-)
Click to expand...
Don't, it happens :) But seriously that is the reason I used bold letters to quote your used/free ram.
Still it would be useful if we know the bandwidth you are serving and the hits you get.
You might want to use a monitor tool like Munin. It would give you a lot of information.
 
The size of the mysqld looking ok, but I am wondering why your httpd is consuming so much RAM. Depending on what apache modules you are using, this could be a hint for a memory leak in one of them. What is RAM usage of the httpd after a restart and some requests?
 
User23 said:
The size of the mysqld looking ok, but I am wondering why your httpd is consuming so much RAM. Depending on what apache modules you are using, this could be a hint for a memory leak in one of them. What is RAM usage of the httpd after a restart and some requests?
Click to expand...

Then it clears up.. In this case, Apache had an uptime of some weeks, and I think it's well known that Apache grows and grows. So I enabled logrotation, and Apache is OK now :-)


-- Frederique
 
frijsdijk said:
Then it clears up.. In this case, Apache had an uptime of some weeks, and I think it's well known that Apache grows and grows. So I enabled logrotation, and Apache is OK now :-)
Click to expand...

It's not well known to me that the Apache ram usage "grows and grows", even not with GBytes of logging files. I would keep an eye on it. :)
 
It does sound like you have a leak in some Apache module. You could just set the following option in your Apache config. I'm guessing you have a value of 0 in your current setup. Try a value like 500.

MaxRequestsPerChild: maximum number of requests a server process serves
 
You must log in or register to reply here.
Back
Top