swap_pager_getswapspace(x): failed

My server got tons of swap space errors:

Code:
+swap_pager_getswapspace(9): failed
+swap_pager_getswapspace(5): failed
+swap_pager_getswapspace(5): failed
+swap_pager_getswapspace(9): failed
+swap_pager_getswapspace(9): failed
+swap_pager_getswapspace(6): failed

It's the second time my server had this problem, every time it killed mysql and caused some down time. Both happened at the quiet time such as Sunday morning, so it seems not related to the server load. After the server recovered from the problem, everything seems fine:

% swapinfo
Code:
Device          1K-blocks     Used    Avail Capacity
/dev/ad4s1b       2097152    74796  2022356     4%

The server has 2GB ram, MySQL was configured 500MB cache.

My questions:

- How to find out what caused the problem, or how to prevent this from happening again in the future?

- If it's not possible to track down the source, is increasing swap space a possible solution?

Thanks!
 
Same problem developed on my machine with 2GB RAM and 4GB swap. The problem kills both mysql and apache2, then eventually won't let me login, leaving me no choice but to hard reboot.

2 years without problem until I installed postfix. Now, disabling postfix don't matter, the problem continues. Really, 4GB is not enough for swap?

I'm running FreeBSD 7.1-RELEASE, apache-2.2.14_5, mysql-server-5.0.90.

Code:
[root@virtual /var/log]# swapinfo -h
Device          1M-blocks     Used    Avail Capacity
/dev/mirror/gm0s1b      4096      44K     4.0G     0%

last pid: 18032;  load averages:  0.10,  0.17,  0.22                                                        up 0+01:28:13  18:56:55
63 processes:  1 running, 62 sleeping
CPU:  0.0% user,  0.0% nice,  0.3% system,  0.3% interrupt, 99.4% idle
Mem: 581M Active, 455M Inact, 136M Wired, 56M Cache, 112M Buf, 736M Free
Swap: 4096M Total, 44K Used, 4096M Free

  PID USERNAME       THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
 1139 mysql           18  44    0   250M 50288K ucond  0   0:00  0.00% mysqld
  938 bind             5   4    0 72900K 60020K kqread 1   0:01  0.00% named
17317 www              1  20    0 44492K 27004K lockf  1   0:11  0.00% httpd
 2853 www              1  20    0 43488K 26220K lockf  0   2:42  0.00% httpd
17045 www              1  20    0 43484K 26216K lockf  0   0:18  0.20% httpd
17046 www              1  20    0 43484K 26212K lockf  0   0:09  0.00% httpd
17069 www              1   4    0 43480K 26208K kqread 0   0:11  0.00% httpd
17310 www              1  20    0 43480K 26088K lockf  0   0:09 12.89% httpd
16078 www              1  20    0 42464K 25392K lockf  0   1:00  0.00% httpd
17318 www              1  20    0 42456K 25164K lockf  0   0:09  0.39% httpd
17324 www              1  20    0 42428K 25116K lockf  1   0:17  0.20% httpd
17311 www              1  20    0 40408K 22936K lockf  0   0:15  0.10% httpd
 2194 root             1  44    0 35004K 17468K select 0   0:01  0.00% httpd
 1067 root             1  44    0 25812K 23156K select 0   0:00  0.00% perl5.10.1
 2842 www              5  51    0 23016K  9668K select 1   0:00  0.00% mono
13913 felix            1  44    0  8428K  2988K select 1   0:00  0.00% sshd
13909 root             1   4    0  8428K  2984K sbwait 0   0:00  0.00% sshd
 1049 root             1  44    0  5972K  2916K select 0   0:00  0.00% proftpd
 2253 root             1  44    0  5752K  2560K select 1   0:00  0.00% sshd
13915 felix            1   8    0  4400K  1856K wait   0   0:00  0.00% bash
13921 root             1   8    0  4400K  1852K wait   0   0:00  0.00% bash
 1055 root             1   8    0  4312K  1728K nanslp 0   0:00  0.00% smartd
13920 felix            1   8    0  3620K  1348K wait   0   0:00  0.00% su
18019 root             1  44    0  3496K  1456K CPU0   0   0:00  0.00% top
 1100 mysql            1   8    0  3464K  1148K wait   0   0:00  0.00% sh
 2266 root             1   8    0  3212K  1080K nanslp 0   0:00  0.00% cron
  874 root             1  44    0  3184K  1056K select 0   0:02  0.00% syslogd
 2329 root             1   5    0  3184K   912K ttyin  0   0:00  0.00% getty
 2331 root             1   5    0  3184K   912K ttyin  0   0:00  0.00% getty
 2333 root             1   5    0  3184K   912K ttyin  0   0:00  0.00% getty
 2334 root             1   5    0  3184K   912K ttyin  0   0:00  0.00% getty
 2332 root             1   5    0  3184K   912K ttyin  0   0:00  0.00% getty
 2335 root             1   5    0  3184K   912K ttyin  1   0:00  0.00% getty
 2330 root             1   5    0  3184K   912K ttyin  1   0:00  0.00% getty
 2336 root             1   5    0  3184K   912K ttyin  0   0:00  0.00% getty
  854 root             1   8    0  3160K  1480K nanslp 1   0:01  0.00% ipmon
  807 root             1  44    0  1888K   416K select 1   0:00  0.00% devd
  141 root             1  20    0  1380K   704K pause  0   0:00  0.00% adjkerntz
 
I finally realized it's a DDoS attack, how silly I am! Too many connections within a very short time used up the memory and swap. My server has little protection, it always happened at very quiet time, I guess my server was used to do some basic hacker training. :(
 
Back
Top