• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

php-fpm (5.4.13) takes up all CPU resources

psyc

Member


Messages: 23

#26
This is normal for php-fpm to consume more process, as you application is using them. But by reading your latest top state, it looks more likely the issue is with mysql. Can you post your my.cnf config here, as it is consuming too much resources and there might be something going on in the background.

If you are logging slow queries then increasing the value or disabling it might help you a lot. As the main bottleneck in high traffic server is database and file I/O, so, if you are using HDD and logging every possible things then you are killing your application. Be reasonable while logging things, or use SSD.

While posting the process usage, the top -H or htop gives more detailed info.

Well I can just give you suggestions, but it is up to you to follow the best practices that you feel, I feel logging is always a big bottleneck. I have even disabled the nginx access log and just use error log in my production server. I use Piwik to do access analysis, which saves me from file I/O alot, as storing values in database is much more faster than files.
 

psyc

Member


Messages: 23

#27
meteor8488 said:
Is there any method I can restart php-fpm automatically?
Oh yes, you can always set a cron job, but this will result in chaos as your users start getting 50x errors while surfing.
 

meteor8488

Member


Messages: 80

#28
psyc said:
This is normal for php-fpm to consume more process, as you application is using them. But by reading your latest top state, it looks more likely the issue is with mysql. Can you post your my.cnf config here, as it is consuming too much resources and there might be something going on in the background.

If you are logging slow queries then increasing the value or disabling it might help you a lot. As the main bottleneck in high traffic server is database and file I/O, so, if you are using HDD and logging every possible things then you are killing your application. Be reasonable while logging things, or use SSD.

While posting the process usage, the top -H or htop gives more detailed info.

Well I can just give you suggestions, but it is up to you to follow the best practices that you feel, I feel logging is always a big bottleneck. I have even disabled the nginx access log and just use error log in my production server. I use Piwik to do access analysis, which saves me from file I/O alot, as storing values in database is much more faster than files.
Hi, I'm using SSD Intel 520 R1 to host my database. Below is my MySQL configuration:
Code:
[client]
port		= 3306
socket		= /tmp/mysql.sock

[mysqld]
server-id	= 1
port		= 3306
socket		= /tmp/mysql.sock
default-storage-engine=MyISAM
ignore-builtin-innodb
skip-external-locking
skip-networking
skip-name-resolve
#event_scheduler = 1

sql-mode = NO_UNSIGNED_SUBTRACTION

max_allowed_packet = 2M
max_connections = 768
max_connect_errors = 1844674407370954751

slow_launch_time=2
general-log=0
slow_query_log=1
slow-query-log-file=/web/log/slowquery.log
long_query_time=2
wait_timeout = 10
connect_timeout = 10
interactive_timeout = 10

tmp_table_size=64M
max_heap_table_size=64M
#max_heap_table_size=32M
key_buffer_size = 1792M
join_buffer_size = 3M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
sort_buffer_size = 2M
myisam_sort_buffer_size = 192M
table_open_cache = 4096
#table_open_cache = 512

query_cache_type = 0
query_cache_limit = 2M
query_cache_size = 0
#query_cache_size = 32M
#query_cache_size = 256M
query_cache_min_res_unit = 4K
query_alloc_block_size = 1024
query_prealloc_size = 8192

thread_concurrency = 48
thread_cache_size = 512

open-files-limit=10240

low-priority-updates=1
#concurrent_insert=ALWAYS
concurrent_insert=2

[mysqldump]
quick
max_allowed_packet = 32M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 64M
write_buffer = 64M

[mysqlhotcopy]
interactive-timeout
 

meteor8488

Member


Messages: 80

#29
psyc said:
You are logging loads of processes, if your application is heavy then some php functions are always going to take huge resources, so, you can change the timeout for slowlog, as when php starts logging every slow processes, this is overkill for php-fpm and your server.

I had the same issue, I tried every possible thing in earth, before changing the log options of php-fpm. You might want to change these variables.
Code:
slowlog = slow.log
request_slowlog_timeout = 30s
request_terminate_timeout = 120s
rlimit_core = unlimited

# In production to restart child process after 100 process server is overkill too
# As some pages are really huge, and if you are using e-commerce, you cannot use cache in every page
# Hence, the process gets bigger, so, increase 

pm.max_requests = 1000
pm.process_idle_timeout = 10s;
Your config is set for high traffic, but to use just one socket for processing all the processes is again overkill. If you have high traffics, then you might want to use few more sockets for php-fpm and nginx (I guess you have fair load balance compiled while installing nginx), this way your processes gets in balanced. eg: make three copies of php-fpm pool connections, and make it to listen few sockets like:
Code:
[www1] 
listen = /tmp/php-fpm1.sock;
# Other Config here

[www2]
listen = /tmp/php-fpm2.sock;
# Other Config here

[www3]
listen = /tmp/php-fpm3.sock;
# Other Config here

#####################################
# This will not only help you in balancing the load for socket buffer
# But also acts as fail safe method when one of php-fpm processes dies
# I guess you already have emergency threshold method to restart the child processes when it dies 
####################################
and in nginx listen directive use all 3 sockets one by one like:
Code:
fair; # To use fair load balancer
server unix:/tmp/php-fpm1.sock;
server unix:/tmp/php-fpm2.sock;
server unix:/tmp/php-fpm3.sock;
Other thing you have to do is decrease the pm.max_children and distribute among the 3 pools nicely, this will save you lots of cpu processes and eat up little bit more of your memory.

Other best practice for high traffic site using nginx is to do
Code:
multi_accept off; # Saves your children going wild
accept_mutex on; # Gives little bit needed relaxation for server
accept_mutex_delay 500ms;
Hi, I tried to use these parameters, but the issue still exists. And I found that one table in my database has an incorrect index, and I've fixed it. Now the CPU usage of MySQL is pretty low, usually less than 10%.

But sometimes, php-fpm still eats up all CPU resources.
 

meteor8488

Member


Messages: 80

#30
The issue happened again and again today:
Code:
last pid: 45484;  load averages: 121.21, 124.91, 107.61                                                                                                                      up 1+12:27:57  23:54:43
239 processes: 128 running, 110 sleeping, 1 lock
CPU:  9.9% user,  0.0% nice, 89.2% system,  0.8% interrupt,  0.1% idle
Mem: 4195M Active, 19G Inact, 7118M Wired, 563M Cache, 416M Free
ARC: 4042M Total, 1697M MFU, 873M MRU, 14M Anon, 97M Header, 1362M Other
Swap: 36G Total, 1212K Used, 36G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
44515 www           1  42    0   172M 46664K kqread 18   0:22 24.27% php-fpm
44517 www           1  52    0   172M 46884K accept  3   0:21 21.68% php-fpm
44475 www           1  46    0   172M 48660K accept 17   0:22 21.58% php-fpm
44485 www           1  42    0   172M 48472K CPU2    2   0:24 20.26% php-fpm
44526 www           1  42    0   172M 48344K zio->i 11   0:22 20.07% php-fpm
44472 www           1  43    0   172M 48500K CPU17  11   0:22 19.48% php-fpm
44521 www           1  48    0   172M 47276K accept 10   0:24 19.29% php-fpm
44476 www           1  40    0   176M 49032K CPU14  14   0:23 19.19% php-fpm
44528 www           1  48    0   176M 49376K accept  4   0:20 18.99% php-fpm
44497 www           1  40    0   172M 48332K accept 13   0:24 18.55% php-fpm
44506 www           1  48    0   172M 47052K accept 20   0:20 17.19% php-fpm
44495 www           1  46    0   172M 49244K accept  9   0:23 17.09% php-fpm
44507 www           1  46    0   176M 48316K accept 14   0:24 16.36% php-fpm
44577 www           1  76    0   176M 49812K RUN    14   0:23 14.79% php-fpm
44590 www           1  52    0   176M 49108K RUN    15   0:23 14.60% php-fpm
44570 www           1  52    0   176M 49644K RUN     3   0:23 14.36% php-fpm
44439 www           1  52    0   176M 49352K RUN    13   0:23 14.26% php-fpm
44451 www           1  52    0   172M 49176K RUN    17   0:23 14.26% php-fpm
44544 www           1  76    0   176M 50808K RUN     5   0:23 14.16% php-fpm
44468 www           1  52    0   172M 48736K RUN     3   0:23 14.16% php-fpm
44582 www           1  52    0   172M 48988K RUN     0   0:23 14.16% php-fpm
44471 www           1  52    0   172M 48340K accept 22   0:24 14.06% php-fpm
44414 www           1  52    0   172M 48384K RUN     5   0:23 14.06% php-fpm
44427 www           1  52    0   172M 47912K RUN     5   0:23 14.06% php-fpm
44443 www           1  52    0   172M 47868K RUN     1   0:23 14.06% php-fpm
44531 www           1  52    0   172M 48804K RUN     8   0:23 14.06% php-fpm
44588 www           1  52    0   172M 48300K RUN     7   0:23 14.06% php-fpm
44416 www           1  52    0   172M 48888K RUN    14   0:22 14.06% php-fpm
44445 www           1  76    0   172M 48180K RUN    17   0:23 13.96% php-fpm
44578 www           1  52    0   172M 48124K RUN    17   0:23 13.96% php-fpm
44555 www           1  76    0   172M 47796K CPU10   5   0:23 13.96% php-fpm
44456 www           1  52    0   172M 48492K RUN     8   0:23 13.96% php-fpm
44545 www           1  76    0   172M 48400K RUN    23   0:23 13.96% php-fpm
44560 www           1  76    0   176M 49340K RUN    12   0:23 13.96% php-fpm
44571 www           1  52    0   172M 48504K RUN     8   0:22 13.96% php-fpm
44461 www           1  52    0   172M 48648K RUN    12   0:22 13.96% php-fpm
44567 www           1  76    0   172M 47944K RUN     3   0:23 13.87% php-fpm
44575 www           1  52    0   172M 49164K RUN     8   0:23 13.87% php-fpm
44431 www           1  76    0   176M 48868K CPU16  16   0:23 13.87% php-fpm
44417 www           1  76    0   172M 48012K RUN     4   0:22 13.87% php-fpm
44566 www           1  52    0   172M 47672K RUN    21   0:23 13.77% php-fpm
44587 www           1  76    0   176M 49436K CPU19  19   0:23 13.77% php-fpm
44447 www           1  76    0   172M 47496K RUN    10   0:23 13.77% php-fpm
Restarting php-fpm will not fix this issue. I have no idea about what I can do now.
 

meteor8488

Member


Messages: 80

#31
I tried to reduce the php-fpm children to see what will be happen.
Code:
last pid: 47361;  load averages: 32.38, 78.56, 97.15498                                                                                                                      up 1+12:38:45  00:05:31
90 processes:  30 running, 60 sleeping
CPU: 10.6% user,  0.0% nice, 87.7% system,  1.2% interrupt,  0.5% idle
Mem: 4002M Active, 19G Inact, 7071M Wired, 852M Cache, 379M Free
ARC: 4096M Total, 1520M MFU, 1096M MRU, 2132K Anon, 100M Header, 1379M Other
Swap: 36G Total, 1212K Used, 36G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
47244 www           1  96    0   172M 50424K CPU5    5   0:42 75.88% php-fpm
47265 www           1  96    0   172M 49748K CPU2   22   0:35 75.59% php-fpm
47287 www           1  95    0   172M 50228K CPU22  20   0:31 74.66% php-fpm
47289 www           1  95    0   172M 49860K CPU23  22   0:31 74.07% php-fpm
47260 www           1  95    0   176M 53844K CPU11  11   0:36 73.29% php-fpm
47259 www           1  95    0   172M 50436K CPU15  20   0:36 73.00% php-fpm
47254 www           1  52    0   176M 53036K RUN     5   0:37 72.66% php-fpm
47263 www           1  95    0   172M 50492K CPU6    6   0:35 72.27% php-fpm
47245 www           1  94    0   176M 51960K CPU21  21   0:40 71.88% php-fpm
47255 www           1  94    0   176M 51552K CPU1    1   0:37 71.09% php-fpm
47238 www           1  94    0   172M 50964K CPU17  17   0:47 70.90% php-fpm
47258 www           1  94    0   176M 52228K CPU10  11   0:35 70.90% php-fpm
47302 www           1  94    0   176M 49884K CPU4    2   0:27 70.90% php-fpm
47282 www           1  52    0   176M 54564K CPU16  16   0:30 70.75% php-fpm
47243 www           1  52    0   176M 51288K RUN    14   0:41 70.46% php-fpm
47241 www           1  94    0   176M 53916K CPU0    0   0:41 70.26% php-fpm
47309 www           1  94    0   176M 52916K CPU9    2   0:22 70.26% php-fpm
47313 www           1  94    0   172M 46240K CPU8    8   0:21 70.17% php-fpm
47306 www           1  52    0   176M 54196K select  7   0:25 69.19% php-fpm
47274 www           1  93    0   172M 50100K RUN    18   0:33 68.36% php-fpm
47305 www           1  93    0   172M 49176K CPU7    7   0:25 68.36% php-fpm
47290 www           1  52    0   172M 49992K RUN     2   0:29 68.26% php-fpm
47304 www           1  52    0   176M 52032K select  6   0:25 67.38% php-fpm
47319 www           1  93    0   172M 51096K CPU13  22   0:19 67.38% php-fpm
47318 www           1  93    0   176M 54432K CPU19  13   0:19 65.97% php-fpm
47266 www           1  93    0   176M 52092K CPU18  18   0:34 65.87% php-fpm
47307 www           1  92    0   172M 49728K CPU3    4   0:24 65.09% php-fpm
47242 www           1  52    0   172M 50372K CPU14  21   0:38 64.36% php-fpm
47321 www           1  92    0   172M 46408K CPU20  20   0:16 63.87% php-fpm
47349 www           1  52    0   172M 41120K RUN     3   0:07 43.65% php-fpm
It seems that php-fpm always eats all CPU resources.
 

psyc

Member


Messages: 23

#32
meteor8488 said:
I tried to use these parameters, but the issue still exists.
Exactly which variables did you change?

meteor8488 said:
And I found that one table in my database has an incorrect index, and I've fixed it. Now the CPU usage of MySQL is pretty low, usually less than 10%.
Glad to hear that, I wish I knew more about MyISAM, but with the RAM you have I would have used at least 4 GB for key_buffer_size. Anyways, that's not the problem you have.

Your server looks very busy, I don't know what's the daily traffic visit but it looks impressive, and it's normal for php-fpm to consume more resources. If you want to control the process flow then multi accept off in nginx should help a lot, as the process passes through Master to Children, instead of all child being active to take connections. This will reduce some CPU consumption.

I really feel the CPU consumption by PHP-FPM is Normal for high traffic sites, even in your case.
 

xtaz

Well-Known Member

Thanks: 65
Messages: 310

#33
If you haven't done already I would suggest running PHP-APC (http://php.net/manual/en/book.apc.php) with the cache size set high enough to cover all the php files your site has. This caches the compiled object code in memory and means PHP-FPM doesn't have to recompile the script on every page load as it's just served directly from memory.

If that doesn't help then something else you can look at is configuring microcaching in nginx itself, check out the fastcgi_cache* directives. Configuring these will mean that pages are served directly from nginx and it only sends the request to PHP-FPM for 1 in every 1000 requests or something along those lines to keep the cache up to date.
 

psyc

Member


Messages: 23

#34
xtaz said:
If you haven't done already I would suggest running PHP-APC ( http://php.net/manual/en/book.apc.php ) with the cache size set high enough to cover all the php files your site has. This caches the compiled object code in memory and means PHP-FPM doesn't have to recompile the script on every page load as it's just served directly from memory.
I highly doubt if he hasn't used APC or some other opcode caching, as I could see he also has Memcached process running. He must already have APC.
xtaz said:
If that doesn't help then something else you can look at is configuring microcaching in nginx itself, check out the fastcgi_cache* directives. Configuring these will mean that pages are served directly from nginx and it only sends the request to PHP-FPM for 1 in every 1000 requests or something along those lines to keep the cache up to date.
I have checked his configuration, it already has Microcache settings, btw, Nginx doesn't do anything for PHP-FPM, Nginx passes the codes in <?php ?> to PHP-FPM and serves other static pages. This is also the reason lots of people use Nginx just for static page serving as its caching mechanism is really good. But if he has lots of <<<HEREDOCS inside PHP then Nginx also cannot help him, he have to optimize his code.
 

meteor8488

Member


Messages: 80

#35
xtaz said:
If you haven't done already I would suggest running PHP-APC ( http://php.net/manual/en/book.apc.php ) with the cache size set high enough to cover all the php files your site has. This caches the compiled object code in memory and means PHP-FPM doesn't have to recompile the script on every page load as it's just served directly from memory.

If that doesn't help then something else you can look at is configuring microcaching in nginx itself, check out the fastcgi_cache* directives. Configuring these will mean that pages are served directly from nginx and it only sends the request to PHP-FPM for 1 in every 1000 requests or something along those lines to keep the cache up to date.
Hi, I'm using PHP-APC for a long time. And I'm also using memcached for MySQL.

I was thinking about using nginx fastcgi cache. But my concern is that my website is a forum which is like the freebsd FreeBSD forums. And usually 4000 people on line. The articles update every minute. Is it ok for me to use fastcgi cache?

For example, now nginx cached the page http://forums.freebsd.org/showthread.php?p=220066#post220066. And I replied to this page. Will nginx update the cache? Or do I need to purge the cache?
 

meteor8488

Member


Messages: 80

#36
psyc said:
I highly doubt if he hasn't used APC or some other opcode caching, as I could see he also has Memcached process running. He must already have APC.

I have checked his configuration, it already has Microcache settings, btw, Nginx doesn't do anything for PHP-FPM, Nginx passes the codes in <?php ?> to PHP-FPM and serves other static pages. This is also the reason lots of people use Nginx just for static page serving as its caching mechanism is really good. But if he has lots of <<<HEREDOCS inside PHP then Nginx also cannot help him, he have to optimize his code.
Thanks @psyc,

Yesterday my webserver almost crashed. PHP-FPM ate up all CPU resources the whole night. I tried to restart PHP-FPM, but it's almost the same time when PHP-FPM is up and PHP-FPM ate up all CPU.

But today my server is running very well. It's quite strange.
 
Last edited by a moderator:

psyc

Member


Messages: 23

#37
Why I felt like you already had Microcache enabled, might be I surfed quickly. Microcache stores only static values, so, it doesn't interfare with your PHP process. Its a must in your scenario, but it may not help a lot in your PHP-FPM. I am really interested in your server setup, as I also have high traffic server, might be not high as yours, but the most resource consuming process is PHP-FPM too. Though I have multi accept off in Nginx, which controls a lot of process flow. Your Nginx configuration is also quiet interesting, Personally I would not keep that much values for Open_cache after all its not the actual caching mechanism.

When you have Forum site, it is really good to have INNODB storage engine. The writes are really fast.

BTW, are you still using pm static in php-fpm.conf ?
 

meteor8488

Member


Messages: 80

#38
psyc said:
Why I felt like you already had Microcache enabled, might be I surfed quickly. Microcache stores only static values, so, it doesn't interfare with your PHP process. Its a must in your scenario, but it may not help a lot in your PHP-FPM. I am really interested in your server setup, as I also have high traffic server, might be not high as yours, but the most resource consuming process is PHP-FPM too. Though I have multi accept off in Nginx, which controls a lot of process flow. Your Nginx configuration is also quiet interesting, Personally I would not keep that much values for Open_cache after all its not the actual caching mechanism.

When you have Forum site, it is really good to have INNODB storage engine. The writes are really fast.

BTW, are you still using pm static in php-fpm.conf ?
The reason why I'm still using myisam is that the code of the forum is designed only for myisam. They are some unique indexes which are not supported by innodb.

For now I'm still using pm static. I'm using three php-fpm and every of them has 60 pm.
 

xtaz

Well-Known Member

Thanks: 65
Messages: 310

#39
meteor8488 said:
Hi, I'm using PHP-APC for a long time. And I'm also using memcached for MySQL.

I was thinking about using nginx fastcgi cache. But my concern is that my website is a forum which is like the freebsd FreeBSD forums. And usually 4000 people on line. The articles update every minute. Is it ok for me to use fastcgi cache?

For example, now nginx cached page http://forums.freebsd.org/showthread.php?p=220066#post220066. And I replied this page. Will nginx update the cache? Or do I need to purge the cache?
Yes, you're right to be concerned about this. It totally depends on your content as to if this is suitable or not. As you suspect all .php pages are cached as static content until the timeout expires. So if you post a new forum article then it will not show any updates until the cache expires. The general idea is that the cache is set at a small time depending on the traffic level. Say two seconds. So if you receive 2000 requests/second then PHP is only hit once every 4000 requests, 3999 of those requests receive the cached page.

There are ways to control what content is cached, and what is passed through by making the code send certain headers or cookies, but for a forum this kind of caching is unwanted unless you have no other option.
 

freebuser

Member

Thanks: 5
Messages: 96

#44
Sorry I saw your previous thread and jump to the conclusion that it's jail related. Anyway as you said if it's stopping the issue when restarting the service I thought that might be due to rc order where php-fpm starts before the other services start eg. MySQL. If you're running different services on different jails you may need to consider the jail startup sequence.
 

dvl@

Aspiring Daemon
Developer

Thanks: 57
Messages: 530

#45
Sorry I saw your previous thread and jump to the conclusion that it's jail related. Anyway as you said if it's stopping the issue when restarting the service I thought that might be due to rc order where php-fpm starts before the other services start eg. Mysql. If you're running different services on different jails you may need to consider the jail startup sequence.
If that's the case, it's a bug with the start up with should specify such things. I will test that theory soon. Thanks.
 
Last edited:

dvl@

Aspiring Daemon
Developer

Thanks: 57
Messages: 530

#46
I have tried this, but no luck, still starts at 100%:

Code:
# PROVIDE: php-fpm
# REQUIRE: LOGIN apache22 libvirtd
 
Last edited by a moderator:

dvl@

Aspiring Daemon
Developer

Thanks: 57
Messages: 530

#47
I also noticed this in my logs as I was messing around with various options:

Code:
[19-Aug-2015 23:17:37] NOTICE: fpm is running, pid 44683
[19-Aug-2015 23:17:37] NOTICE: ready to handle connections
[19-Aug-2015 23:17:41] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 5 total children
[19-Aug-2015 23:17:42] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 6 total children
[19-Aug-2015 23:17:43] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 7 total children
[19-Aug-2015 23:17:44] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 8 total children
[19-Aug-2015 23:17:45] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 9 total children
[19-Aug-2015 23:17:46] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it
I found a clue here: http://serverfault.com/questions/65...te-enough-children-and-stops-serving-requests

In /usr/local/etc/php-fpm.conf I changed

Code:
pm = dynamic
to
Code:
pm = ondemand
This option change merely delays the 100% php-fpm processes until the first incoming http request.
 
Last edited by a moderator:

dvl@

Aspiring Daemon
Developer

Thanks: 57
Messages: 530

#48
I have abandoned Nginx & php-fpm and moved to Apache & mod_rewrite.
 

diizzy

Active Member

Thanks: 47
Messages: 221

#49
Nginx and php-fpm works fine on my machines, are you running something exotic on yours?
//Danne
 
Last edited by a moderator: