FreeBSD 12.4 - performance regression somewhere?

rainer_d · Dec 20, 2022

Hi,

I have a HPE DL380Gen10. After the update to 12.4, it shows a lot of load.

Code:

(server-prod </root>) 0 # top -b 20
last pid: 49342;  load averages: 423.85, 399.58, 348.51  up 0+10:33:23    10:38:39
546 processes: 430 running, 115 sleeping, 1 zombie
CPU: 89.0% user,  0.0% nice,  4.4% system,  0.2% interrupt,  6.4% idle
Mem: 3022M Active, 8197M Inact, 37G Wired, 76G Free
ARC: 25G Total, 12G MFU, 9363M MRU, 237M Anon, 238M Header, 3780M Other
     14G Compressed, 25G Uncompressed, 1.82:1 Ratio
Swap: 64G Total, 64G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
59235 root          1  80    0   212M    55M RUN     15   2:37  29.39% php-fpm
 9389 root          1  80    0   212M    55M CPU3     3   2:13  29.30% php-fpm
94937 root          1  81    0   212M    54M RUN     11   7:57  29.05% php-fpm
12494 root          1  80    0   212M    55M CPU15   15   1:22  28.86% php-fpm
14319 root          1  81    0   212M    55M RUN      8   1:20  28.17% php-fpm
52501 root          1  81    0   212M    55M RUN      4   0:14  27.39% php-fpm
82538 root          1  80    0   212M    55M CPU15   15   0:16  27.10% php-fpm
62323 root          1  80    0   212M    55M RUN      9   2:05  26.66% php-fpm
80293 root          1  81    0   212M    55M RUN     10   1:33  25.68% php-fpm
 8670 root          1  79    0   212M    55M CPU14   14   0:52  25.20% php-fpm
70489 root          1  81    0   212M    55M RUN      1   2:06  24.85% php-fpm
48369 root          1  81    0   212M    55M RUN      0   0:04  24.85% php-fpm
75881 root          1  81    0   212M    55M RUN     14   2:32  24.66% php-fpm
46526 root          1  79    0   212M    55M CPU7     7   2:32  24.17% php-fpm
73903 root          1  79    0   212M    54M CPU12   12   1:40  24.17% php-fpm
22940 root          1  81    0   212M    55M RUN      4   3:05  23.78% php-fpm
52520 root          1  81    0   212M    55M RUN      6   1:39  23.68% php-fpm
25714 root          1  79    0   212M    54M CPU12   12   5:55  23.49% php-fpm
12415 root          1  81    0   212M    54M RUN      0  10:14  23.29% php-fpm
39214 root          1  79    0   212M    55M RUN     12   1:39  23.10% php-fpm

It worked more or less well on 12.3

This server uses the SmartPQI-driver for the Adaptec 3154-8i HBA with firmware 3.53

In htop one can see that most of the load is in userland:

I'm reverting back to the 12.3 boot-environment. I will try 13.1 next.

mer · Dec 20, 2022

rainer_d said:
I'm reverting back to the 12.3 boot-environment. I will try 13.1 next.

If you try 13.x keep in mind that ZFS changed to OpenZFS and don't do "zpool upgrade" when booted in 13.x or you may lose the ability to boot into 12.x.

rainer_d · Dec 20, 2022

Thanks. I have rarely upgraded a root zpool for fear of not being able to boot anymore...

mer · Dec 20, 2022

Interesting how many php-fpm processes are running. Did you try truss or other utility to see where the processes are? They could be sitting on select or some other "wait for input"

rainer_d · Dec 20, 2022

These were sometimes almost 500 PHP-FPM processes. Currently (back on 12.3) there are maybe 60.

These normally run as the user of the individual fpm-pool. Only the master-process runs as root.

I will have to see if this repeats with 13.1. Thank god for boot-environments...

mer · Dec 20, 2022

rainer_d said:
Thank god for boot-environments...

Agree completely

cracauer@ · Dec 20, 2022

I wonder whether this is another case of slow regex...

covacat · Dec 20, 2022

cracauer@ said:
I wonder whether this is another case of slow regex...

12.4 should not be affected by that

_martin · Dec 20, 2022

mer said:
They could be sitting on select or some other "wait for input"

Process would not be actively waiting for user input.

The very first thing that came to my mind is what covacat mentioned in other thread and cracauer@ said here - slow regex maybe? I went through the 12.4 relnotes and haven't seen mentioned it there though.

vmstat would help to see where are those load numbers coming from. truss-ing the process is also a good idea.

covacat · Dec 20, 2022

load is in userland difficult to tell with tools like truss
maybe opcache is not working
binaries should be mostly interchangeable between 12.3 and 12.4 so you can play the elimination game but it's kind of complex anyway

_martin · Dec 20, 2022

In the aforementioned thread I liked the idea of flame graphs, this is worth checking out. I personally have this in my todo list as I'm not familiar with those (shame on me as I have Brendan Gregg's book on my shelf).

covacat · Dec 20, 2022

the 12.4 box has a faster cpu than 12.3 box and a newer revision of php 7.4
results are kind of expected exception being intl_calendar

GitHub - rusoft/php-simple-benchmark-script: Очень простой скрипт тестирования быстродействия PHP \| Very simple script for testing of PHP operations speed (rusoft repo mirror) Очень простой скрипт тестирования быстродействия PHP \| Very simple script for testing of PHP operations speed (rusoft repo mirror) - GitHub - rusoft/php-simple-benchmark-script: Очень простой скрип... github.com	12.4	12.3	percent
01_math	4.285	6.075	70.5
02_string_concat	1.927	2.596	74.2
03_1_string_number_concat	1.961	2.439	80.4
03_2_string_number_format	1.739	2.379	73.1
04_string_simple_functions	2.19	3.559	61.5
05_string_multibyte	1.281	1.733	73.9
06_string_manipulation	4.109	7.659	53.6
07_regex	2.246	3.526	63.7
08_1_hashing	2.931	3.527	83.1
08_2_crypt	10.062	11.019	91.3
09_json_encode	3.102	4.733	65.5
10_json_decode	3.987	6.958	57.3
11_serialize	2.081	2.953	70.5
12_unserialize	2.525	3.211	78.6
13_array_fill	5.172	6.774	76.4
14_array_range	2.441	3.351	72.8
14_array_unset	4.003	5.189	77.1
15_clean_loops	2.747	2.841	96.7
16_loop_ifelse	4.226	8.596	49.2
17_loop_ternary	5.631	9.523	59.1
18_1_loop_defined_access	1.713	2.104	81.4
18_2_loop_undefined_access	8.497	11.192	75.9
19_type_functions	1.833	2.866	64.0
20_type_casting	1.831	3.1	59.1
21_0_loop_exception_none	0.303	0.415	73.0
21_1_loop_exception_try	0.333	0.443	75.2
21_2_loop_exception_catch	4.826	7.234	66.7
22_loop_null_op	2.363	3.965	59.6
23_loop_spaceship_op	2.009	2.506	80.2
26_1_class_public_properties	0.262	0.447	58.6
26_2_class_getter_setter	0.842	1.082	77.8
26_3_class_magic_methods	1.568	2.236	70.1
27_simplexml	4.529	7.175	63.1
28_domxml	4.75	7.483	63.5
29_datetime	0.944	1.43	66.0
30_intl_number_format	1.276	2.014	63.4
31_intl_message_format	5.213	8.644	60.3
32_intl_calendar	1.846	1.285	143.7
33_phpinfo_generate	7.093	9.311	76.2
	120.677	173.573	69.5

mer · Dec 20, 2022

_martin said:
Process would not be actively waiting for user input.

I was using "input" generically; hence why I added sitting on select/poll. They sit on file descriptors, which could represent a network socket or something else.