UFS PHP-FPM processes hanging in UFS-state

CyberCr33p

Well-Known Member

Reaction score: 29
Messages: 311

This issue happened in 3 different servers the last months.

Some PHP-FPM processes hang in UFS-state.

The websites that cause the hang are all Prestashop and it's related to their var/cache files (I believe it happens when they clear the cache and the files are regenerated).

This is a php-fpm process that hang:

Code:
procstat -kk 73548
  PID    TID COMM                TDNAME              KSTACK
73548 100630 php-fpm             -                   mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_xlock_hard+0x1f5 __lockmgr_args+0x2bf ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a vfs_hash_get+0xb0 ffs_vgetf+0x3f softdep_sync_buf+0xbee ffs_syncvnode+0x26f ffs_truncate+0x71b ufs_direnter+0x5f8 ufs_makeinode+0x5fb ufs_create+0x34 VOP_CREATE_APV+0x76 


And for all system:


I have to force reboot the server as /home is not possible to clean unmount. Then I run fsck which fixes some issues because of the unclean reboot. Then I run fsck again that shows no errors. Then I mount /home and first rename the cache folders and finally delete var/cache and allow it to regenerate prestashop files. But the same issue happens again after few days.

Any idea how to troubleshoot it?
 
OP
CyberCr33p

CyberCr33p

Well-Known Member

Reaction score: 29
Messages: 311

Code:
mount | grep home

/dev/mirror/gm0p7 on /home (ufs, local, noatime, nosuid, with quotas, soft-updates, acls)
/dev/mirror/gm0p8 on /home2 (ufs, local, noatime, noexec, nosuid, soft-updates)
/home/www on /home/jail/php72/home/www (nullfs, local, noatime, nosuid)
/tmp on /home/jail/php72/tmp (nullfs, local, noatime, nosuid)
/tmpfs on /home/jail/php72/tmpfs (nullfs, local, noatime, nosuid)
devfs on /home/jail/php72/dev (devfs, local, multilabel)
/home/www on /home/jail/php71/home/www (nullfs, local, noatime, nosuid)
/tmp on /home/jail/php71/tmp (nullfs, local, noatime, nosuid)
/tmpfs on /home/jail/php71/tmpfs (nullfs, local, noatime, nosuid)
devfs on /home/jail/php71/dev (devfs, local, multilabel)
/home/www on /home/jail/php56/home/www (nullfs, local, noatime, nosuid)
/tmp on /home/jail/php56/tmp (nullfs, local, noatime, nosuid)
/tmpfs on /home/jail/php56/tmpfs (nullfs, local, noatime, nosuid)
devfs on /home/jail/php56/dev (devfs, local, multilabel)
/tmp on /home/jail/redis/tmp (nullfs, local, noatime, nosuid)
devfs on /home/jail/redis/dev (devfs, local, multilabel)


And here is dumpfs /dev/mirror/gm0p7:

 
Top