FreeBSD 11.2 system no responsive - all processes in D state

CyberCr33p

Active Member

Reaction score: 20
Messages: 242

I have an issue with 11.2-RELEASE-p3 which happened to 3 servers with similar hardware. These servers run without issues in the past.

When the issue happens the server is pingable but can't SSH.

Datacenter connect KVM but didn't show anything useful.

The problem is that all processes that are related somehow to /home stuck in D state.

I was able to shutdown mysql which has the data in /var partition.

The servers use UFS SU+J but I don't think it's related as they run stable for a long time.

Here is the output of ps -aux:

Code:
USER       PID  %CPU %MEM     VSZ     RSS TT  STAT STARTED         TIME COMMAND
root        11 728.5  0.0       0     128  -  RNL  18Sep18 182777:12.78 [idle]
root        25  98.5  0.0       0     112  -  DL   18Sep18     45:18.06 [bufdae
root         0   0.0  0.0       0     480  -  DLs  18Sep18    142:23.56 [kernel
root         1   0.0  0.0    5424     128  -  SLs  18Sep18      0:02.80 /sbin/i
root         2   0.0  0.0       0      16  -  DL   18Sep18      0:00.00 [crypto
root         3   0.0  0.0       0      16  -  DL   18Sep18      0:00.00 [crypto
root         4   0.0  0.0       0      48  -  DL   18Sep18     11:02.67 [cam]
root         5   0.0  0.0       0      16  -  DL   18Sep18      0:00.17 [soaiod
root         6   0.0  0.0       0      16  -  DL   18Sep18      0:00.17 [soaiod
root         7   0.0  0.0       0      16  -  DL   18Sep18      0:00.17 [soaiod
root         8   0.0  0.0       0      16  -  DL   18Sep18      0:00.17 [soaiod
root         9   0.0  0.0       0      16  -  DL   18Sep18      0:00.00 [sctp_i
root        10   0.0  0.0       0      16  -  DL   18Sep18      0:00.00 [audit]
root        12   0.0  0.0       0     256  -  WL   18Sep18     36:53.30 [intr]
root        13   0.0  0.0       0      48  -  DL   18Sep18     23:03.75 [geom]
root        14   0.0  0.0       0      80  -  DL   18Sep18      0:26.05 [usb]
root        15   0.0  0.0       0      16  -  DL   18Sep18     49:00.44 [rand_h
root        16   0.0  0.0       0      16  -  DL   18Sep18      0:00.95 [enc_da
root        17   0.0  0.0       0      16  -  DL   18Sep18      0:00.19 [g_mirr
root        18   0.0  0.0       0      16  -  DL   18Sep18      0:00.65 [g_mirr
root        19   0.0  0.0       0      16  -  DL   18Sep18      0:13.16 [g_mirr
root        20   0.0  0.0       0      16  -  DL   18Sep18      8:17.13 [g_mirr
root        21   0.0  0.0       0      16  -  DL   18Sep18      0:11.60 [g_mirr
root        22   0.0  0.0       0      48  -  DL   18Sep18     10:04.32 [pageda
root        23   0.0  0.0       0      16  -  DL   18Sep18      0:00.00 [vmdaem
root        24   0.0  0.0       0      16  -  DNL  18Sep18      0:00.02 [pageze
root        26   0.0  0.0       0      16  -  DL   18Sep18      1:54.92 [bufspa
root        27   0.0  0.0       0      16  -  DL   18Sep18    131:28.87 [syncer
root        28   0.0  0.0       0      16  -  DL   18Sep18      1:38.53 [vnlru]
root        29   0.0  0.0       0      16  -  DL   18Sep18      1:52.41 [g_mirr
root        30   0.0  0.0       0      16  -  DL   18Sep18      1:01.67 [g_mirr
kagias     560   0.0  0.0   10348    6140  -  D    08:05        0:00.03 imap: [
root       617   0.0  0.0    7812    3340  -  Is   09:45        0:00.05 /usr/lo
root       816   0.0  0.0    6468    2484  -  I    10:10        0:00.00 cron: r
root       992   0.0  0.0    6676    2592  -  D    09:45        0:00.00 grep -q
nobody    1016   0.0  0.0   32492   11580  -  I    18:51        0:00.12  postfw
root      1062   0.0  0.0    7812    3260  -  Is   09:45        0:00.05 /usr/lo
root      1079   0.0  0.9 2146172  301968  -  Ss   Sun22        0:58.29 php-fpm
root      1352   0.0  0.0    6676    2580  -  D    09:55        0:00.00 grep -q
valiraki  1685   0.0  0.0   10336    5640  -  D    09:44        0:00.01 imap: [
slumdg10  1701   0.0  0.0    9312    5072  -  D    10:07        0:00.01 pop3: [
nobody    1886   0.0  0.0   32492   11580  -  S    18:52        0:00.11  postfw
disoma2   1936   0.0  0.0   10336    5564  -  D    09:39        0:00.00 imap: [
drcs      2024   0.0  0.6 2156736  205804  -  DJ   09:33        0:00.13 php-fpm
iatrougr  2077   0.0  0.0   13576    9076  -  D    09:38        0:00.01 lmtp: [
slumdg10  2330   0.0  0.0    9312    5236  -  D    09:32        0:00.06 pop3: [
root      2394   0.0  0.0    7812    3340  -  Is   10:05        0:00.05 /usr/lo
angelzaf  2445   0.0  0.0   11512    6972  -  D    10:07        0:00.09 pop3: [
slumdg10 21831   0.0  0.0   10356    5660  -  D    09:40        0:00.01 imap: [
slumdg10 22268   0.0  0.0   10356    5660  -  D    09:52        0:00.01 imap: [
root     22511   0.0  0.0    6676    2592  -  D    10:05        0:00.00 grep -q
valiraki 22848   0.0  0.0    9320    4708  -  D    10:03        0:00.00 pop3: [
apollon  23115   0.0  0.0   10348    5656  -  D    09:35        0:00.01 imap: [
elcampo  23380   0.0  0.0   12460    6268  -  D    13:24        0:00.27 imap: [
valiraki 23385   0.0  0.0    9320    4708  -  D    10:02        0:00.00 pop3: [
apollon  23465   0.0  0.0   10336    5640  -  D    09:39        0:00.01 imap: [
apollon  23572   0.0  0.0   10336    5608  -  D    09:35        0:00.01 imap: [
osanet   55716   0.0  0.0    9320    4700  -  D    09:39        0:00.00 pop3: [
disoma2  55755   0.0  0.0   10336    5564  -  D    09:50        0:00.00 imap: [
slumdg10 55813   0.0  0.0    9320    4736  -  D    10:05        0:00.00 pop3: [
slumdg10 55861   0.0  0.0    9312    5052  -  D    09:57        0:00.01 pop3: [
root     55870   0.0  0.0    6468    2484  -  I    10:05        0:00.00 cron: r
slumdg17 55940   0.0  0.0    9308    4800  -  D    09:52        0:00.00 pop3: [
chriskar 56049   0.0  0.0    9328    5016  -  D    09:38        0:00.01 pop3: [
root     56362   0.0  0.0    6468    2484  -  I    09:45        0:00.00 cron: r
slumdg10 56469   0.0  0.0   10356    5644  -  D    09:57        0:00.01 imap: [
slumdg10 56799   0.0  0.0   13580    9112  -  D    09:37        0:00.01 lmtp: [
disoma2  56800   0.0  0.0   10336    5564  -  D    10:03        0:00.00 imap: [
surfaces 56857   0.0  0.0    9312    4968  -  D    09:37        0:00.00 pop3: [
root     56957   0.0  0.0    7812    3340  -  Is   10:10        0:00.06 /usr/lo
kyrlas   57291   0.0  0.0    9328    5192  -  D    09:49        0:00.01 pop3: [
slumdg10 57629   0.0  0.0    9300    4680  -  D    09:44        0:00.00 pop3: [
www      57642   0.0  5.2 2382904 1719488  -  D    07:43        0:05.78 nginx:
kyrlas   58015   0.0  0.0    9328    5192  -  D    09:41        0:00.01 pop3: [
kyrlas   58261   0.0  0.0    9328    5192  -  D    09:42        0:00.01 pop3: [
root     58717   0.0  0.0    6468    2484  -  I    10:00        0:00.00 cron: r
slumdg10 58856   0.0  0.0    9308    4836  -  D    09:59        0:00.00 pop3: [
slumdg18 59111   0.0  0.0   13584    9084  -  D    09:55        0:00.01 lmtp: [
root     59417   0.0  0.0    6676    2580  -  D    09:50        0:00.00 grep -q
philip   59431   0.0  0.6 2142400  195024  -  DJ   09:30        0:00.67 php-fpm
disoma2  59577   0.0  0.0   10336    5564  -  D    09:50        0:00.01 imap: [
valiraki 60053   0.0  0.0   10336    5652  -  D    10:04        0:00.01 imap: [
root     60382   0.0  0.0    6676    2592  -  D    09:40        0:00.00 grep -q
slumdg10 60645   0.0  0.0    9312    5052  -  D    09:52        0:00.01 pop3: [
valiraki 60784   0.0  0.0   10336    5640  -  D    09:40        0:00.00 imap: [
root     60835   0.0  0.0   12456   12552  -  Ss   18Sep18      0:57.05 /usr/sb
elcampo  60918   0.0  0.0    9300    4948  -  D    09:39        0:00.00 pop3: [
disoma2  60956   0.0  0.0   10336    5564  -  D    09:33        0:00.01 imap: [
georgiou 61080   0.0  0.0   10336    5596  -  D    09:50        0:00.01 imap: [
root     61596   0.0  0.0    6468    2484  -  I    10:05        0:00.00 cron: r
disoma2  62541   0.0  0.0   10336    5564  -  D    10:01        0:00.00 imap: [
root     62546   0.0  0.0    6468    2484  -  I    09:45        0:00.00 cron: r
slumdg10 62548   0.0  0.0    9312    5052  -  D    09:37        0:00.01 pop3: [
disoma2  62798   0.0  0.0   10336    5564  -  D    10:01        0:00.00 imap: [
disoma2  63240   0.0  0.0   10336    5564  -  D    09:33        0:00.00 imap: [
valiraki 63606   0.0  0.0    9312    4692  -  D    09:41        0:00.00 pop3: [
slumdg10 64044   0.0  0.0    9308    4816  -  D    10:00        0:00.00 pop3: [
kyrlas   64109   0.0  0.0    9328    5192  -  D    09:46        0:00.01 pop3: [
traktoor 64319   0.0  0.0   10336    5596  -  D    09:50        0:00.00 imap: [
slumdg10 64491   0.0  0.0    9308    4836  -  D    09:43        0:00.00 pop3: [
www      64657   0.0  5.2 2382904 1720184  -  D    07:43        0:06.65 nginx:
slumdg10 64686   0.0  0.0   10356    5644  -  D    09:56        0:00.00 imap: [
backslas 64814   0.0  0.0   10336    5596  -  D    09:42        0:00.01 imap: [
costas   64969   0.0  0.0    9316    5008  -  D    09:47        0:00.01 pop3: [
root     65169   0.0  0.0    6468    2484  -  I    09:50        0:00.00 cron: r
chriskar 65430   0.0  0.0    9308    4952  -  D    09:38        0:00.00 pop3: [
slumdg10 65487   0.0  0.0    9332    4960  -  D    10:02        0:00.00 pop3: [
slumdg17 65866   0.0  0.0    9300    4656  -  D    09:58        0:00.00 pop3: [
slumdg10 66021   0.0  0.0    9332    4960  -  D    09:57        0:00.01 pop3: [
www      66116   0.0  5.2 2382904 1720748  -  D    07:43        0:12.85 nginx:
disoma2  66416   0.0  0.0   10336    5564  -  D    10:01        0:00.00 imap: [
vedolger 66571   0.0  0.9 2145140  301528  -  S    09:00        0:00.13 php-fpm
valiraki 67457   0.0  0.0   10336    5688  -  D    09:30        0:00.01 imap: [
root     67514   0.0  0.0    7812    3260  -  Is   10:10        0:00.06 /usr/lo
valiraki 68081   0.0  0.0    9300    4684  -  D    10:00        0:00.00 pop3: [
costas   68273   0.0  0.0   10336    5592  -  D    09:57        0:00.00 imap: [
root     68307   0.0  0.0    6468    2484  -  I    09:50        0:00.00 cron: r
disoma2  68413   0.0  0.0   10336    5564  -  D    09:33        0:00.01 imap: [
disoma2  68611   0.0  0.0   10336    5564  -  D    10:03        0:00.00 imap: [
valiraki 68628   0.0  0.0    9332    4712  -  D    09:42        0:00.00 pop3: [
root     68663   0.0  0.0    7812    3340  -  Is   09:35        0:00.06 /usr/lo
valiraki 69097   0.0  0.0    9320    4708  -  D    09:38        0:00.00 pop3: [
valiraki 69417   0.0  0.0   10336    5636  -  D    10:05        0:00.02 imap: [
disoma2  69499   0.0  0.0   10336    5564  -  D    09:33        0:00.00 imap: [
root     69632   0.0  0.0    7812    3260  -  Is   09:35        0:00.05 /usr/lo
slumdg10 70065   0.0  0.0    9308    4836  -  D    09:41        0:00.00 pop3: [
www      70177   0.0  5.2 2382904 1720724  -  D    07:43        0:14.14 nginx:
costas   70283   0.0  0.0   10368    5752  -  D    09:53        0:00.00 imap: [
milas    70542   0.0  0.0    9300    4720  -  D    10:04        0:00.00 pop3: [
root     70814   0.0  0.0    7844    3320  -  I    Tue14        0:08.07 dovecot
karavid  70856   0.0  0.0   10344    5604  -  D    09:39        0:00.00 imap: [
slumdg10 71069   0.0  0.0    9308    4836  -  D    09:34        0:00.01 pop3: [
osanet   71388   0.0  0.0    9320    4700  -  D    09:49        0:00.00 pop3: [
slumdg10 71467   0.0  0.0   10356    5660  -  D    09:44        0:00.01 imap: [
georgiou 71584   0.0  0.0   10336    5596  -  D    09:58        0:00.00 imap: [
kagias   71696   0.0  0.0   10336    5596  -  D    09:55        0:00.00 imap: [
slumdg10 71851   0.0  0.0    9332    4960  -  D    09:52        0:00.01 pop3: [
root     72140   0.0  0.0   10044    5788  -  I    Tue14        0:29.05 dovecot
root     72274   0.0  0.0   13164    8300  -  Is   09:38        0:00.01 sshd: c
root     72328   0.0  0.0    7812    3340  -  Is   09:55        0:00.06 /usr/lo
nobody   72616   0.0  0.0   32492   11564  -  I    13:08        0:00.19  postfw
slumdg10 72696   0.0  0.0    9332    4960  -  D    09:37        0:00.00 pop3: [
milas    72954   0.0  0.0    9312    4692  -  D    09:37        0:00.01 pop3: [
disoma2  74069   0.0  0.0    9312    4848  -  D    10:01        0:00.00 pop3: [
elcampo  74407   0.0  0.0    9300    4948  -  D    09:40        0:00.00 pop3: [
angelzaf 74446   0.0  0.0   13728    9308  -  D    09:45        0:00.01 lmtp: [
slumdg10 74954   0.0  0.0    9324    5012  -  D    09:48        0:00.00 pop3: [
osanet   75118   0.0  0.0    9308    4844  -  D    09:35        0:00.00 pop3: [
georgiou 75142   0.0  0.0   10980    6396  -  D    09:58        0:00.01 imap: [
chasapis 75240   0.0  0.6 2148544  210392  -  DJ   09:32        0:00.40 php-fpm
backslas 75325   0.0  0.0   10336    5596  -  D    10:06        0:00.00 imap: [
slumdg10 75397   0.0  0.0    9300    4680  -  D    09:42        0:00.00 pop3: [
valiraki 76989   0.0  0.0    9320    4708  -  D    09:48        0:00.00 pop3: [
georgiou 78130   0.0  0.0   10336    5728  -  D    09:53        0:00.01 imap: [
root     78173   0.0  0.0    7812    3260  -  Is   09:55        0:00.05 /usr/lo
georgiou 78216   0.0  0.0   10336    5728  -  D    09:58        0:00.00 imap: [
slumdg10 78405   0.0  0.0    9300    4680  -  D    09:36        0:00.00 pop3: [
cieltec5 78786   0.0  0.0    9320    4968  -  D    09:45        0:00.00 pop3: [
slumdg10 79527   0.0  0.0    9300    4680  -  D    10:02        0:00.00 pop3: [
root     79645   0.0  0.0    6676    2592  -  D    09:50        0:00.00 grep -q
slumdg10 79675   0.0  0.0    9308    4836  -  D    09:50        0:00.00 pop3: [
root     79844   0.0  0.0    6464    1256  -  DsJ  18Sep18      0:04.70 /usr/sb
valiraki 80017   0.0  0.9 2145140  301524  -  S    08:30        0:00.19 php-fpm
slumdg10 80462   0.0  0.0    9320    4736  -  D    09:59        0:00.00 pop3: [
elcampo  80641   0.0  0.0    9300    4948  -  D    09:40        0:00.02 pop3: [
costas   80671   0.0  0.0   10368    5752  -  D    09:59        0:00.00 imap: [
root     81075   0.0  0.0    6676    2580  -  D    10:10        0:00.00 grep -q
valiraki 81210   0.0  0.0    9320    4708  -  D    09:47        0:00.00 pop3: [
slumdg10 81219   0.0  0.0    9308    4836  -  D    09:40        0:00.00 pop3: [
slumdg10 81314   0.0  0.0   10336    5564  -  D    09:48        0:00.00 imap: [
slumdg10 81856   0.0  0.0    9300    4680  -  D    09:39        0:00.00 pop3: [
valiraki 82367   0.0  0.0    9300    4684  -  D    09:51        0:00.00 pop3: [
slumdg17 82461   0.0  0.0   10336    5564  -  D    09:40        0:00.01 imap: [
slumdg10 82649   0.0  0.0    9320    4928  -  D    09:34        0:00.00 pop3: [
apollon  82987   0.0  0.0   10336    5640  -  D    09:37        0:00.01 imap: [
planet   83036   0.0  1.0 2170424  342228  -  D    09:32        0:04.05 php-fpm
georgiou 89331   0.0  0.0   10980    6360  -  D    09:58        0:00.01 imap: [
apollon  89408   0.0  0.0   10348    5660  -  D    09:37        0:00.01 imap: [
valiraki 89652   0.0  0.0    9320    4708  -  D    10:03        0:00.00 pop3: [
georgiou 89711   0.0  0.0   10336    5948  -  D    09:57        0:00.01 imap: [
slumdg10 89761   0.0  0.0   13572    9088  -  D    09:44        0:00.02 lmtp: [
root     89912   0.0  0.6 2139336  187344  -  DsJ  Tue15        1:01.51 php-fpm
slumdg17 90450   0.0  0.0   13564    9004  -  D    09:56        0:00.01 lmtp: [
valiraki 90485   0.0  0.0   10424    5892  -  D    09:48        0:00.00 imap: [
slumdg10 90489   0.0  0.0    9320    5028  -  D    09:50        0:00.01 pop3: [
slumdg10 90624   0.0  0.0    9300    4680  -  D    09:41        0:00.00 pop3: [
root     90629   0.0  0.0    6468    2484  -  I    10:10        0:00.00 cron: r
costas   91017   0.0  0.0    9316    5008  -  D    09:50        0:00.01 pop3: [
root     91161   0.0  0.0       0      16  -  DL   18Sep18      0:00.72 [accoun
valiraki 92072   0.0  0.0    9332    4712  -  D    09:51        0:00.00 pop3: [
root     92210   0.0  0.6 2138304  186496  -  DJ   09:33        0:00.00 php-fpm
nobody   92331   0.0  0.0   32492   11552  -  I    Wed12        0:00.49  postfw
georgiou 92399   0.0  0.0   10336    5592  -  D    09:52        0:00.00 imap: [
surfaces 92682   0.0  0.0    9312    4968  -  D    09:42        0:00.00 pop3: [
pccdkeys 92686   0.0  0.8 2179396  276200  -  DJ   09:33        0:00.47 php-fpm
root     92809   0.0  0.0   13164    8300  -  Is   09:34        0:00.01 sshd: c
slumdg10 92953   0.0  0.0    9320    4928  -  D    09:41        0:00.00 pop3: [
valiraki 92978   0.0  0.0    9328    4708  -  D    10:06        0:00.00 pop3: [
planet   93053   0.0  1.1 2195512  368528  -  D    09:26        0:24.99 php-fpm
milas    93435   0.0  0.0    9324    4708  -  D    09:57        0:00.00 pop3: [
slumdg10 94054   0.0  0.0   13584    9108  -  D    09:55        0:00.01 lmtp: [
georgiou 94081   0.0  0.0   10336    5604  -  D    10:05        0:00.00 imap: [
osanet   94530   0.0  0.0    9308    4844  -  D    10:05        0:00.01 pop3: [
apollon  95026   0.0  0.0   10348    5656  -  D    09:37        0:00.01 imap: [
slumdg17 95027   0.0  0.0    9300    4656  -  D    09:56        0:00.00 pop3: [
traktoor 95474   0.0  0.0   10344    5604  -  D    09:39        0:00.00 imap: [
costas   95568   0.0  0.0   10352    5656  -  D    09:53        0:00.01 imap: [
slumdg10 95748   0.0  0.0    9332    4960  -  D    09:47        0:00.00 pop3: [
slumdg10 96442   0.0  0.0   13584    9108  -  D    09:44        0:00.02 lmtp: [
slumdg10 96654   0.0  0.0    9300    4680  -  D    09:40        0:00.00 pop3: [
slumdg10 97302   0.0  0.0    9320    4928  -  D    09:40        0:00.00 pop3: [
planet   97733   0.0  1.1 2193144  367164  -  D    09:26        0:24.45 php-fpm
root     98097   0.0  0.0    6676    2580  -  D    09:35        0:00.00 grep -q
root     98162   0.0  0.0    7812    3340  -  Is   09:40        0:00.05 /usr/lo
slumdg10 98423   0.0  0.0    9308    4816  -  D    09:58        0:00.01 pop3: [
apollon  98527   0.0  0.0   10336    5608  -  D    09:37        0:00.00 imap: [
root     98951   0.0  0.0    8492    2832  -  I    18Sep18      0:00.61 /usr/lo
valiraki 99162   0.0  0.0   10336    5636  -  D    10:01        0:00.01 imap: [
milas    99389   0.0  0.0    9324    4708  -  D    09:59        0:00.00 pop3: [
root     50418   0.0  0.0    6996    3496 v0  R+   10:15        0:00.00 ps -aux
root     90389   0.0  0.0    6956    3100 v0  Is   18Sep18      0:00.01 login [
root     92268   0.0  0.0    7492    4488 v0  S    10:06        0:00.11 -csh (c
root     97507   0.0  0.0    6412    1832 v1  Is+  18Sep18      0:00.00 /usr/li
root       548   0.0  0.0    6412    1832 v2  Is+  18Sep18      0:00.00 /usr/li
root      5324   0.0  0.0    6412    1832 v3  Is+  18Sep18      0:00.00 /usr/li
root      5859   0.0  0.0    6412    1832 v4  Is+  18Sep18      0:00.00 /usr/li
root     12110   0.0  0.0    6412    1832 v5  Is+  18Sep18      0:00.00 /usr/li
root     14102   0.0  0.0    6412    1832 v6  Is+  18Sep18      0:00.00 /usr/li
root     19279   0.0  0.0    6412    1832 v7  Is+  18Sep18      0:00.00 /usr/li
Here is the only error message in /var/log/messages but happens after the issue:

Code:
kernel: sonewconn: pcb 0xfffff805a1088cb0: Listen queue overflow: 3073 already in queue awaiting acceptance (67 occurrences)
kernel: sonewconn: pcb 0xfffff805a1088cb0: Listen queue overflow: 3073 already in queue awaiting acceptance (42 occurrences)
kernel: sonewconn: pcb 0xfffff805a1088cb0: Listen queue overflow: 3073 already in queue awaiting acceptance (50 occurrences)
kernel: sonewconn: pcb 0xfffff805a1088cb0: Listen queue overflow: 3073 already in queue awaiting acceptance (42 occurrences)
kernel: sonewconn: pcb 0xfffff805a1088cb0: Listen queue overflow: 3073 already in queue awaiting acceptance (49 occurrences)
The last changes made on these servers for main OS:

Code:
Upgrade of pkgconf-1.4.2,1 to pkgconf-1.5.3,1
Upgrade of dovecot-2.3.2.1_1 to dovecot-2.3.3
Upgrade of bison-3.0.5,1 to bison-3.1,1
Upgrade of dovecot-pigeonhole-0.5.2_3 to dovecot-pigeonhole-0.5.3
Upgrade of vim-8.1.0342 to vim-8.1.0443
Upgrade of sshguard-2.1.0_1 to sshguard-2.2.0
Upgrade of git-lite-2.19.0 to git-lite-2.19.0_1
Upgrade of help2man-1.47.6 to help2man-1.47.7
Upgrade of nano-3.0_1 to nano-3.1
Upgrade of nginx-1.14.0_11,2 to nginx-1.14.0_12,2
Upgrade of liblz4-1.8.2,1 to liblz4-1.8.3,1
Upgrade of oniguruma-6.8.2 to oniguruma-6.9.0
Upgrade of sqlite3-3.24.0_1 to sqlite3-3.25.1
Upgrade of p5-Net-DNS-1.17,1 to p5-Net-DNS-1.18,1
Upgrade of bind911-9.11.4P1_2 to bind911-9.11.4P2
And for a JAIL I run in /home/jail/php56 :

Code:
Upgrade of php56-5.6.37 to php56-5.6.38
Upgrade of php56-bcmath-5.6.37 to php56-bcmath-5.6.38
Upgrade of php56-bz2-5.6.37 to php56-bz2-5.6.38
Upgrade of php56-calendar-5.6.37 to php56-calendar-5.6.38
Upgrade of php56-ctype-5.6.37 to php56-ctype-5.6.38
Upgrade of php56-curl-5.6.37 to php56-curl-5.6.38
Upgrade of php56-dom-5.6.37 to php56-dom-5.6.38
Upgrade of php56-exif-5.6.37 to php56-exif-5.6.38
Upgrade of php56-fileinfo-5.6.37 to php56-fileinfo-5.6.38
Upgrade of php56-filter-5.6.37 to php56-filter-5.6.38
Upgrade of php56-ftp-5.6.37 to php56-ftp-5.6.38
Upgrade of php56-gd-5.6.37 to php56-gd-5.6.38
Upgrade of php56-gettext-5.6.37 to php56-gettext-5.6.38
Upgrade of php56-hash-5.6.37 to php56-hash-5.6.38
Upgrade of php56-iconv-5.6.37 to php56-iconv-5.6.38
Upgrade of php56-imap-5.6.37 to php56-imap-5.6.38
Upgrade of php56-json-5.6.37 to php56-json-5.6.38
Upgrade of php56-mbstring-5.6.37 to php56-mbstring-5.6.38
Upgrade of php56-mcrypt-5.6.37 to php56-mcrypt-5.6.38
Upgrade of php56-mysql-5.6.37 to php56-mysql-5.6.38
Upgrade of php56-mysqli-5.6.37 to php56-mysqli-5.6.38
Upgrade of php56-opcache-5.6.37 to php56-opcache-5.6.38
Upgrade of php56-openssl-5.6.37 to php56-openssl-5.6.38
Upgrade of php56-pdo-5.6.37 to php56-pdo-5.6.38
Upgrade of php56-pdo_mysql-5.6.37 to php56-pdo_mysql-5.6.38
Upgrade of php56-pdo_sqlite-5.6.37 to php56-pdo_sqlite-5.6.38
Upgrade of php56-phar-5.6.37 to php56-phar-5.6.38
Upgrade of php56-posix-5.6.37 to php56-posix-5.6.38
Upgrade of php56-session-5.6.37 to php56-session-5.6.38
Upgrade of php56-simplexml-5.6.37 to php56-simplexml-5.6.38
Upgrade of php56-soap-5.6.37 to php56-soap-5.6.38
Upgrade of php56-sockets-5.6.37 to php56-sockets-5.6.38
Upgrade of php56-sqlite3-5.6.37 to php56-sqlite3-5.6.38
Upgrade of php56-tidy-5.6.37 to php56-tidy-5.6.38
Upgrade of php56-tokenizer-5.6.37 to php56-tokenizer-5.6.38
Upgrade of php56-xml-5.6.37 to php56-xml-5.6.38
Upgrade of php56-xmlreader-5.6.37 to php56-xmlreader-5.6.38
Upgrade of php56-xmlrpc-5.6.37 to php56-xmlrpc-5.6.38
Upgrade of php56-xmlwriter-5.6.37 to php56-xmlwriter-5.6.38
Upgrade of php56-xsl-5.6.37 to php56-xsl-5.6.38
Upgrade of php56-zip-5.6.37 to php56-zip-5.6.38
Upgrade of php56-zlib-5.6.37 to php56-zlib-5.6.38
Upgrade of oniguruma-6.8.2 to oniguruma-6.9.0
Upgrade of nano-3.0_1 to nano-3.1
Upgrade of sqlite3-3.24.0_1 to sqlite3-3.25.1
Also few days ago I upgrade from FreeBSD-11.2-RELEASE-p2 to FreeBSD-11.2-RELEASE-p3 (maybe something wrong with these patches?)

Any idea how to troubleshoot the issue?
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Reaction score: 20
Messages: 242

The same or similar issue exist with FreeBSD 12.0. The process that goes to D-State is Nginx.

Any idea how to troubleshoot it?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,060
Messages: 31,663

D state is "waiting for disk", so I'd check things like the disks themselves, controllers and such. Maybe run smartctl(8) on each disk.

If your disks are hanging up the system you won't find anything in the logs either, since they're written to disk they end up waiting too and never manage to actually write the log file. In this case it might prove useful to configure a remote syslog server.
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Reaction score: 20
Messages: 242

SMART shows that disks are fine. It happens in different servers randomly which have different disk models so I don't believe it's related to bad disks.

I was logged in using SSH before this happened and I was able to shutdown other services. I also tried to connect from another terminal using SSH but it didn't allow me maybe because the user homedir is in /home

I was able to unmount /home2 (same disks)

Then when I tried to "shutdown -r now" it goes to single user mode because it took a lot of time to unmount /home (Nginx serves files from /home).
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,060
Messages: 31,663

It happens in different servers randomly which have different disk models so I don't believe it's related to bad disks.
Same or similar controllers perhaps? Any NFS usage? That could potentially also hangup the whole filesystem.
 
OP
OP
CyberCr33p

CyberCr33p

Active Member

Reaction score: 20
Messages: 242

I don't use NFS. Someone on IRC told me to try to disable sendfile for Nginx to see if it helps.
 

PMc

Aspiring Daemon

Reaction score: 184
Messages: 516

If an application triggers that, on multiple systems, then one might consider it an OS flaw. This should be a matter from the device driver downwards to the hardware.
If it appears to happen on multiple systems, I would look for similarities in the hardware. The controllers, as SirDice mentioned. The Bios configuration, something that might lead to improper PCI timings, memory chips not certified to the board and/or run at suboptimal parameters, ... things like that.
It may happen rarely, and it usually starts with one process blocking in "D", but then all others going to the same filesystem/disk/controller will join in.
 
Top