Solved No servers are running but not... some

spag

spag

Hi I have a strange problem across all jails. After latest updates ( freebsd-update) most of the jails are not able to start any services. The only allowed ones are on port 80,8080,22,81,25,514,5666 ( no matter if this is apache, node or nginx ). Other servers (in jails) just show they are running but there is no PID and there is nothing listening in socks. Maybe someone have some clue how to fix this.

Code: 
man002# uname -a
FreeBSD man002 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

Code: 
root@machinex:~ # ps -aux
USER    PID %CPU %MEM    VSZ    RSS TT  STAT STARTED    TIME COMMAND
root   3751  0.0  0.0  14524   1492  -  SsJ   3:25PM 0:00.00 /usr/sbin/syslogd -s
root   3778  0.0  0.0  25580   3868  -  IsJ   3:25PM 0:00.00 /usr/local/libexec/vsftpd
root   3806  0.0  0.2 305316  29624  -  SsJ   3:25PM 0:00.02 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
www    3807  0.0  0.2 305316  29632  -  IJ    3:25PM 0:00.00 php-fpm: pool PYDIO (php-fpm)
www    3808  0.0  0.2 305316  29632  -  IJ    3:25PM 0:00.00 php-fpm: pool PYDIO (php-fpm)
www    3809  0.0  0.2 305316  29632  -  IJ    3:25PM 0:00.00 php-fpm: pool PYDIO (php-fpm)
www    3810  0.0  0.2 305316  29632  -  IJ    3:25PM 0:00.00 php-fpm: pool PYDIO (php-fpm)
nagios 3814  0.0  0.0  25512   4056  -  IsJ   3:25PM 0:00.01 /usr/local/sbin/nrpe2 -c /usr/local/etc/nrpe.cfg -d
root   3818  0.0  0.1 111148  13572  -  IsJ   3:25PM 0:00.00 nginx: master process /usr/local/sbin/nginx
www    3819  0.0  0.1 111148  13616  -  INJ   3:25PM 0:00.00 nginx: worker process (nginx)
mysql  3827  0.0  0.0  17080   1780  -  IsJ   3:25PM 0:00.01 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/machine.pid
mysql  3996  0.0  2.7 646788 453440  -  IJ    3:25PM 0:00.26 /usr/local/libexec/mysqld --defaults-extra-file=/var/db/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --plugin-dir=/usr/local/lib/mysql/plugin --user=mysql --log-e
nobody 4030  0.0  0.0  21316   2440  -  IsJ   3:25PM 0:00.00 /usr/local/bin/memcached -d -m 2048 -l 127.0.0.1 -p 11211 -d -u nobody -P /var/run/memcached/memcached.pid
root   4033  0.0  0.0  31332   2448  -  SsJ   3:25PM 0:00.01 /usr/local/sbin/htcacheclean -p/var/cache/apache3 -d180 -l128M -t -n -i
root   4052  0.0  0.0  61200   3860  -  IsJ   3:25PM 0:00.00 /usr/sbin/sshd
root   4062  0.0  0.2 397396  33488  -  SsJ   3:25PM 0:00.12 /usr/local/sbin/httpd -DNOHTTPACCEPT
www    4071  0.0  0.1 129836   9420  -  IJ    3:25PM 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root   4074  0.0  0.0  16612   1636  -  SsJ   3:25PM 0:00.00 /usr/sbin/cron -s
www    4077  0.0  0.2 397396  35832  -  IJ    3:25PM 0:00.03 /usr/local/sbin/httpd -DNOHTTPACCEPT
www    4078  0.0  0.2 397396  33500  -  IJ    3:25PM 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
www    4079  0.0  0.2 397396  33500  -  IJ    3:25PM 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
www    4080  0.0  0.2 397396  33500  -  IJ    3:25PM 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
www    4082  0.0  0.2 397396  33500  -  IJ    3:25PM 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
www    4161  0.0  0.2 397396  34784  -  IJ    3:27PM 0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
nagios 4780  0.0  0.0  25512   4120  -  IJ    3:55PM 0:00.02 /usr/local/sbin/nrpe2 -c /usr/local/etc/nrpe.cfg -d
nagios 4781  0.0  0.0  25512   4132  -  IJ    3:55PM 0:00.00 /usr/local/sbin/nrpe2 -c /usr/local/etc/nrpe.cfg -d
nagios 4782  0.0  0.0  16792   1792  -  IJ    3:55PM 0:00.00 /usr/local/libexec/nagios/check_disk -w 20% -c 10% -p /
root   4795  0.0  0.0  47748   2104  2  SJ    3:56PM 0:00.01 login [pam] (login)
root   4805  0.0  0.0  23584   3472  2  SJ    3:56PM 0:00.02 -csh (csh)
root   4807  0.0  0.0  16680   1724  2  R+J   3:56PM 0:00.00 ps -aux

Code: 
root@machine:~ # sockstat
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     login      4795  3  dgram  -> /var/run/logpriv
www      httpd      4161  3  tcp4   10.123.123.123:8080   *:*
www      httpd      4082  3  tcp4   10.123.123.123:8080   *:*
www      httpd      4080  3  tcp4   10.123.123.123:8080   *:*
www      httpd      4079  3  tcp4   10.123.123.123:8080   *:*
www      httpd      4078  3  tcp4   10.123.123.123:8080   *:*
www      httpd      4077  3  tcp4   10.123.123.123:8080   *:*
root     cron       4074  4  dgram  -> /var/run/logpriv
www      httpd      4071  3  stream /var/run/cgisock.4062
root     httpd      4062  3  tcp4   10.123.123.123:8080   *:*
root     sshd       4052  3  tcp4   10.123.123.123:22     *:*
www      nginx      3819  9  tcp4   10.123.123.123:80     *:*
www      nginx      3819  10 tcp4   10.123.123.123:81     *:*
www      nginx      3819  11 stream -> ??
root     nginx      3818  3  stream -> ??
root     nginx      3818  9  tcp4   10.123.123.123:80     *:*
root     nginx      3818  10 tcp4   10.123.123.123:81     *:*
root     nginx      3818  11 stream -> ??
nagios   nrpe2      3814  3  dgram  -> /var/run/logpriv
nagios   nrpe2      3814  4  tcp4   10.123.123.123:5666   *:*
www      php-fpm    3810  0  stream /var/run/php-fpm.socket
www      php-fpm    3809  0  stream /var/run/php-fpm.socket
www      php-fpm    3808  0  stream /var/run/php-fpm.socket
www      php-fpm    3807  0  stream /var/run/php-fpm.socket
root     php-fpm    3806  5  stream -> ??
root     php-fpm    3806  7  stream -> ??
root     php-fpm    3806  8  stream /var/run/php-fpm.socket
root     vsftpd     3778  3  tcp4   10.123.123.123:21     *:*
root     syslogd    3751  4  dgram  /var/run/log
root     syslogd    3751  5  dgram  /var/run/logpriv
root     syslogd    3751  6  udp4   10.123.123.123:514    *:*


And now failed services: memcached port 11211, 3306, 27017(mongodb)
Code: 
root@machine:~ # telnet localhost 11211
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Trying ::1...
telnet: socket: Protocol not supported
root@machine:~ # cat /var/run/memcached/memcached.pid
cat: /var/run/memcached/memcached.pid: No such file or directory
root@machine:~ # telnet localhost 3306
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Trying ::1...
telnet: socket: Protocol not supported
root@machine:~ # cat /var/db/mysql/machine.pid
cat: /var/db/mysql/machine.pid: No such file or directory

There is nothing inside log files that says it is wrong.I found also CURL command is not working. Just hanging without any action. Pings are working normally.

Looks like to me this is some kind of main box configuration thing (socks etc.) but I cannot find any clue where to even look.

Any help would be much appreciated, thanks.
 
Keep in mind that jails do not have a lo0 and therefor no 127.0.0.1 address. Also note that, unless configured, pings from inside a jail do not work either.
 
Changing services to Jail IP address did not help. Pings are working fine from jails. PF conf is disabled.

Did solve this problem by doing full system re-installation. It was faster than tracking this issue. Thanks.
 
Faster, but you didn't learn what the problem was. If this continues, you will end up doing a full reinstall even when it is not the fastest option. Just sayin' ...
 
Hi,
I upgraded from 10.1 to 10.2 and ran into the same issue

uname -a
Code: 
FreeBSD nas11 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015 
root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

I have problems with samba, mysql-server and even java itself

java -version returns nothing. Just hangs. I can exit with ctrl+c
MySQL never creates a PID or socket

I re-installed all packages -> same issues
I tried to restore a jail from backup -> same issues
I created a new jail -> same issues

Some services like Netatalk are running fine.

Actually I don't know if this is even a network issue or software issue.
Any help appreciated otherwise I will re-install from backup since I am out of ideas.
 
At least the Samba problem seem to be related to Python, but why Java and MySQL server are not starting any more... I don’t know.
 
Had the same issue on few other machines too - temporary fix that works was bringing back basejail - not sure about stability of whole system so all machines will be reinstalled as we did not find any faster solution to this.
 
Dr.Moe said:
Hi,
I upgraded from 10.1 to 10.2 and ran into the same issue

java -version returns nothing. Just hangs. I can exit with ctrl+c
MySQL never creates a PID or socket
Click to expand...

Hi, I just encountered the very same issue with java (yet not with postgres or nginx) on one of my jail. Not immediately after the upgrade to 10.2 but after a minor upgrade.

Did you solved it?

For further details: Thread openjdk8-in-kernel-lock-umtxn-state-after-upgrade.54865/

(edited link to thread)
 
You must log in or register to reply here.
Back
Top