Services not starting on boot | pid files: not readable.

Hi,
I have a big problem and I don't know how to fix it.
sshd, cron and jails services not starting on boot.
I can run all services by command line:

Code:
service cron start
/etc/rc.d/cron: DEBUG: checkyesno: cron_dst is set to YES.
/etc/rc.d/cron: DEBUG: pid file (/var/run/cron.pid): not readable.
/etc/rc.d/cron: DEBUG: checkyesno: cron_enable is set to YES.
Starting cron.
/etc/rc.d/cron: DEBUG: run_rc_command: doit:  limits -C daemon  /usr/sbin/cron  -s

How to resolve it ?


Code:
service sshd status
/etc/rc.d/sshd: DEBUG: pid file (/var/run/sshd.pid): not readable.
/etc/rc.d/sshd: DEBUG: checkyesno: sshd_enable is set to YES.
sshd is not running.

Code:
service cron status
/etc/rc.d/cron: DEBUG: checkyesno: cron_dst is set to YES.
/etc/rc.d/cron: DEBUG: pid file (/var/run/cron.pid): not readable.
/etc/rc.d/cron: DEBUG: checkyesno: cron_enable is set to YES.
cron is not running.

before run the services manually pid files are not exists...

Code:
root@host:/var/run # ls
ConsoleKit          devd.pid            ld-elf.so.hints     os-release          syslogd.sockets     xrdp-sesman.pid
clean_var           devd.pipe           ld-elf32.so.hints   ppp                 tpm                 xrdp.pid
consolekit.pid      devd.seqpacket.pipe libvirt             samba4              user
cups                dhclient            log                 sudo                utx.active
dbus                dmesg.boot          logpriv             syslog.pid          wpa_supplicant

Code:
root@host:~ # service -e
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/hostid_save
/etc/rc.d/zfsbe
/etc/rc.d/zfs
/etc/rc.d/cleanvar
/etc/rc.d/kldxref
/etc/rc.d/devmatch
/etc/rc.d/ip6addrctl
/etc/rc.d/netif
/etc/rc.d/devd
/etc/rc.d/resolv
/usr/local/etc/rc.d/vboxnet
/etc/rc.d/newsyslog
/etc/rc.d/syslogd
/etc/rc.d/ntpdate
/etc/rc.d/savecore
/etc/rc.d/dmesg
/etc/rc.d/virecover
/etc/rc.d/motd
/etc/rc.d/ntpd
/etc/rc.d/rctl
/usr/local/etc/rc.d/dbus
/usr/local/etc/rc.d/samba_server
/usr/local/etc/rc.d/xrdp-sesman
/usr/local/etc/rc.d/xrdp
/usr/local/etc/rc.d/libvirtd
/etc/rc.d/sshd
/etc/rc.d/sendmail
/etc/rc.d/cron
/etc/rc.d/jail
/etc/rc.d/moused
/etc/rc.d/mixer
/etc/rc.d/gptboot
/etc/rc.d/bgfsck
 
Considering the fact that all / most services complain about a pid file in /var/run not being readable I'd start there. Does /var/run even exist? Does it have to be mounted and maybe it's not? Or maybe it's readwrite?

With these things always start looking for commonly occurring factors and start your way from there.
 
Considering the fact that all / most services complain about a pid file in /var/run not being readable I'd start there. Does /var/run even exist? Does it have to be mounted and maybe it's not? Or maybe it's readwrite?

With these things always start looking for commonly occurring factors and start your way from there.
var/run exist (after start services manually the pid files have been created and services working)

Code:
root@host:/var # ls -l
total 68
drwxr-x---   2 root     wheel     2 Oct 23 07:30 account
drwxr-xr-x   4 root     wheel     4 Oct 23 07:30 at
drwxr-x---   4 root     audit     4 Jan 25 16:30 audit
drwxrwx---   2 root     authpf    2 Oct 23 07:30 authpf
drwxr-x---   2 root     wheel    15 Feb 28 03:04 backups
drwxr-xr-x   5 root     wheel     5 Jan 26 11:15 cache
drwxr-x---   2 root     wheel     3 Jan 25 16:30 crash
drwxr-x---   3 root     wheel     3 Oct 23 07:30 cron
drwxr-xr-x  16 root     wheel    21 Jan 26 13:24 db
dr-xr-xr-x   2 root     wheel     2 Oct 23 07:30 empty
drwxrwxr-x   2 root     games     2 Oct 23 07:30 games
drwx------   2 root     wheel     2 Oct 23 07:30 heimdal
drwxr-xr-x   7 root     wheel     7 Jan 26 11:15 lib
drwxr-xr-x   6 root     wheel    53 Feb 28 03:02 log
drwxrwxr-x   2 root     mail     13 Feb 28 03:04 mail
drwxr-xr-x   2 daemon   wheel     3 Jan 25 16:37 msgs
drwxr-xr-x   2 root     wheel     2 Oct 23 07:30 preserve
drwxr-xr-x  13 root     wheel    33 Feb 28 03:04 run
drwxrwxr-x   2 root     daemon    2 Oct 23 07:30 rwho
drwxr-xr-x  10 root     wheel    10 Jan 25 21:26 spool
drwxrwxrwt   3 root     wheel     3 Feb 27 14:00 tmp
drwxr-xr-x   3 unbound  unbound   3 Oct 23 07:30 unbound
drwxr-xr-x   2 root     wheel     4 Oct 23 07:38 yp

Code:
root@host:/var/run # ls -l
total 52
drwxr-xr-x  2 root  wheel        3 Feb 27 13:46 ConsoleKit
-rw-r--r--  1 root  wheel        0 Feb 27 14:45 clean_var
-rw-r--r--  1 root  wheel        5 Feb 27 13:46 consolekit.pid
[B]-rw-------  1 root  wheel        4 Feb 27 14:16 cron.pid[/B]
drwxr-xr-x  3 root  wheel        3 Jan 25 21:26 cups
drwxr-xr-x  2 root  wheel        4 Feb 27 13:45 dbus
-rw-------  1 root  wheel        4 Feb 27 13:45 devd.pid
srw-rw-rw-  1 root  wheel        0 Feb 27 13:45 devd.pipe
srw-rw-rw-  1 root  wheel        0 Feb 27 13:45 devd.seqpacket.pipe
drwxr-xr-x  2 root  wheel        2 Oct 23 07:30 dhclient
-rw-r--r--  1 root  wheel    20692 Feb 27 13:45 dmesg.boot
-rw-r--r--  1 root  wheel        2 Feb 27 13:52 jail_freeradius.id
-rw-r--r--  1 root  wheel        2 Feb 27 13:52 jail_projekty.id
-r--r--r--  1 root  wheel      338 Feb 27 14:45 ld-elf.so.hints
-r--r--r--  1 root  wheel      139 Feb 27 14:45 ld-elf32.so.hints
drwxr-xr-x  7 root  wheel        7 Feb 26 10:16 libvirt
srw-rw-rw-  1 root  wheel        0 Feb 27 13:45 log
srw-------  1 root  wheel        0 Feb 27 13:45 logpriv
-r--r--r--  1 root  wheel      228 Feb 27 13:45 os-release
drwxrwx---  2 root  network      2 Oct 23 07:30 ppp
drwxr-xr-x  5 root  wheel        8 Feb 27 13:45 samba4
-rw-r--r--  1 root  wheel        5 Feb 27 13:52 sshd.pid
drwx--x--x  2 root  wheel        2 Jan 14 02:12 sudo
-rw-------  1 root  wheel        4 Feb 27 13:45 syslog.pid
-rw-r--r--  1 root  wheel        0 Feb 27 13:45 syslogd.sockets
drwx------  2 root  _tss         2 Jan  5 07:07 tpm
drwxr-xr-x  3 root  wheel        3 Jan 26 10:46 user
-rw-r--r--  1 root  wheel      591 Feb 28 13:28 utx.active
drwxr-xr-x  2 root  wheel        2 Oct 23 07:30 wpa_supplicant
-rw-------  1 root  wheel        4 Feb 27 13:45 xrdp-sesman.pid
-rw-------  1 root  wheel        4 Feb 27 13:45 xrdp.pid
 
Code:
Performing sanity check on Samba configuration: OK
/usr/local/etc/rc.d/samba_server: DEBUG: run_rc_command: doit: samba_server_cmd 
/usr/local/etc/rc.d/samba_server: DEBUG: checkyesno: nmbd_enable is set to YES.
/usr/local/etc/rc.d/samba_server: DEBUG: pid file (/var/run/samba4/nmbd.pid): not readable.
/usr/local/etc/rc.d/samba_server: DEBUG: checkyesno: nmbd_enable is set to YES.
Starting nmbd.
/usr/local/etc/rc.d/samba_server: DEBUG: run_rc_command: doit:  limits -C daemon  /usr/local/sbin/nmbd --daemon --configfile=/usr/local/etc/smb4.conf
/usr/local/etc/rc.d/samba_server: DEBUG: checkyesno: smbd_enable is set to YES.
/usr/local/etc/rc.d/samba_server: DEBUG: pid file (/var/run/samba4/smbd.pid): not readable.
/usr/local/etc/rc.d/samba_server: DEBUG: checkyesno: smbd_enable is set to YES.
Starting smbd.
/usr/local/etc/rc.d/samba_server: DEBUG: run_rc_command: doit:  limits -C daemon  /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
/usr/local/etc/rc.d/samba_server: DEBUG: checkyesno: winbindd_enable is set to YES.
/usr/local/etc/rc.d/samba_server: DEBUG: pid file (/var/run/samba4/winbindd.pid): not readable.
/usr/local/etc/rc.d/samba_server: DEBUG: checkyesno: winbindd_enable is set to YES.
Starting winbindd.
/usr/local/etc/rc.d/samba_server: DEBUG: run_rc_command: doit:  limits -C daemon  /usr/local/sbin/winbindd --daemon --configfile=/usr/local/etc/smb4.conf
 
Code:
root@host:~ # ls -ld /var /var/run
drwxr-xr-x  25 root  wheel  25 Feb 28 15:38 /var
drwxr-xr-x  13 root  wheel  33 Feb 28 14:40 /var/run
 
That all looks normal: permissions on the directories themselves, and the content.

Could it be that your root account has become corrupted? Maybe it got changed so its uid/gid is not zero?
Maybe the file system that /var/run is mounted read-only? That doesn't match the error messages though; they say "not readable", not "not writable".

Bizarre.
 
That all looks normal: permissions on the directories themselves, and the content.

Could it be that your root account has become corrupted? Maybe it got changed so its uid/gid is not zero?
Maybe the file system that /var/run is mounted read-only? That doesn't match the error messages though; they say "not readable", not "not writable".

Bizarre.
I'm trying all day to find the problem and nothing.
How to check if the root acoount is corrupted ?
How to check if /var/run is mounted read-only ?
 
Look at /etc/passwd and /etc/group. Log in as root, use the command "id".
Look at the output of mount. Try to create a file in /var/run, try to read the files that are there.
 
This error comes from /etc/rc.subr and means that the function check_pidfile() fails to find the pid file after launching the process. The pid just doesn't exist, but why?
 
Please share the output of this command: mount | grep -e var -e run. I can't help wonder what filesystem is involved.
Code:
root@host:~ # mount | grep -e var -e run
zroot/recv-7498-4/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/recv-7498-4/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/recv-7498-4/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/recv-7498-4/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/recv-7498-4/mail on /var/mail (zfs, local, nfsv4acls)

I tried to restore from snapshot all files and now everything has wrong path - it's possible to remount it ?
 
I tried to restore from snapshot all files and now everything is wrong - it's possible to remount it ?
What does "everything is wrong" mean? What are the exact symptoms? When you say "remount", do you mean "undo restore from snapshot"? As far as I know, it is not possible to undo a restore from snapshot.

Let me go back to the beginning, and instead of looking at countless details (which don't seem to be helping), let me ask you a question: What did you do between the last time the system worked, and the first time it didn't? What configuration could that operation have changed?

And please observe the following: I'm not trying to solve the problem for you. I'm just showing you tools and techniques for gathering data to help you solve the problem yourself.
 
Ok.
30 minutes ago I did zfs send and receive from all system snapshot.
After that I have this:

Code:
root@host:~ # mount | grep -e var -e run
zroot/recv-7498-4/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/recv-7498-4/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/recv-7498-4/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/recv-7498-4/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/recv-7498-4/mail on /var/mail (zfs, local, nfsv4acls)

I had to restart server on Friday and after that I saw the problem with pid files.
Before restart I think on Wednesday I did freebsd-update fetch and freebsd-update install
 
Judging by that output I'd say it looks as if OP somehow managed to mount a snapshot of some sort on top of the main system.

Code:
zroot/recv-7498-4/log on /var/log
This does not look very healthy to me and it could definitely explain a thing or two, snapshots are RO afterall. The only thing I fail to understand is how you could create such a mess.

At this point I would reboot the box in single user mode and check what happens then. Ensure that the normal root partition gets mounted and accessed and not some recv-7498 mutation.
 
Judging by that output I'd say it looks as if OP somehow managed to mount a snapshot of some sort on top of the main system.
Thank you for noticing this! I feel embarrassed that it didn't catch my attention. A normal ZFS mount looks like this: "home on /home (zfs, NFS exported, local, nfsv4acls)"
 
Back
Top