Solved Errors with VNET jail ( failed to unlink nh#1 )

A

avner

I have 3 vnet jails on my system all cloned from each other. After trying to install postgreSQL and Orthanc in the 3rd jail, named orthanc, I am no longer able to start this jail. I wonder if one of the pg_hba.conf settings is contributing to the problem as I was editing that file just before the problem developed ( not immediately but hours later ). I had been playing with the settings to get orthanc to be able to communicate with postgreSQL via a socket (IPC) but not sure that is even possible. Now just looking for help to get my jail restarted.

Code: 
root@daria:/var/log # jail -f "/etc/jail.conf.d/orthanc.conf" -c orthanc
jail -f "/etc/jail.conf.d/orthanc.conf" -c orthanc
orthanc: created
jail: orthanc: /bin/sh /etc/rc: failed
orthanc: removed
root@daria:/var/log #


Output of messages:
Code: 
root@daria:/var/log # tail messages
Nov 19 08:14:19 daria kernel: [nhop_ctl] inet.0 nhop_free: failed to unlink nh#1/inet/ng0_orthanc/resolve
Nov 19 08:14:19 daria kernel: [nhop_ctl] inet.0 nhop_free: failed to unlink nh#2/inet/ng0_orthanc/192.168.15.1
Nov 19 08:18:04 daria su[42107]: abner to root on /dev/pts/2
Nov 19 08:18:51 daria su[43881]: abner to root on /dev/pts/3
Nov 19 08:23:32 daria kernel: [nhop_ctl] inet.0 nhop_free: failed to unlink nh#1/inet/ng0_orthanc/resolve
Nov 19 08:23:32 daria kernel: [nhop_ctl] inet.0 nhop_free: failed to unlink nh#4/inet/ng0_orthanc/192.168.15.1
Nov 19 08:32:07 daria su[81312]: abner to root on /dev/pts/4
Nov 19 08:34:29 daria su[99650]: abner to root on /dev/pts/5
Nov 19 08:39:48 daria su[6046]: abner to root on /dev/pts/1
Nov 19 08:42:38 daria kernel: [nhop_ctl] inet.0 nhop_free: failed to unlink nh#2/inet/ng0_orthanc/192.168.15.1

My jail.conf.d/orthanc.conf file content:
Code: 
orthanc {
        host.hostname = "orthanc.int.mynetwork.net"; # hostname
        path = "/usr/jails/orthanc";           # root directory

        exec.clean;
        exec.system_user = "root";
        exec.jail_user = "root";

        # networking with netgraph
        vnet;
        vnet.interface = "ng0_orthanc";             # vnet interface(s)
        exec.prestart += "jng bridge orthanc bce1"; # bridge interface(s)
        exec.poststop += "jng shutdown orthanc";    # destroy interface(s)
        exec.start += "/sbin/dhclient ng0_orthanc"; # get ip address via dhcp

        # Standard recipe
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown jail";
        exec.consolelog = "/var/log/jail_orthanc_console.log";
        mount.devfs;                    # mount devfs

        # Optional (default off)
        devfs_ruleset = "11";           # rule to unhide bpf for DHCP
        #allow.mount;                   # mount /etc/fstab.xxx
        #allow.set_hostname = 1;        # Allow hostname to change
        allow.sysvipc = 1;              # Allow SysV Interprocess Comm.
        #allow.raw_sockets = 1;          # at suggestion of amoradi.org
}

My postgreSQL pg_hba.conf file:
Code: 
# TYPE      DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local           orthanc0        orthuser                                md5
local           all                   postgres                               trust
# IPv4 local connections:
host             orthanc0        orthuser        127.0.0.1/32            md5
# 127.0.0.1   orthanc0        orthuser        127.0.0.1/32            md5
# IPv6 local connections:
# host    all             all             ::1/128                 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                             trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                     trust


from jail console log:

Code: 
root@daria:/var/log # tail jail_orthanc_console.log
DHCPREQUEST on ng0_orthanc to 255.255.255.255 port 67
DHCPACK from 192.168.15.1
bound to 192.168.15.42 -- renewal in 150 seconds.
Usage: /etc/rc [fast|force|one|quiet](start|stop|restart|rcvar|enable|disable|delete|enabled|describe|extracommands|status|poll)
DHCPDISCOVER on ng0_orthanc to 255.255.255.255 port 67 interval 5
DHCPOFFER from 192.168.15.1
DHCPREQUEST on ng0_orthanc to 255.255.255.255 port 67
DHCPACK from 192.168.15.1
bound to 192.168.15.42 -- renewal in 150 seconds.
Usage: /etc/rc [fast|force|one|quiet](start|stop|restart|rcvar|enable|disable|delete|enabled|describe|extracommands|status|poll)
root@daria:/var/log # tail -n 30 jail_orthanc_console.log
DHCPDISCOVER on ng0_orthanc to 255.255.255.255 port 67 interval 8
DHCPOFFER from 192.168.15.1
DHCPREQUEST on ng0_orthanc to 255.255.255.255 port 67
DHCPACK from 192.168.15.1
bound to 192.168.15.42 -- renewal in 150 seconds.
Usage: /etc/rc [fast|force|one|quiet](start|stop|restart|rcvar|enable|disable|delete|enabled|describe|extracommands|status|poll)
DHCPDISCOVER on ng0_orthanc to 255.255.255.255 port 67 interval 4
DHCPOFFER from 192.168.15.1
DHCPREQUEST on ng0_orthanc to 255.255.255.255 port 67
DHCPACK from 192.168.15.1
bound to 192.168.15.42 -- renewal in 150 seconds.
Usage: /etc/rc [fast|force|one|quiet](start|stop|restart|rcvar|enable|disable|delete|enabled|describe|extracommands|status|poll)
DHCPDISCOVER on ng0_orthanc to 255.255.255.255 port 67 interval 6
DHCPOFFER from 192.168.15.1
DHCPREQUEST on ng0_orthanc to 255.255.255.255 port 67
DHCPACK from 192.168.15.1
bound to 192.168.15.42 -- renewal in 150 seconds.
Usage: /etc/rc [fast|force|one|quiet](start|stop|restart|rcvar|enable|disable|delete|enabled|describe|extracommands|status|poll)
DHCPDISCOVER on ng0_orthanc to 255.255.255.255 port 67 interval 5
DHCPOFFER from 192.168.15.1
DHCPREQUEST on ng0_orthanc to 255.255.255.255 port 67
DHCPACK from 192.168.15.1
bound to 192.168.15.42 -- renewal in 150 seconds.
Usage: /etc/rc [fast|force|one|quiet](start|stop|restart|rcvar|enable|disable|delete|enabled|describe|extracommands|status|poll)
DHCPDISCOVER on ng0_orthanc to 255.255.255.255 port 67 interval 5
DHCPOFFER from 192.168.15.1
DHCPREQUEST on ng0_orthanc to 255.255.255.255 port 67
DHCPACK from 192.168.15.1
bound to 192.168.15.42 -- renewal in 150 seconds.
Usage: /etc/rc [fast|force|one|quiet](start|stop|restart|rcvar|enable|disable|delete|enabled|describe|extracommands|status|poll)



Also here is the postgreSQL related block from my orthanc.conf file:
Code: 
  "PostgreSQL" : {
          "EnableIndex" : true,
          "EnableStorage" : false,
          "Host" : "localhost",
          "Port" : 5432,
          "UnixSocket" : "/tmp/.s.PGSQL.5432",
          "Database" : "orthanc0",
          "Username" : "orthuser",
          "Password" : "xxxxxxxxxx",
          "EnableSsl" : false,
          "MaximumConnectionRetries" : 10,
          "ConnectionRetryInterval" : 5,
          "IndexConnectionsCount" : 1
   },
 
avner said:
I wonder if one of the pg_hba.conf settings is contributing to the problem
Click to expand...
If there's an issue with that PostgreSQL itself would simply fail to start. The rest of that jail would still start.

Code: 
jail: orthanc: /bin/sh /etc/rc: failed
This seems to indicate an issue in the jail itself. Does /usr/jails/orthanc/etc/rc actually exist? Maybe there's an error in the jail's /etc/rc.conf, that would result in strange boot errors too.

I would suggest removing this as it's likely the cause of the failure.
Code: 
exec.start += "/sbin/dhclient ng0_orthanc"; # get ip address via dhcp
 
SirDice said:
If there's an issue with that PostgreSQL itself would simply fail to start. The rest of that jail would still start.

Code: 
jail: orthanc: /bin/sh /etc/rc: failed
This seems to indicate an issue in the jail itself. Does /usr/jails/orthanc/etc/rc actually exist? Maybe there's an error in the jail's /etc/rc.conf, that would result in strange boot errors too.

I would suggest removing this as it's likely the cause of the failure.
Code: 
exec.start += "/sbin/dhclient ng0_orthanc"; # get ip address via dhcp
Click to expand...
I tried this but jail would still not start. I then commented out the whole vnet section of my jail.conf and still jail would not start. One thing I noticed that strikes me as odd is that ng0_orthanc is listed when I enter ifconfig on the host.
 
avner said:
Nov 19 08:14:19 daria kernel: [nhop_ctl] inet.0 nhop_free: failed to unlink nh#1/inet/ng0_orthanc/resolve
Nov 19 08:14:19 daria kernel: [nhop_ctl] inet.0 nhop_free: failed to unlink nh#2/inet/ng0_orthanc/192.168.15.1
Click to expand...
These I get too. Along with these:
rtadvd[7848]: <rm_ifinfo_index>: ifinfo not found (idx=15)
Probably all harmless (just recognizing, the first seem related to netgraph, the second related to IPv6 being surprized of VNET interfaces moving from host to jail).

But this one is not harmless:
avner said:
Usage: /etc/rc [fast|force|one|quiet](start|stop|restart|rcvar|enable|disable|delete|enabled|describe|extracommands|status|poll)
Click to expand...
Your jail config is hosed.
I suggest you sort this out:
  1. get the basic jail rc script to cleanly execute and start/stop
  2. get the networking interfaces to appear and get functional
  3. then maybe apply dhcp and such
  4. then care about the applications
BTW:
avner said:
exec.poststop += "jng shutdown orthanc"; # destroy interface(s)
Click to expand...
I'm not sure this cumulative syntax is allowed. The examples use it only for array parameters, not for strings. For strings it seems that $variables do work, and actual variables that should be expanded by the shell want to have \$ in front, and commands want to be separated by ; and/or linebreak.
 
I appreciate the help but I have virtually identical jail-specific jail.conf files under /etc/jail.conf.d/ and the other two jails are working fine. I personally think the problem is with permissions/ownership of some of the configuration files. I was editing these files directly from the host shortly before the problem began. I tried chmod -R 777 /my/jail/root but the jail still would not start. I don't know if there is any way to correct the ownership of these files without starting the jail. Fortunately there is little of value in this jail except for the content of a couple of configuration files I was working on. At this point my plan is to create a new jail, install postgresql, save what I need of the configuration files and then delete the non-functional jail.

Lessons learned:
1. Edit config files and generally manage software from inside the jail ( possible exception pkg -j myjail ).
2. It is important to have backups.
 
avner said:
I tried chmod -R 777 /my/jail/root but the jail still would not start.
Click to expand...
Sigh. Please, for your sanity's sake and mine, NEVER chmod 777 anything. It's NEVER a good thing to "try". Or to test any kind of "permission" issue you think you may have.
avner said:
I don't know if there is any way to correct the ownership of these files without starting the jail.
Click to expand...
I suggest removing it and creating a fresh new one. The effort of trying to restore it doesn't outweigh the effort of just creating a blank new jail.
avner said:
It is important to have backups.
Click to expand...
Always make backups.
 
You must log in or register to reply here.
Back
Top