Solved System crashing on reboot. Failed to unlink nh#* rebooting in 15 sec

[bryanjlaz]

I am posting this in emulation virtualization because I'm not exactly sure where it fits. It could also be a topic in Networking. I was editing my ipfw rules file for a jail that runs a web server. I'm still in a bit of a learning phase with ipfw and was trying to ultimately get it set where obviously access is most minimal. A couple of times during the process I locked myself out but was able to regain access through jexec *jail /bin/sh and changing the .rules file. The last time was not so lucky. I could access the host but when I rebooted the jails I received the message "apache is dying", apache being the name of my jail. I proceeded to reboot the host and upon startup was met the message in my post title, "Failed to unlink nh#* rebooting in 15 seconds". I'm not certain of what the exact number was following nh# but I can run it through again and get it if it's vital info. I can boot into single user mode but when I try to edit any files, any at all, I receive "unable to create *". This was with ee as well. Any ideas would be greatly appreciated.

 

[PMc]

when I rebooted the jails I received the message "apache is dying", apache being the name of my jail.

This is normal. When networking is used, a jail may, after having been shutdown, stay in the "dying" state for some 10-15 minutes until the TCP counters timeout and all networking can be removed.

I proceeded to reboot the host and upon startup was met the message in my post title, "Failed to unlink nh#* rebooting in 15 seconds".

You mean these kind of things?

kernel: [926261] [nhop_ctl] inet6.0 nhop_free: failed to unlink nh#7/inet6/ndata1l/fd00::2101
kernel: [926264] [nhop_ctl] inet.0 nhop_free: failed to unlink nh#2/inet/ndata1l/resolve

I am seeing lots of these whenever stopping my jails, and they do not seem to do anything worrysome - jail start and stop works just fine here.

The "failed to unlink" and the "rebooting in 15 seconds" may not have anything to do with each other. Question now is, where does the "rebooting in 15 seconds" come from - I am not used to that one...

Out of curiosity: are you using netgraph for connecting the jails?

I'm not certain of what the exact number was following nh# but I can run it through again and get it if it's vital info. I can boot into single user mode but when I try to edit any files, any at all, I receive "unable to create *". This was with ee as well. Any ideas would be greatly appreciated.

When booting single-user, Your root filesystem is usually mounted read-only - so it is not possible to edit any files. You need to re-mount it writeable first. mount -u -o rw / should work.

 

[Alex Seitsinger]

This is normal. When networking is used, a jail may, after having been shutdown, stay in the "dying" state for some 10-15 minutes until the TCP counters timeout and all networking can be removed.


You mean these kind of things?

kernel: [926261] [nhop_ctl] inet6.0 nhop_free: failed to unlink nh#7/inet6/ndata1l/fd00::2101
kernel: [926264] [nhop_ctl] inet.0 nhop_free: failed to unlink nh#2/inet/ndata1l/resolve

I am seeing lots of these whenever stopping my jails, and they do not seem to do anything worrysome - jail start and stop works just fine here.

The "failed to unlink" and the "rebooting in 15 seconds" may not have anything to do with each other. Question now is, where does the "rebooting in 15 seconds" come from - I am not used to that one...

Out of curiosity: are you using netgraph for connecting the jails?


When booting single-user, Your root filesystem is usually mounted read-only - so it is not possible to edit any files. You need to re-mount it writeable first. mount -u -o rw / should work.

I also receive(d) these errors (see below) when stopping a vnet jail that connects using an epair and bridge (on the host). A quick google shows points the error comes from route(8).

[nhop_ctl] inet.0 nhop_free: failed to unlink nh#3/inet/<host-nic>/resolve
[nhop_ctl] inet.0 nhop_free: failed to unlink nh#2/inet/<host-nic>/<host-subnet-gw>

I can confirm that this issue seems to come from the epair (e0b_bastille0 in my case) in the vnet jail. Adding the following line to your jail.conf(8) prevents the kernel panic and allows the jail to shutdown normally.

exec.prestop += "/sbin/ifconfig e0b_bastille0 destroy";