jails networking issue with jail on FreeBSD 14.0-RELEASE

J

jrushford

Yesterday, I saw that FreeBSD 14.0-RELEASE was available for download and I've installed it on my desktop machine, I haven't seen any issues until I built a jail on it that I'm going to use for nginx. So, I've built a thick jail using vnet networking. I followed the handbook section precisely at this point: creating-vnet-jail. It works perfectly for a few minutes. I'm able to while in the jail, to ssh to any host on my subnet and from any host on my subnet, I can ssh into the jail. After a few minutes though, I can no-longer ssh into the jail as I get connection refused and while in the jail, I'm no longer able to ssh out to anything including the jail host. pings to and from the jail fail also. I've looked into the logs on the host and the jail console log and I don't see anything that would cause this behavior. When I restart the jail, it works for awhile but then the networking is kaput again after a few minutes. I don't have any ideas. Was wondering if anyone has seen this sort of behavior?

relevant portion of my rc.conf on the jail host:

Code: 
defaultrouter="192.168.1.1"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.1.2/24 addm igb0 up"
ifconfig_igb0="up"

my jail.conf:

Code: 
nginx {
# startup/logging
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";

# permissions
  allow.raw_sockets;
  exec.clean;
  mount.devfs;
  devfs_ruleset = 5;

# hostname/path
host.hostname = "${name}";
path = "/ztank/jails/containers/${name}";

# vnet/vimage
vnet;
vnet.interface = "${epair}b";

# network
$id = "4";
$ip = "192.168.1.${id}/24";
$gateway = "192.168.1.1";
$bridge = "bridge0";
$epair = "epair${id}";

# ADD TO bridge interface
exec.prestart += "ifconfig ${epair} create up";
exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
exec.start += "ifconfig ${epair}b ${ip} up";
exec.start += "route add default ${gateway}";
exec.poststop = "ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "ifconfig ${epair}a destroy";
}
 
I haven't made any changes but stopping and then restarting the jail clears the issue for awhile. After restarting I can reach the jail via ssh and from the jail I can ssh to other hosts on my network. I cannot reach the defaultrouter though from the jail so the jail is unable to get outside to the internet. After some time though, about an hour, I can no-longer ssh into the jail and I can no-longer reach anything on my network from inside the jail. The logs show nothing unusual. Restarting the jail clears things up for awhile except for reaching the defaultrouter.
 
Now this is strange. I decided that I might try using a bhyve VM instead of the jail for nginx. So, I installed vm-bhyve on the host and per the instructions I added this to rc.conf and rebooted the host:

Code: 
vm_enable="YES"
vm_dir="zfs:ztank/vm"

Now, after rebooting, the jail nginx is now working as it should. I can reach the defaultrouter from the jail and I can install packages from the jail. From the jail I can reach hosts on my network and from hosts on my network, I can now reach the jail. It's been some time since rebooting and there seems to no-longer an issue with the jail networking. I even backed out the vm rc.conf changes and rebooted and the jail is still working fine. The only change I made was to add the two lines for the vm-bhyve package to rc.conf. I don't know why it's fixed now. I'm wondering if the handbook section left something out with the vnet networking that enabling the vm-bhyve package somehow fixed the bridge0 or igb0 interface configuration?
 
There was an issue with the handbook section. I had to add the ifconfig_igb0="up" line in my rc.conf to ensure that the physical interface was brought up. There was no mention of this in the handbook section but it is mentioned in the bridge(4) man page. I think the handbook needs updating.
 
In the past the handbook was not very precise how to setup vnet jails, but I have missed to update it .
I don't understand why this problem occur after some time, but this is my working bridge setup , which I am using with 13.2 .

rc.conf

Code: 
### Start Network Bridge MANGED BY SALT###
defaultrouter="192.168.10.1"
cloned_interfaces="bridge1"
create_args_bridge1="addm vtnet0"
ifconfig_bridge1="inet 192.168.10.100 netmask 255.255.255.0"
ifconfig_vtnet0='up'
### End Network Bridge MANGED BY SALT###

I am curious if this works for you , because I don't want to update without knowing vnet is working properly :)
 
Last edited by a moderator:
Strangely enough, I had the same issue. After installing vm-bhyve as I intended too, this resolved the issue.
The only issue I'm getting is that I'm seeing around 50% of traffic in to the virtual machine.

I would be interested in what others get but I get a slight issue with speed.
90MB/s on the host
90MB/s in the jail
62MB/s from within the VM in the Jail

I expect this is more the bHyve than anything else.
 
Sebastian, thanks for your reply. I did modify my rc.conf to resemble your settings and had no problem after a reboot.
These are my settings:

Code: 
defaultrouter="192.168.1.1"
cloned_interfaces="bridge0"
create_args_bridge0="addm igb0"
ifconfig_bridge0="inet 192.168.1.2/24"
ifconfig_igb0="up"

So far I haven't seen any issues, my nginx jail is functioning very well, and if I don't reply with any you can assume it's all good.
 
Last edited by a moderator:
doublerabbit said:
Strangely enough, I had the same issue. After installing vm-bhyve as I intended too, this resolved the issue.
The only issue I'm getting is that I'm seeing around 50% of traffic in to the virtual machine.

I would be interested in what others get but I get a slight issue with speed.
90MB/s on the host
90MB/s in the jail
62MB/s from within the VM in the Jail

I expect this is more the bHyve than anything else.
Click to expand...
Yeah it's very strange that installing vm-bhyve clears this up. I've not had any issues with my jail since.
 
jrushford said:
Sebastian, thanks for your reply. I did modify my rc.conf to resemble your settings and had no problem after a reboot.
These are my settings:

defaultrouter="192.168.1.1" cloned_interfaces="bridge0" create_args_bridge0="addm igb0" ifconfig_bridge0="inet 192.168.1.2/24" ifconfig_igb0="up"

So far I haven't seen any issues, my nginx jail is functioning very well, and if I don't reply with any you can assume it's all good.
Click to expand...

Thanks for your response. Yeah ,this issue occurred to me after updating to 13.2 from 13.1 . My server was no longer reboot stable and I opened a bug id which results in the configration above.
 
You must log in or register to reply here.
Back
Top