iocage IPv6 jail: network services can't bind to address

Hi all,

I just started using sysutils/iocage to manage my jails. So far so good, networking is fine, I get an IPv4 and IPv6 address, I can resolve DNS queries with a local_unbound service on my host, everyone's happy. However, running jails that use IPv6 addresses proves not easy.

My test jail:
Code:
# iocage clone f0c02024-1741-11e5-b9bb-eca86bf1c8a3 tag=test
# iocage set defaultrouter6="fe80::1ee6:c7ff:fe52:740%em0" test
# iocage set ip6_addr="em0|2001:41d0:e:10a7:10:10:10:10" test
# iocage start test

Then, I install www/nginx on test; then configure it:
Code:
server {
  listen [2001:41d0:e:10a7:10:10:10:10]:80;
  server_name my.name.com;
  root /usr/local/www/nginx-dist;
}

Then, add to rc.conf:
Code:
nginx_enable="YES"
nginx_debug="YES"

Finally, start the jail:
Code:
# iocage stop test
# iocage start test

Check sockstat's output: no dice, www/nginx is not running and fails with the following errors:
Code:
2015/06/24 19:42:36 [emerg] 18050#0: bind() to [2001:41d0:e:10a7:10:10:10:10]:80 failed (49: Can't assign requested address)
2015/06/24 19:42:36 [emerg] 18051#0: bind() to [2001:41d0:e:10a7:10:10:10:10]:80 failed (49: Can't assign requested address)
However, should I restart the jail:
Code:
# iocage restart test
www/nginx seems happy and starts. It will also start, if after a cold boot I instruct the jail to:
Code:
# jexec NUMBER service nginx start
After some discussion on #freebsd and #iocage, no solution so far, but:
  • The issue is not limited to www/nginx: www/apache24 faces the same issues;
  • While the jail starts, I see this, at the same moment nginx complains it can't bind to address:
    Code:
    inet6 2001:41d0:e:10a7:10:10:10:10 prefixlen 64 tentative
  • Setting netwait_enable does not work in a jail (look at the rc script)
  • The issue only appears during a jail cold boot.
  • The 2001:41d0:e:10a7:10:10:10:10 address, even if it appears in the jails /etc/hosts does not change anything
  • The 2001:41d0:e:10a7:10:10:10:10 address does not exist on my system while the jail is off/down.
  • Inside the jail, ifconfig(8) returns the expected IPv6 address
  • I run the GENERIC kernel
  • IPv6 connectivity is perfectly fine (name resolution and network access)
I hope you have all the details needed,
Thanks in advance.

EDIT: At the moment, a working solution, as suggested here: https://lists.freebsd.org/pipermail/freebsd-jail/2013-July/002347.html is to add the following sysctl setting to /etc/sysctl.conf:
Code:
## -- for jails
net.inet6.ip6.dad_count=0
 
Last edited:
I can verify this issue is real and that the solution works. My setup:

Each jail is assigned both IPv4 and IPv6 addresses. When the jails are started during boot of the host, all IPv4 services are started normally and automatically and NONE of the IPv6 services start automatically (for example Unbound with IPv6 addresses in the unbound.conf, in one jail and sshd with no ListenAddress assignments, in each jail).

If the iocage restart jailname is issued on the host, both the IPv4 and IPv6 services start normally.

If the fix sysctl net.inet6.ip6.dad_count=0 is issued and the jail stopped and started, all services start normally.

So adding
Code:
net.inet6.ip6.dad_count=0
to the /etc/sysctl.conf file is the cure for now.
 
Your fix suggests that there is a bridge involved somewhere in your setup and that bridge is not passing router solicitation messages and other icmp6(4) traffic trough properly. This will cause the assigned IPv6 addresses to stay in tentative mode.
 
I'd be lying if I said I understood what you mean. What kind of "bridge"? The IP addresses for the jails are fixed - assigned by me, not DHCP derived and not statelessly configured. Since I assign them, how could they be tentative?

Puzzled but I want to learn?

Using sysutils/iocage here. Each jail starts and reports this: "Started (shared IP mode) OK" I searched the iocage(8) documentation for "shared IP mode" and found nothing and a Google search doesn't tell me what that means. Is that a bridge of some sort?

Thanks in advance
 
A bridge is something that connects two or more LAN segments together on the layer 2 (data link) to form a single broadcast domain (a "logical" LAN network). I have witnessed the same problems in the past with IPv6 addresses when I was using a variety of low quality wireless access points that are in effect bridges between the wired upstream and the wireless network they create. The solution was also the same, setting the dad_count sysctl(8) to zero. The broken access points were interfering with the Neighbour Discovery Protocol that is a very central part of IPv6 (https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol) making the host believe that the IPv6 address it was trying to use was already reserved by some other host in the same network segment, that's why the address stayed in tentative mode forever.
 
Hmmm. I'm building out a new server (have two identical boxes and build up one and put it into production while I tinker with improvements to the "new" one, then swap them out when the new one is ready; then start anew with the former server, rebuilding.) In the meantime there is a server operating on the LAN and facing the internet.

Okay, I have two NICs on the new server mobo (Supermicro brand). One is presently facing the LAN (igb0) with dhcpd assigned IP addresses of 10.0.0.123 and fd00::183 (using a stateful dhcpd6 assignment process presently). The other NIC (igb1) is presently not connected to anything while I set up the new sever. It has assigned IP addresses of 10.0.1.1 and fd00::1:1

All of the jails are attached to the igb1 NIC with their individually assigned -4 and -6 IPs.

The igb0 NIC has rtsold running on its behalf which I presumed meant it was seeking a dhcpd6 server for an IPv6 assignment or if not found, then it would do a stateless IPv6 autoconfiguration. Is the problem that the rtsold daemon is also pinging the igb1 NIC to see what it is up to?

I'm pretty sure these are good quality NICs.

(PS - I'm not an IT guy (healthcare for a career, now retired) so I'm learning this from you experts as I stumble along)
 
Your fix suggests that there is a bridge involved somewhere in your setup and that bridge is not passing router solicitation messages and other icmp6(4) traffic trough properly. This will cause the assigned IPv6 addresses to stay in tentative mode.
Nope, no such thing on my setup. Here is the ifconfig(8) output:
Code:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether ec:a8:6b:f1:c8:a3
        inet 151.80.XX.YYY netmask 0xffffff00 broadcast 151.80.43.255 
        inet6 fe80::eea8:6bff:fef1:c8a3%em0 prefixlen 64 scopeid 0x1 
        inet6 2001:41d0:e:aaaa::1 prefixlen 64 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.10.10.1 netmask 0xffffff00 
        inet6 2001:470:aaaa:7765::1 prefixlen 64 
        inet 10.10.10.3 netmask 0xffffff00 
        inet6 2001:470:aaaa:7765:706f:7068:6f62:65 prefixlen 64 
        inet 10.10.10.2 netmask 0xffffff00 
        inet 10.10.10.5 netmask 0xffffff00 
        inet6 2001:470:aaaa:7765:7363:6f70:6f70:686f prefixlen 64 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo2: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.10.20.1 netmask 0xffffff00 
        inet6 2001:470:aaaa:6261::1 prefixlen 64 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo3: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.10.30.1 netmask 0xffffff00 
        inet6 2001:470:aaaa:6f74::1 prefixlen 64 
        inet 10.10.30.19 netmask 0xffffff00 
        inet6 2001:470:aaaa:6f74:0:73:616d:6261 prefixlen 64 
        inet 10.10.30.12 netmask 0xffffff00 
        inet6 2001:470:aaaa:6f74:0:6261:636b:7570 prefixlen 64 
        inet 10.10.30.77 netmask 0xffffff00 
        inet6 2001:470:aaaa:6f74:7465:7272:6172:6961 prefixlen 64 
        inet 10.10.30.2 netmask 0xffffff00 
        inet6 2001:470:aaaa:6f74:746f:7272:656e:74 prefixlen 64 
        inet 10.10.30.5 netmask 0xffffffff 
        inet6 2001:470:aaaa:6f74::7a:6e63 prefixlen 64 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo4: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
        tunnel inet 151.80.43.167 --> 216.66.84.46
        inet6 2001:470:1f14:cde::2 --> 2001:470:1f14:cde::1 prefixlen 128 
        inet6 fe80::eea8:6bff:fef1:c8a3%gif0 prefixlen 64 scopeid 0x7 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::eea8:6bff:fef1:c8a3%tun0 prefixlen 64 scopeid 0x9 
        inet 10.21.0.1 --> 10.21.0.2 netmask 0xffffff00 
        inet6 2001:470:aaaa:7670::1 prefixlen 64 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 982
 
No bridge here either. On github/iocage under "known issues" this very DAD issue is discussed and therein it is said that the fix cited above is the only known cure at this time.
 
I can confirm that this isn't solely a problem of iocage but jails in general. I use manual jail.conf method and encountered the same problem.
 
I had the same problem with Intel Pro NIC using em driver which I resorted by issuing this command sysctl net.inet6.ip6.dad_count=0 which fixed the problem but I have no issues with igb driver. I think it's more of a driver issue.
 
I can confirm the same, have seen this on different servers i have been running.
I have two different NICs that has the same problem:
Code:
re0@pci0:2:0:0: class=0x020000 card=0x78161462 chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

em0@pci0:4:0:0: class=0x020000 card=0x83691043 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
sysctl net.inet6.ip6.dad_count=0 fixes this, for me it seems like rc.conf is executed before ipv6 is loaded on the jail.

Is this just a iocage problem?
If my memory serves me right I did start the same jail from jail.conf and there it works as expected, also talked to the developer about this, but seems like he is as stumped as I by this. If anyone else can confirm that this is only on iocage jails and not on jail started from rc.conf, it would be nice.
 
It's definetly not iocage specific, same issue using "basic" jail with jail.conf on FreeBSD 11.2. This should be documented in handbook :)
 
Back
Top