jails Jail host on bridge unreachable over TCP (IPv6-only)

sdavidb

sdavidb

Hello everyone...

I'm fairly new to FreeBSD and come from a Linux/Docker background, but I've been amazed and impresed with the jails functionality in this OS.

I've deployed a small DevOps/CI stack on a small Intel mini-PC in my home, using IPv4, if_bridge, and iocage. I'm trying to migrate that stack to a VPS host, but using only iPV6

I have one virtual NIC from my VPS provider, vtnet0, and I've attached it to a bridge, bridge0, which iocage is configured to use for vnet jail interfaces.

I've set an inet6 address on the bridge, configured it to clone the MAC address of the first interface. When the bridge is cloned, it's created with `addm vtnet0` as an argument.

When my host boots, the bridge gets the VPS-assigned IPv6 address, and then the jailed host performs it's own SLAAC process and finds the correct router. Everything is fine on that front.

What I'm struggling with is getting the jail hosts to listen on a TCP socket on their assigned IPv6 address. I can ping the host, the jail, and they can ping each other, as well as ipv6.google.com. The jail can also access the internet, and I've got DNS64 working, and can install things with pkg in the jail.

On the host, I can also open listening sockets, and connect to them from the jail, but when I try to do so on the jailed guest, nothing gets through, either from the host, or the internet.

Basically, the jailed host can initiate connections just fine, but cannot listen.

I'm not running a firewall quite yet, because I want things to just work before I secure things, and restrict ports.

Here's my /etc/sysctl.conf:

Code: 
hostname="alcatraz"
sshd_enable="YES"
ntpd_enable="YES"

static_routes="linklocal"

# Fix virtio random issue
devmatch_blacklist="virtio_random.ko"

# Disable sendmail
sendmail_enable="NONE"

#ifconfig_vtnet0="DHCP -rxcsum -tso"
#ifconfig_vtnet0_ipv6="inet6 accept_rtadv -rxcsum6 -tso6"

ipv6_activate_all_interfaces="YES"
rtsold_enable="YES"
rtsold_flags="-aF"

## End of VPS configuration

cloned_interfaces="bridge0"
create_args_bridge0="addm vtnet0 up"
autobridge_interfaces="bridge0"
autobridge_bridge0="vtnet0"
ifconfig_vtnet0="up"

ifconfig_bridge0_ipv6="inet6 accept_rtadv -rxcsum6 -tso6 auto_linklocal up"

gateway_enable="YES"
ipv6_gateway_enable="YES"
ipv6_cpe_wanif="bridge0"

and /etc/rc.conf:
Code: 
# $FreeBSD$
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
kern.ipc.maxsockbuf=67108864
net.inet.tcp.sendbuf_max=67108864
net.inet.tcp.recvbuf_max=67108864
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
#net.inet.tcp.recvbuf_inc=524288
#net.inet.tcp.cc.algorithm=htcp

security.jail.allow_raw_sockets=1

net.inet.tcp.tso=0
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

net.link.bridge.ipfw=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0
net.link.bridge.inherit_mac=1

I was hoping to avoid setting up NAT rules at this stage, and to use the if_bridge as a network switch, using my VPS's router.

Do I still have to set up NAT, to allow the jailed hosts to listen and respond to connections?
 
A strange thing I noticed. On my home setup, I have the IPv4 address on the physical interface, re0, and the bridge has no IP. I can access the jail hosts on my home network, as if they're connected to the LAN, but the jails and hosts cannot see each other.

On my VPS, I have the IPv6 address assigned to the bridge, and the jailed host can see it's physical host, but the physical host and nobody on the greater internet can connect to the jailed host interface on bridge.
 
Assuming that you're using separate routing tables for each jail (VNET), you should:

1)
create bridge interface
/sbin/ifconfig bridge create name bridge0

2)
create epair interfaces
/sbin/ifconfig epair create

3)
add the host's external interface as a member of the bridge.
/sbin/ifconfig bridge0 addm re0

4) add the first epair interface to the bridge.
/sbin/ifconfig bridge0 addm epair0a

5) move the second epair interface into the jail.
/sbin/ifconfig epair0b vnet <JAIL-ID>

6)
apply the correct WAN/LAN-accessible IP address to the epair interface within the jail.
/usr/sbin/jexec <JAIL-ID> /sbin/ifconfig epair0b inet 192.168.1.2/24
The routing tables should now offer the communication expected.

I would encourage you to read the manpages on the related topics (as far back as comfortable; jails, jail networking, kernel options, ipv4, ipv6 routing tables, virtual network interfaces) so that experimenting on your own becomes a comfortable and regular exercise to find your answers yourself.
 
... I think i may have forgotten to mention that I'm using iocage to spin up and manage the jails. 😅

A few months back, I decided to see why jails are so great, so I created one manually, and immediately got to playing with ezjail and eventually moved on to iocage.

I've been buried in the manuals for epair and if_bridge for the better part of a year or so, haha...

Regardless, my confusion probably came from the fact that I'm late to the party with IPv6, and I was reading an article about IPv6 jails with various methods of making them work:

"FreeBSD: IPv6 in a VNET jail" - evilham

In some ways, IPv6 is simpler, but at the same time there's a whole new slate of terminology (ndp vs arp, etc)

I think I was combining some of the tools an instructions for running my own IPv6 router, when what I needed was to set up the bridge as a sort of network switch, and allow my host provider's router to set up the jailed hosts' networks.

---
In the end I got it to work!

I started the project from scratch, and went through each part methodically, and now I can reach the jailed host from my home network!

First I set up my host so it would create a bridge, andI set it up like so:
Code: 
                                   ·--> | jail host |  firewall passes
                                  /     | interface |  traffic on bridge
| router | <--> | jail host | <--·                     (switch-like)
                | bridge    | <--·
                                  \     | vnet |       uses SLAAC to
Pre-existing                       ·--> | jail |       gain IPv6
Does SLAAC                                             Connectivity

Then, I configured an iocage jail with a private ipv4 address (10.0.0.1), so that the iocage automation would create the epair interface. It seems to forget to do so unless you specify an address for the interface.

Within the jail, I further configured the epair0b interface with: inet6 accept_rtadv auto_linklocal inside the jailed host's rc.conf.
 
security.jail.param.allow.raw_sockets is sysctl(8) variable not rc.conf

Edit:
nvm you swap the rc.conf with sysctl.conf in your original post. You can ignore the above.
 
sdavidb said:
immediately got to playing with ezjail and eventually moved on to iocage.
Click to expand...

That's how it happened for me, too. I remember liking IOCage a lot. However, now I just use jail.conf with some nullfs mounting for some thick/thin jails.

sdavidb said:
I've been buried in the manuals for epair and if_bridge for the better part of a year or so, haha...
Click to expand...

Indeed sometimes it's tough to interpret things in a manpage without some experience to reference.

sdavidb said:
my confusion probably came from the fact that I'm late to the party with IPv6
Click to expand...

You aren't the only one--I explicitly remove and disable IPv6 support because I haven't bothered to familiarize myself with everyting, yet. Who needs another door available on their system for traffic to travel through?

sdavidb said:
In the end I got it to work!
Click to expand...

You did it! Well done. That's the goal. Now, make sure that your network is still secure after all the changes.
 
How exactly did you set up host bridge?
I am struggling with a similar issue. I can't seem to get any traffic to the host via IPv6, but IPv4 works totally fine. I have access via IPv4, but I am not able to ping the host over IPv6.
 
rr0s said:
How exactly did you set up host bridge?
I am struggling with a similar issue. I can't seem to get any traffic to the host via IPv6, but IPv4 works totally fine. I have access via IPv4, but I am not able to ping the host over IPv6.
Click to expand...
You should confirm the following:
1) If using VNET jails or shared IP
2) Packet filtering on the host in the jail (for VNET jail) have the correct rules to allow traffic to flow.
3) sysctl net.inet.ip.forwarding=1 (for IPv4) net.inet6.ip6.forwarding=1 (for IPv6).
4) Nothing else is prevents traffic from flowing to/from host and jail(s).
 
I've confirmed that all of those are set, but maybe I'm missing something.

I'm setting up my bridge in /etc/rc.conf:

Code: 
# primary network interface
ifconfig_bge3="SYNCDHCP"
ifconfig_bge3_ipv6="inet6 accept_rtadv auto_linklocal"

rtsold_enable="YES"
rtsold_flags="-aF"

# jails
devfs_load_rulesets="YES"
cloned_interfaces="bridge0 tun tap"

# jail bridge
ifconfig_bridge0="addm bge3 addm tap0 up"

I have the following settings in /etc/sysctl.conf:

Code: 
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
net.inet6.ip6.use_tempaddr=0
net.inet6.ip6.prefer_tempaddr=0
net.link.tap.up_on_open=1

And my jail is configured with:

Code: 
foo {
    $id = 14;
    $addr = "192.168.4.${id}";
    $mask = "255.255.255.0";
    $gw = "192.168.4.1";

    vnet;
    vnet.interface = "epair${id}b";

    exec.prestart = "ifconfig epair${id} create up";
    exec.prestart += "ifconfig epair${id}a up descr vnet-${name}";
    exec.prestart += "ifconfig bridge0 addm epair${id}a up";

    exec.start    = "/sbin/ifconfig lo0 127.0.0.1 up";
    exec.start    += "/sbin/ifconfig epair${id}b ${addr} netmask ${mask} up";
    exec.start    += "/sbin/route add default ${gw}";
    exec.start    += "/bin/sh /etc/rc";

    exec.poststop = "ifconfig bridge0 deletem epair${id}a";
    exec.poststop += "ifconfig epair${id}a destroy";

    host.hostname = foo;
    path = /usr/local/jails/${name}/root;
    exec.consolelog = /var/log/jails/foo_console.log;
    mount.devfs;
    devfs_ruleset = 7;
    allow.sysvipc = 1;
    allow.mlock = 1;
    allow.raw_sockets = 1;
    allow.mount.tmpfs = 1;
    persist;
}

And finally, the /etc/rc.conf in the jail:

Code: 
ifconfig_epair14b_ipv6="inet6 accept_rtadv auto_linklocal"
rtsold_enable="YES"

I have stopped pf while I'm trying to get to the bottom of this.

With this setup, it gets assigned the IPv4 address 192.168.4.14, and can ping the jail host and external IPs traffic via IPv4.

It also is assigned an IPv6 address from the router, and can ping external IPv6 addresses, but cannot ping the jail host with it's IPv6 address.

If I understand it - though I may be just missing it completely - this is a fairly similar setup to the diagram that sdavidb posted.
sdavidb said:
Code: 
                                   ·--> | jail host |  firewall passes
                                  /     | interface |  traffic on bridge
| router | <--> | jail host | <--·                     (switch-like)
                | bridge    | <--·
                                  \     | vnet |       uses SLAAC to
Pre-existing                       ·--> | jail |       gain IPv6
Does SLAAC                                             Connectivity
Click to expand...
 
rr0s said:
With this setup, it gets assigned the IPv4 address 192.168.4.14, and can ping the jail host and external IPs traffic via IPv4.
Click to expand...

Does this mean that you fixed your issue?--i'm a little confused because of the thread title. If it's not, I would confirm that things work with IPv4 first then with IPv6.
 
To clarify - the host is unreachable over IPv6, which is the problem. IPv4 works pretty readily.

Maybe I misunderstood the thread title, but it seems to state that the host is unreachable via IPv6 only. The linked article explains how to setup a IPv6 jail.

I can access both the host and public networks via IPv4. This is not the issue.
I can access public networks via IPv6, but not the host via IPv6. This is the issue.

Connecting via IPv4 is not the problem, connecting via IPv6 is.
 
Okay. I have it resolved - I'll leave my solution here for anybody who runs into this issue.

At the root of it, the ICMP6 Neighbor Solicitations weren't being responded to, and that's because the host had it's address assigned on the interface that was what was being bridged to.
The address needs to be applied to the bridge, whether it's static or using SLAAC. This is not the case with IPv4 as there are scoping rules around IPv6 that I don't quite fully understand.

The host's /etc/rc.conf was changed to manage the network address via the bridge:
Code: 
rtsold_enable="YES"
rtsold_flags="-aF"

cloned_interfaces="bridge0 tun tap"
ifconfig_bridge0="addm bge3 addm tap0 up"

# this sets up the IPv6 network on bridge0 rather than bge3
# - allowing network solicitations to be responded to
ifconfig_bridge0_ipv6="inet6 accept_rtadv auto_linklocal"

/etc/sysctl.conf was unchanged, but these are the pertinent settings:

Code: 
net.inet6.ip6.forwarding=1

And the jails /etc/rc.conf stayed the same as well:
Code: 
ifconfig_epair14b_inet6="inet6 accept_rtadv auto_linklocal"
rtsold_enable="YES"

Pretty standard jail.conf:

Code: 
foo {
    $id = 14;
    $addr = "192.168.4.${id}";
    $mask = "255.255.255.0";
    $gw = "192.168.4.1";

    vnet;
    vnet.interface = "epair${id}b";

    exec.prestart = "ifconfig epair${id} create up";
    exec.prestart += "ifconfig epair${id}a up descr vnet-${name}";
    exec.prestart += "ifconfig bridge0 addm epair${id}a up";

    exec.start    = "/sbin/ifconfig lo0 127.0.0.1 up";
    exec.start    += "/sbin/ifconfig epair${id}b ${addr} netmask ${mask} up";
    exec.start    += "/sbin/route add default ${gw}";
    exec.start    += "/bin/sh /etc/rc";

    exec.poststop = "ifconfig bridge0 deletem epair${id}a";
    exec.poststop += "ifconfig epair${id}a destroy";

    host.hostname = foo;
    path = /usr/local/jails/${name}/root;
    exec.consolelog = /var/log/jails/foo_console.log;
    mount.devfs;
    devfs_ruleset = 7;
    allow.sysvipc = 1;
    allow.mlock = 1;
    allow.raw_sockets = 1;
    allow.mount.tmpfs = 1;
    persist;
}

The jail is started with sudo service jail onestart foo, and SLAAC handles the rest. I'm able to ping the host via IPv6, and IPv4.
 
Sorry - typo in that last message:


The jail's /etc/rc.conf should be:
Code: 
ifconfig_epair14b_ipv6="inet6 accept_rtadv auto_linklocal"
rtsold_enable="YES"
 
You must log in or register to reply here.
Back
Top