Solved Jails that can reach the Internet with IP assigned by the home modem/router

S

Sam The Ripper

Hi everyone,
I'm doing a pet project where I'm designing the topology of what I'm going to implement on my physical server when I'll buy it.
So, right now, I'm working in VMware Fusion, just to play with it and understand how things can be configured.

But before exploring the topic, I'll give you some constraints/context:
  1. This lab is ARM-based because ARM laptops are all I have here, since I'm not in my main house.
    That said, the physical server will be amd64. So if there are something that is still not supported on ARM, feel free to let me know! :)
  2. I won't use *ANY* utility from Ports in the FreeBSD server instance, unless they are mandatory. I want to use only the utilities provided in base. Packages/Ports will be installed in Jails accordingly to their use.
  3. Jails contain only the base set. Nothing else.
OK, let's go to the details of my problems.
I set up two VMs in VMWare Fusion, a server and a client, just to fake my home configuration where there will be my laptops and the server.
The server is a FreeBSD VM and the client is a NetBSD VM. I chose two different OSes because I want to emulate my heterogeneous target environment.

Let's focus on the server VM, since NetBSD was just installed and configured with DHCP, just to check the connectivity via SSH towards the jail. So there is nothing special configured in it.

The server VM has ONE "physical" interface, called em0.
What I want to achieve is to let the jail lease the IP from the DHCP of my modem/router, and let it connect to the internet. The jail IP will always be the same, as I'm going to reserve it.

VMWare Fusion create a NAT network between the VMs and the laptop, so it's similar to how things will be in my house.
Since I want to do things step by step, for now I'm not going to use DHCP. In fact, I'm going to set the IP in the jail configuration, to check if everything is working properly.
So I created the jail with the following configurations:
Bash: 
# cat /etc/jail.conf.d/test_ldep_local.conf
test {
     allow.mount;
     mount.devfs;
     vnet;
     jid              =     "2";
     path             =     "/usr/jails/containers/test.ldep.local";
     host.hostname    =     "test.ldep.local";
     vnet.interface  +=     "epair0b";
     exec.prestart   +=     "ifconfig epair0 create up";
     exec.prestart   +=     "ifconfig bridge0 addm epair0a";
     exec.start       =     "/bin/sh /etc/rc";
     exec.start      +=     "ifconfig epair0b inet 172.16.47.240 netmask 255.255.255.0";
     exec.start      +=     "route add default 172.16.47.1";
     exec.stop        =     "/bin/sh /etc/rc.shutdown jail";
     exec.poststop   +=     "ifconfig epair0a destroy";
}

The bridge called bridge0 is configured like this at the boot time:
Bash: 
# cat /etc/rc.conf
ifconfig_em0="up"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 up SYNCDHCP"

So bridge0 is going to lease from the VMware Fusion's built-in DHCP server the IP instead of the physical interface.
From the jail I can reach the NetBSD VM via SSH, and from NetBSD I can reach the jail via SSH.
Here's the output from FreeBSD:
Bash: 
root@homeserver:~ # service jail onestart
Starting jails: test.
root@homeserver:~ # jexec test sh
root@test:/ # ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:b8:6c:cc:56:0b
    inet 172.16.47.240 netmask 0xffffff00 broadcast 172.16.47.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@test:/ # nc -v 172.16.47.129 22
Connection to 172.16.47.129 22 port [tcp/ssh] succeeded!
SSH-2.0-OpenSSH_9.6 NetBSD_Secure_Shell-20231220-hpn13v14-lpk

and here's the output from NetBSD:
Bash: 
$ ifconfig
wm0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
    capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
    capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
    enabled=0
    ec_capabilities=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
    ec_enabled=0x2<VLAN_HWTAGGING>
    address: 00:0c:29:a8:83:8b
    media: Ethernet autoselect (1000baseT full-duplex,master)
    status: active
    inet6 fe80::2e74:fb89:34e1:5c4b%wm0/64 flags 0 scopeid 0x1
    inet 172.16.47.129/24 broadcast 172.16.47.255 flags 0
lo0: flags=0x8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33624
    status: active
    inet6 ::1/128 flags 0x20<NODAD>
    inet6 fe80::1%lo0/64 flags 0 scopeid 0x2
    inet 127.0.0.1/8 flags 0
$ nc -v 172.16.47.240 22
Connection to 172.16.47.240 22 port [tcp/ssh] succeeded!
SSH-2.0-OpenSSH_9.7 FreeBSD-20240318

So everything is working, right? Well, not so much.
While Jails<===>VMs connection is working properly, the Jail can't reach the internet.
Bash: 
root@test:/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

Also, I can't lease the IP from the VMware Fusion's DHCP server.
Bash: 
root@test:/ # ifconfig epair0b inet 172.16.47.240 delete
root@test:/ # dhclient epair0b
Can't find free bpf: No such file or directory
exiting.


I'm sure I'm overlooking something, but right now I don't have any more ideas, so here I am asking you. :)


Cheers!
 
Ok, I found the problem.
It was actually easy to fix, but my mind was too tired for the too much trials and errors.

The error "Can't find free bpf: No such file or directory" was actually telling me everything, since it seems like there was no devfs mounted in the jail.

Here is how I fixed it:
Bash: 
# cat /etc/jail.conf.d/test_ldep_local.conf
test {
     allow.mount;
     mount.devfs;
     vnet;
     jid              =     "2";
     path             =     "/usr/jails/containers/test.ldep.local";
     host.hostname    =     "test.ldep.local";
     vnet.interface  +=     "epair0b";
     exec.prestart    =     "mount -t devfs devfs /usr/jails/containers/test.ldep.local/dev";
     exec.prestart   +=     "ifconfig epair0 create";
     exec.prestart   +=     "ifconfig epair0a ether 02:09:52:3a:a0:0a up";
     exec.prestart   +=     "ifconfig epair0b ether 02:09:52:3a:a0:0b up";
     exec.prestart   +=     "ifconfig bridge0 addm epair0a";
     exec.start       =     "/bin/sh /etc/rc";
     exec.start      +=     "dhclient epair0b";
     exec.stop        =     "/bin/sh /etc/rc.shutdown jail";
     exec.poststop    =     "umount /usr/jails/containers/test.ldep.local/dev";
     exec.poststop   +=     "ifconfig epair0a destroy";
}

I'm spoofing the MAC Address so I can lease the exact IP I'm going to reserve, so the interface will have ether and hwaddr different, but everything is working properly.

Now the jail can reach internet.

Do you have any tips or something to improve the configuration?
 
You must log in or register to reply here.
Back
Top