RPI3 B some issues with jails

DiscmanDaemon

Member

Reaction score: 16
Messages: 40

Hello all,
I have been experimenting with running a project that works great on AMD64/X86 on an RPI3 B, and have had some hiccups with jails, and I am wondering if anyone has experienced anything similar, and has any idea as to the cause.

I've noticed the standard method of NATting jails on a loopback to the LAN does not seem to work (as covered here). Interestingly, when I tried enabling raw sockets, I can ping, and traceroutes with IMCP, but anything with tcp seems to go nowhere. I do have a dns set in /etc/resolv.conf

Interestingly, if I directly link a jail to the external interface "ue0|some.in-network.ip.address" everything works as expected, so I'm sure something must be up with the NATting.

Here is my /etc/pf.conf
Code:
ext_if="ue0"
jail_if="lo1"
IP_PUB="192.168.1.233"
NET_JAIL="10.0.0.0/24"
scrub in all
# nat all jail traffic
nat pass on $ext_if from $NET_JAIL to any -> $IP_PUB

# passing everything for test
pass out
pass in
Output from ifconfig
Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
        ether b8:27:eb:d3:80:33
        hwaddr b8:27:eb:d3:80:33
        inet 192.168.1.235 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.0.0.1 netmask 0xffffffff
        inet 10.0.0.10 netmask 0xffffffff
        groups: lo
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
If anyone has any suggestions for debugging or remedying this issue, it would be greatly appreciated! it should be noted this exact same setup works great on X86/AMD64 architecture boxes
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,371
Messages: 29,907

The IP addresses don't line up. In your pf.conf you're using 192.168.1.233 but the interface has address 192.168.1.235.

To make this more dynamic use this instead:
Code:
nat pass on $ext_if from $NET_JAIL to any -> ($ext_if)
 

Kristof Provost

Member
Developer

Reaction score: 24
Messages: 44

It might be a good idea to tcpdump / wireshark and see if you can work out what's wrong with TCP. I've seen reports of issues with pf and TCP checksums in combination with certain network cards, but I've never been able to get my hands on affected hardware.
 
OP
OP
DiscmanDaemon

DiscmanDaemon

Member

Reaction score: 16
Messages: 40

The IP addresses don't line up. In your pf.conf you're using 192.168.1.233 but the interface has address 192.168.1.235.
My bad on the cut-and-paste. I have this same issue on 2 seperate RPIs and they have 233 and 235 as their addresses.

I am beginning to wonder if this is caused by a mismatch between the world in the jails and the kernel on the pi as they are different versions. I am going to try building world specifically for the version of FreeBSD that is on the RPI and use that as the basejail and see if we get a different result as I have exhausted ideas on PF (I did take a look with wireshark and found nothing useful-wireshark outside the jail showed no sign of the packets and wireshark inside the jail just showed the packets not leaving the jail)

If this is indeed the case, this thread should be moved back to embedded since it would have nothing to do with the PF.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,371
Messages: 29,907

I am beginning to wonder if this is caused by a mismatch between the world in the jails and the kernel on the pi as they are different versions.
That shouldn't be much of a problem as long as the host system's version is higher than then the jail's version. It's perfectly acceptable to have a 11.1-RELEASE host with a 10.3-RELEASE jail. The other way around however (jail is newer than host) is going to cause problems.
 
Top