Fixing network access in Jail

I am mostly a Linux user, and my knowledge on FreeBSD is quite limited (learning now, as I need to use some FreeBSD servers). On some of them, when I create a jail, it doesn't have networking access, even if I configure it correctly (well, I would say it is correct, but if it doesn't work, something is missing).

Follows details. Not sure if I post everything needed. Let me know if I can help debugging the issue.

Jail has IP 10.20.0.13. It is correctly defined in the network interface:

Code:
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether xxx
        hwaddr xxx
        inet 10.20.0.1 netmask 0xffffffff broadcast 10.20.0.1
        inet 10.20.0.4 netmask 0xffffffff broadcast 10.20.0.4
        inet 10.20.0.13 netmask 0xffffffff broadcast 10.20.0.13

Note that some of the other configured jails have networking, so the interface should be working correctly.

My ezjail-admin config is:

Code:
export jail_pdfs2_hostname="pdfs2"
export jail_pdfs2_ip="bce0|10.20.0.13"
export jail_pdfs2_rootdir="/usr/jails/pdfs2"
export jail_pdfs2_exec_start="/bin/sh /etc/rc"
export jail_pdfs2_exec_stop=""
export jail_pdfs2_mount_enable="YES"
export jail_pdfs2_devfs_enable="YES"
export jail_pdfs2_devfs_ruleset="devfsrules_jail"
export jail_pdfs2_procfs_enable="YES"
export jail_pdfs2_fdescfs_enable="YES"
export jail_pdfs2_image=""
export jail_pdfs2_imagetype=""
export jail_pdfs2_attachparams=""
export jail_pdfs2_attachblocking=""
export jail_pdfs2_forceblocking=""
export jail_pdfs2_zfs_datasets=""
export jail_pdfs2_cpuset=""
export jail_pdfs2_fib=""
export jail_pdfs2_parentzfs=""
export jail_pdfs2_parameters=""
export jail_pdfs2_post_start_script=""
export jail_pdfs2_retention_policy=""

I have [FONT=Courier New]/etc/resolv.conf[/FONT] in the jail configured with the same IP as other working jails.

The firewall (pf) is allowing all outgoing traffic...

[FONT=Courier New]netstat -r[/FONT] result is similar for both working/non-working jails

No idea where else I should look. Thanks
 
It may be best to show the output of ping, for example pinging other addresses that are assigned on your machine
 
I can't ping because I do not have raw sockets enabled. But I am able to contact other jails using curl, for example.
But if I try to contact an external website using the domain, I get a 'can't resolve host'.
If I try to do the same by IP, the connection hangs, up to timeout.
Thanks!
 
What are your other network interfaces besides bce0? And in particular is NAT being performed on some interface on this machine? In any case I would tcpdump the interfaces that you expect traffic from this jail to traverse on its way to the internet. Then observe what happens to the traffic.

It would probably help you that you enable raw sockets temporarily to assist in debugging.

It may also be good to provide some of the output of netstat -r. I am not an expert.
 
It does not help not being the person responsible for the machine configuration. I just joined recently, and need to understand how things are configured.
As far as I could understand, googling around, NAT is not enabled (I can't find anything similar to what is described here: https://www.freebsd.org/doc/handbook/firewalls-ipfw.html).
From outside, HTTP is being redirected using squid.

Other interfaces: bce1, igb0, igb1, pflog0 and two loop devices (lo0 and lo1). I tried to configure one jail following ezjail-admin documentation, creating a loop device (lo1) but the jail ended up without external access. bce1 is also configured, but with an IP in other group (10.0.0.X).

My experience with tcpdump is not much. Tried this:
Code:
@Host:
sudo tcpdump  -i bce0 host XX.XX.XX.XX   # public IP of other machine, other continent
@Jail:
telnet XX.XX.XX.XX  22
and tcp dump shows nothing...
 
Got some time to continue in my fight with this jail. I enabled icmp.

From the jail, I can ping the host:
Code:
[root@pdfs2 /]# ping 10.20.0.1
PING 10.20.0.1 (10.20.0.1): 56 data bytes
64 bytes from 10.20.0.1: icmp_seq=0 ttl=64 time=0.039 ms

Also, traceroute works, as expected
Code:
[root@pdfs2 /]# traceroute 10.20.0.1 
traceroute to 10.20.0.1 (10.20.0.1), 64 hops max, 40 byte packets
 1  10.20.0.1 (10.20.0.1)  0.046 ms  0.028 ms  0.019 ms

But I can't ping or traceroute anything outside:
Code:
[root@pdfs2 /]# ping  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss
[root@pdfs2 /]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  * * *
 2  * * *
 3  *^C

My netstat routing info inside the jail:
Code:
[root@pdfs2 /]# netstat -r
netstat: kvm not available: /dev/mem: No such file or directory
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
pdfs2.research.an link#1             UHS         lo0

note that the destination name is the hostname of the jail, defined in /etc/hosts

pf, in the host, has skip for this network interface.

by the way, netstat routing info in the host:

Code:
[ambs@research ~]$ netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.0.1           UGS        bce1
10.0.0.0/24        link#2             U          bce1
research           link#2             UHS         lo0
...

and I still couldn't find any difference in the config of the two jails, the existing one that has net access, and the one I created, which doesn't.
 
You posted ifconfig of bce0 but only routes regarding bce1 - the whole picture would make things easier. :)

I am not very experienced with jails myself, but I would guess that you are missing NAT of your jail IP via the machine's IP. Without this, your jails sends out packets coming from 10.20.0.13, so you would need to tell your gateway that your host is responsible for 10.20.0.13 (or /24) in order to route answer traffic back to you.

Best regards
 
Your jails seem to bound in bce0. However, your gateway appears to be bce1. It would help if you could post the full output of #ifconfig.

Most likely you are missing a NAT rule. Something like that should work:

Code:
nat pass on bce1 from 10.20.0.0/24 to any -> bce1
 
Sorry for not posting the complete ifconfig.

Code:
[ambs@research ~]$ ifconfig -a
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 78:2b:cb:32:61:12
        hwaddr 78:2b:cb:32:61:12
        inet XXX.XX.XXX.XXX netmask 0xffffff00 broadcast XXX.XX.XXX.XXX
        inet 10.20.0.1 netmask 0xffffffff broadcast 10.20.0.1
        inet 10.20.0.4 netmask 0xffffffff broadcast 10.20.0.4
        inet 10.20.0.3 netmask 0xffffffff broadcast 10.20.0.3
        inet 10.20.0.5 netmask 0xffffffff broadcast 10.20.0.5
        inet 10.20.0.6 netmask 0xffffffff broadcast 10.20.0.6
        inet 10.20.0.10 netmask 0xffffffff broadcast 10.20.0.10
        inet 10.20.0.11 netmask 0xffffffff broadcast 10.20.0.11
        inet 10.20.0.12 netmask 0xffffffff broadcast 10.20.0.12
        inet 10.20.0.14 netmask 0xffffffff broadcast 10.20.0.14
        inet 10.20.0.16 netmask 0xffffffff broadcast 10.20.0.16
        inet 10.20.0.13 netmask 0xffffffff broadcast 10.20.0.13
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 78:2b:cb:32:61:13
        hwaddr 78:2b:cb:32:61:13
        inet 10.0.0.241 netmask 0xffffff00 broadcast 10.0.0.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:9b:ea:fc
        hwaddr 00:1b:21:9b:ea:fc
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:9b:ea:fd
        hwaddr 00:1b:21:9b:ea:fd
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.1.1 netmask 0xffffffff
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        groups: lo
 
Your jails seem to bound in bce0. However, your gateway appears to be bce1. It would help if you could post the full output of #ifconfig.

Most likely you are missing a NAT rule. Something like that should work:

Code:
nat pass on bce1 from 10.20.0.0/24 to any -> bce1

Added this to pf, but no joy :( Thanks
 
Sorry, I might be able to help if it were ipfw. I would still say you would want to do something like:


Code:
# tcpdump -i bce0 host x.x.x.x and icmp

and

Code:
# tcpdump -i bce1 host x.x.x.x and icmp

where x.x.x.x is the IP of the jail in question. Then see where the traffic goes. Try pinging a few different addresses, as you already have done, and observe the differences (when does it stop working?). For example, if your default gateway is through bce1, you will expect to see the outgoing pings for 8.8.8.8 going through it on tcpdump.

Keep in mind you can always provide the -S option to ping to explicitly provide the source address you intend (ie the x.x.x.x above) if ever in doubt.
 
Your jails are bound to bce0 but the network is connected to bce1. There's no need for NAT but you do need to enable routing in order to route traffic between the interfaces. The rest of your network also needs to know the route (back) to 10.20.0.0/24.
 
Correct, assuming that 10.0.0.1/32 the gateway, maybe his Internet router, is configured to perform NAT for 10.20.0.0/24, then all he needs is a static route there.

I think that for his FreeBSD box, the only thing he needs is a gateway_enable="YES".
 
Back
Top