Issue accessing Internet from jail

ppbitb

ppbitb

Hi folks,

I am using this script to setup networking and a jail and test connectivity to the Internet. Everything works except the last ping 8.8.8.8 from the jail. What could be wrong?

Bash: 
# On host hn0 has ip address 10.0.0.4 and can ping 8.8.8.8

service pf stop
ifconfig bridge create
ifconfig bridge0 inet 192.0.2.254/24 up
ifconfig epair create
jail -c -f /etc/jail.conf.d/d2d_prod.conf
ifconfig bridge0 addm hn0 addm epair0a up
ifconfig epair0a inet 192.0.2.1/24 up
jexec d2d_prod ifconfig epair0b inet 192.0.2.2/24 up
jexec d2d_prod route add default 192.0.2.254 # Also tried 192.0.2.1 as gateway with no success

# Not sure if this is needed but doesn't hurt
ifconfig bridge0 up
ifconfig epair0a up
ifconfig hn0 up

# Throwing spaghetti at the wall at this point
sysctl net.inet.ip.forwarding=1

# All this works
jexec d2d_prod ping -c 4 192.0.2.2
jexec d2d_prod ping -c 4 192.0.2.1
jexec d2d_prod ping -c 4 192.0.2.254
jexec d2d_prod ping -c 4 10.0.0.4

# Doesn't work
jexec d2d_prod ping -c 4 8.8.8.8

Here is the content of /etc/jail.conf.d/d2d_prod.conf
Bash: 
d2d_prod {
        host.hostname = d2d_prod.example.com;
        path = "/usr/jails/d2d_prod";
        exec.clean; # Clear environment variables
        exec.system_user = "root";
        exec.jail_user = "root";
        vnet;
        vnet.interface = "epair0b";
        allow.raw_sockets; # allows pinging
        mount.devfs;
        exec.start = "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown";
        allow.set_hostname = 1;
}
 
Did you enable routing on the host? sysrc gateway_enable="YES"
 
It should have the same outcome, yes. Don't set the sysctl though, every FreeBSD admin is going to expect to see gateway_enable. You can only ping local IP addresses, and nothing appears to be routed beyond the host. Hence the question regarding routing.

On the host, can you post the output of ifconfig hn0, ifconfig bridge0 and netstat -rn?

Oh, and I suddenly realized the issue. Things are probably routed out of the host on hn0 (you can verify this with tcpdump(1)). Assuming there's an upstream router that's actually routing to the internet, does it know where 192.0.2.0/24 is? Your packets are probably going out but the router has no idea what to do with the return packets. And thus you never receive a response. Also note that 192.0.2.0/24 is not a private range, it's a range specifically intended for documentation.

 
Enable mac address spoofing on the Hyper-V host for that virtual machine (network adapter -> advanced fetures -> Enable MAC address spoofing)
Your router at 10.0.0.1 need to have route for 192.0.2.0/24 via gw 10.0.0.4. Otherwise you need to assign a 10.0.0.0/24 address so the jail can talk directly to your router instead of routing the traffic via 10.0.0.4
 
I don't know much about jails, but all I needed to access the Internet was to include:-

Code: 
   ip4.addr = 192.168.1.221;                   # IP address of the jail
    interface = em0;

in /etc/jail.conf for the jail, where the host's address is 192.168.1.21.
 
Don't set the sysctl though, every FreeBSD admin is going to expect to see gateway_enable
Click to expand...
Ack. I was planning to use the sysrc method to get the settings to survive reboots, once it works.

On the host, can you post the output of ifconfig hn0, ifconfig bridge0 and netstat -rn
Click to expand...
Bash: 
[root@pfreebsd ~]# ifconfig hn0
hn0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80019<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
        ether 60:45:bd:6f:44:5e
        inet 10.0.0.4 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet 100GBase-CR4 <full-duplex,rxpause,txpause>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
[root@pfreebsd ~]#
[root@pfreebsd ~]# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:bd:71
        inet 192.168.2.254 netmask 0xffffff00 broadcast 192.168.2.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: hn0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
[root@pfreebsd ~]#
[root@pfreebsd ~]# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.0.1           UGS         hn0
10.0.0.0/24        link#2             U           hn0
10.0.0.4           link#2             UHS         lo0
127.0.0.1          link#1             UH          lo0
168.63.129.16      10.0.0.1           UGHS        hn0
169.254.169.254    10.0.0.1           UGHS        hn0
192.168.2.0/24     link#5             U       bridge0
192.168.2.1        link#6             UHS         lo0
192.168.2.254      link#5             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           URS         lo0
::1                               link#1                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           URS         lo0
fe80::/10                         ::1                           URS         lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
ff02::/16                         ::1                           URS         lo0
[root@pfreebsd ~]#

Things are probably routed out of the host on hn0 (you can verify this with tcpdump(1))
Click to expand...
Yes and I verified that I could see them
Bash: 
[root@pfreebsd ~]# tcpdump -ln -i hn0 | grep -v <my-ip>
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on hn0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:40:35.969101 IP 192.168.2.2 > 8.8.8.8: ICMP echo request, id 9554, seq 132, length 64
19:40:36.995299 IP 192.168.2.2 > 8.8.8.8: ICMP echo request, id 9554, seq 133, length 64
19:40:38.019063 IP 192.168.2.2 > 8.8.8.8: ICMP echo request, id 9554, seq 134, length 64
19:40:39.055325 IP 192.168.2.2 > 8.8.8.8: ICMP echo request, id 9554, seq 135, length 64

Assuming there's an upstream router that's actually routing to the internet, does it know where 192.0.2.0/24 is? Your packets are probably going out but the router has no idea what to do with the return packets.
Click to expand...
This is something I have been wondering about as I read tutorial after tutorial talking about VNET, epair and bridge. Don't I need something like NAT so that returning packets can be forwarded back to the source? Unless the bridge handles that automatically?
Maybe I do not understand the purposes and conditions under which the tutorials are applicable.
Tutorials I have been following, in no particular order:

Also note that 192.0.2.0/24 is not a private range, it's a range specifically intended for documentation.
Click to expand...
Ah, thank you, I did not know that. I have updated my script to use 192.168.2.xx.

VladiBG said:
Enable mac address spoofing on the Hyper-V host for that virtual machine (network adapter -> advanced fetures -> Enable MAC address spoofing)
Click to expand...
Hmm, thanks but what are you talking about? Although this happens to be an Azure VM, so it might be running Hyper-V!

balanga said:
I don't know much about jails, but all I needed to access the Internet was to include:-

Code: 
   ip4.addr = 192.168.1.221;                   # IP address of the jail
    interface = em0;

in /etc/jail.conf for the jail, where the host's address is 192.168.1.21.
Click to expand...
I don't think this would work. Azure gives me the single private IP 10.0.0.4 (in addition to a public IP). I don't think it will happily let me use another IP in the same network.
 
Thanks to all who responded, I got it to work with NAT. No change to Azure.
Final working script:
Bash: 
sysctl net.inet.ip.forwarding=1
ifconfig bridge create
ifconfig bridge0 inet 192.168.2.254/24 up
ifconfig epair create
jail -c -f /etc/jail.conf.d/d2d_prod.conf

ifconfig bridge0 addm hn0 addm epair0a up

ifconfig epair0a inet 192.168.2.1/24 up

jexec d2d_prod ifconfig epair0b inet 192.168.2.2/24 up

jexec d2d_prod route add default 192.168.2.254

ifconfig bridge0 up
ifconfig epair0a up
ifconfig hn0 up

Added the following to pf.conf:
Bash: 
nat pass on hn0 inet from 192.168.2.0/24 to any -> (hn0)
 
You must log in or register to reply here.
Back
Top