FreeBSD 11 jail can ping but not access the outside

mthdr

New Member


Messages: 5

Hello,

This question, or those similar to it, appear several times in these forums, but I can not find a solution within those threads; apologies if I have missed something within them and this is a duplicate.

I have setup a jail with ezjail in FreeBSD 11 on a Raspberry pi. The host (rPi) IP is 192.168.1.137 and is connected to a router which is connected to a cable modem. The jail IP is 10.0.0.1.

rc.conf

Code:
hostname="rpi2"

ifconfig_ue0="inet 192.168.1.137 netmask 255.255.255.0"
defaultrouter="192.168.1.1"

sshd_enable="YES"
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

syslogd_flags="-ss"

growfs_enable="YES"
kldxref_enable="YES"

zfs_enable="YES"
jail_enable="YES"

#jail networks
cloned_interfaces="lo1"
ifconfig_lo1_alias0="inet 10.0.0.1 netmask 255.255.255.255"

#enable the gateway and packet filter
gateway_enable="YES"

pf_enable="YES"
jail.conf
Code:
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;

mount.fstab           = "/etc/jail/fstab.${name}";
mount.devfs;
mount.fdescfs;
mount.procfs;

devfs_ruleset         = "4";

#default path
path = "/local/jails/$name";

backuptest {
        path = "/local/backup_jails/$name";
        host.hostname = "backuptest";
        ip4.addr = "10.0.0.1";
        allow.raw_sockets;
        devfs_ruleset = "5";
}
pf.conf
Code:
EXT_IF="ue0"
JAIL_IF="lo1"

IP_PUB="192.168.1.137"
IP_JAIL="10.0.0.1"
NET_JAIL="10.0.0.0/24"

scrub in all

# NAT all jail traffic
nat pass log (all) on $EXT_IF from $NET_JAIL to any -> $IP_PUB


# passing all traffic
pass out
pass in
I can ping any IP from within the jail, but from within the jail cannot connect to any service (ftp, ssh, etc) nor resolve any host; one exception, I can ssh to the host (192.168.1.137). I can ssh from the host to the jail.

from jail
Code:
root@backuptest:/ # ping ftp.freebsd.org
ping: cannot resolve ftp.freebsd.org: Host name lookup failure
root@backuptest:/ # ping 209.94.190.56
PING 209.94.190.56 (209.94.190.56): 56 data bytes
64 bytes from 209.94.190.56: icmp_seq=0 ttl=50 time=46.081 ms
64 bytes from 209.94.190.56: icmp_seq=1 ttl=50 time=44.702 ms
64 bytes from 209.94.190.56: icmp_seq=2 ttl=50 time=42.298 ms
64 bytes from 209.94.190.56: icmp_seq=3 ttl=50 time=42.421 ms
^C
--- 209.94.190.56 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 42.298/43.876/46.081/1.593 ms
root@backuptest:/ # ftp 209.94.190.56
ftp: Can't connect to `209.94.190.56:21': Operation timed out
ftp: Can't connect to `209.94.190.56:ftp'
Is there anything obvious?

Thanks
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 6,934
Messages: 28,872

Code:
root@backuptest:/ # ping ftp.freebsd.org
ping: cannot resolve ftp.freebsd.org: Host name lookup failure
Your /etc/resolv.conf is missing or not correct.
 
OP
OP
M

mthdr

New Member


Messages: 5

Your /etc/resolv.conf is missing or not correct.
My /etc/resolv.conf is the same as the host, containing only the 8.8.8.8 nameserver. If I remove the file or comment out the nameserver line, the lookup failure is immediate; with it in it takes 30 seconds or so before the ping returns with the failure which leads me to believe the network packet containing resolution is not getting back to the jail.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 6,934
Messages: 28,872

Yeah, you seem to have some problems with NAT. Which then results in resolving issues.

Change this one:
Code:
nat pass log (all) on $EXT_IF from $NET_JAIL to any -> $IP_PUB
To this:
Code:
nat on $ext_if from lo1:network to any -> ($ext_if)
And see if that helps. FTP is a rather nasty protocol to test though, it really, really doesn't like NAT (it has to do with the dynamic creation of the data channel). So test using a "simple" TCP based protocol (ssh(1) will be fine, I often use nc(1) for this) and/or ping(8).

Also keep a tcpdump(1) running on your $ext_if, that'll make it easier to see if your NAT works or not.
 
OP
OP
M

mthdr

New Member


Messages: 5

The change to PF didn't help, I think I've tried that configuration before from another similar thread.

Code:
root@rpi2:~ # pfctl -sn
nat pass log (all) on ue0 inet from 10.0.0.1 to any -> (ue0) round-robin
tcpdump -i ue0 gave way too much information, all of it the constant ssh traffic from my login shell. This is what is in the pflog for ping ftp.freebsd.org from the jail.

Code:
root@rpi2:~ # tcpdump -n -e -ttt -r /var/log/pflog
reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file)
 00:00:00.000000 rule 0/0(match): nat out on ue0: 192.168.1.137.64098 > 8.8.8.8.53: 56483+[|domain]
 00:00:00.070829 rule 0/0(match): nat in on ue0: 8.8.8.8.53 > 10.0.0.1.18379: 56483[|domain]
 00:00:05.059846 rule 0/0(match): nat out on ue0: 192.168.1.137.64098 > 8.8.8.8.53: 56483+[|domain]
 00:00:00.025744 rule 0/0(match): nat in on ue0: 8.8.8.8.53 > 10.0.0.1.18379: 56483[|domain]
Thanks
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 6,934
Messages: 28,872

[BGCOLOR=#dee3e7]tcpdump -i ue0[/BGCOLOR] gave way too much information, all of it the constant ssh traffic from my login shell.
Filter it out; tcpdump -ni ue0 not port 22
 
OP
OP
M

mthdr

New Member


Messages: 5

I used this tcpdump -ni ue0 not port 22 and host 192.168.1.137 as broadcast traffic was getting in there; .137 is the FreeBSD host.

From the jail (10.0.0.1)
Code:
root@backuptest:/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=119 time=25.486 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=22.117 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=27.079 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=119 time=29.047 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=119 time=22.166 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 22.117/25.179/29.047/2.725 ms
root@backuptest:/ # ping ftp.freebsd.org
ping: cannot resolve ftp.freebsd.org: Host name lookup failure
the tcpdump output from the host
Code:
19:17:03.892798 IP 192.168.1.1.53027 > 192.168.1.137.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
19:17:03.892919 IP 192.168.1.137 > 192.168.1.1: ICMP 192.168.1.137 udp port 137 unreachable, length 36
19:17:12.774830 IP 192.168.1.137 > 8.8.8.8: ICMP echo request, id 32396, seq 0, length 64
19:17:12.799933 IP 8.8.8.8 > 192.168.1.137: ICMP echo reply, id 32396, seq 0, length 64
19:17:13.835950 IP 192.168.1.137 > 8.8.8.8: ICMP echo request, id 32396, seq 1, length 64
19:17:13.857808 IP 8.8.8.8 > 192.168.1.137: ICMP echo reply, id 32396, seq 1, length 64
19:17:14.869923 IP 192.168.1.137 > 8.8.8.8: ICMP echo request, id 32396, seq 2, length 64
19:17:14.896689 IP 8.8.8.8 > 192.168.1.137: ICMP echo reply, id 32396, seq 2, length 64
19:17:15.911613 IP 192.168.1.137 > 8.8.8.8: ICMP echo request, id 32396, seq 3, length 64
19:17:15.940439 IP 8.8.8.8 > 192.168.1.137: ICMP echo reply, id 32396, seq 3, length 64
19:17:16.940147 IP 192.168.1.137 > 8.8.8.8: ICMP echo request, id 32396, seq 4, length 64
19:17:16.962060 IP 8.8.8.8 > 192.168.1.137: ICMP echo reply, id 32396, seq 4, length 64
19:17:18.734931 IP 192.168.1.137.55668 > 8.8.8.8.53: 56203+ A? ftp.freebsd.org. (33)
19:17:18.768336 IP 8.8.8.8.53 > 192.168.1.137.55668: 56203 2/0/0 CNAME ftp.geo.freebsd.org., A 209.94.190.56 (71)
19:17:23.737150 IP 192.168.1.137.55668 > 8.8.8.8.53: 56203+ A? ftp.freebsd.org. (33)
19:17:23.761191 IP 8.8.8.8.53 > 192.168.1.137.55668: 56203 2/0/0 CNAME ftp.geo.freebsd.org., A 209.94.190.56 (71)
19:17:38.391679 IP 192.168.1.1.53027 > 192.168.1.137.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
19:17:38.391857 IP 192.168.1.137 > 192.168.1.1: ICMP 192.168.1.137 udp port 137 unreachable, length 36
^C
22 packets captured
99 packets received by filter
0 packets dropped by kernel

Thanks
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 6,934
Messages: 28,872

Your tcpdump(1) output shows the correct packets, you can clearly see the DNS requests and answers. So something else is preventing it from working correctly.

I would suggest not using lo1 but instead tie the jail to ue0 and assign it a 192.168.1.0/24 address. You are behind a router that's also doing NAT. Double NAT (once on the FreeBSD host and again on your router) is always tricky to get it correctly working. Without the "extra" NAT you can simply treat the jail as any other machine on your LAN.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 6,934
Messages: 28,872

Cool. The lo1 trick with NAT is a good solution if the host only has one external IP address for example. But as this is on a LAN you're not limited to a single IP.
 
Top