Jail can only occasionally reach internet

S

Spade

Hello all.

I recently purchased a miniture computer to use as a home server, and decided to try FreeBSD out for the first time. Part of my endeavor involved setting up a jail with ezjail and running nginx inside.

Unfortunately, I'm experiencing an issue wherein the jail is only occasionally able to reach the internet but can reach/be reached by computers on the local network. That is to say that for a length of time after restarting the jail, I can download packages and telnet remote IPs, but will invariably lose the capability to do so later on. Restarting the jail seems to fix this for a little bit.

I've followed the manpage on jails as best as I could and have tried configuring things as directed by quite a few different tutorials, all to no avail, so I would appreciate any help.

Below are snippets from the host machine that I believe to be relevant:

Code: 
/etc/rc.conf
----------------
hostname="machinename.mydomain.net"
wlans_iwm0="wlan0"
ifconfig_wlan0="WPA DHCP"
cloned_interfaces="lo1"
ifconfig_lo1_aliases="\
    inet 192.168.50.181 "
sshd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
jail_enable="YES"
ezjail_enable="YES"

Code: 
/etc/hosts
----------------
127.0.0.1               localhost mydomain.net

Code: 
/etc/resolv.conf
----------------
nameserver 192.168.50.1

Code: 
/usr/local/etc/ezjail/webserver
----------------
export jail_webserver_hostname="webserver"
export jail_webserver_ip="lo1|127.0.1.1,wlan0|192.168.50.181"
export jail_webserver_rootdir="/usr/jails/webserver"
export jail_webserver_exec_start="/bin/sh /etc/rc"
export jail_webserver_exec_stop=""
export jail_webserver_mount_enable="YES"
export jail_webserver_devfs_enable="YES"
export jail_webserver_devfs_ruleset="devfsrules_jail"
export jail_webserver_procfs_enable="YES"
export jail_webserver_fdescfs_enable="YES"
export jail_webserver_image=""
export jail_webserver_imagetype=""
export jail_webserver_attachparams=""
export jail_webserver_attachblocking=""
export jail_webserver_forceblocking=""
export jail_webserver_zfs_datasets=""
export jail_webserver_cpuset=""
export jail_webserver_fib=""
export jail_webserver_parentzfs=""
export jail_webserver_parameters=""
export jail_webserver_post_start_script=""
export jail_webserver_retention_policy=""

Code: 
ifconfig output
----------------
em0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 6c:4b:90:45:9e:5f
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:24:d6:eb:e8:c1
        inet 192.168.50.180 netmask 0xffffff00 broadcast 192.168.50.255
        inet 192.168.50.181 netmask 0xffffffff broadcast 192.168.50.181
        groups: wlan
        ssid CurrentNetwork channel 161 (5805 MHz 11a) bssid 04:d9:f5:91:de:8c
        regdomain FCC country US authmode WPA2/802.11i privacy ON
        deftxkey UNDEF AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 23 bmiss 10
        mcastrate 6 mgmtrate 6 scanvalid 60 wme roaming MANUAL
        parent interface: iwm0
        media: IEEE 802.11 Wireless Ethernet OFDM/54Mbps mode 11a
        status: associated
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 192.168.50.181 netmask 0xffffff00
        inet 127.0.1.1 netmask 0xffffffff
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x4
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Code: 
netstat -rn output
----------------
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.50.1       UGS       wlan0
127.0.0.1          link#2             UH          lo0
127.0.1.1          link#4             UH          lo1
192.168.50.0/24    link#3             U         wlan0
192.168.50.180     link#3             UHS         lo0
192.168.50.181     link#4             UH          lo1

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           URS         lo0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           URS         lo0
fe80::/10                         ::1                           URS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
fe80::%lo1/64                     link#4                        U           lo1
fe80::1%lo1                       link#4                        UHS         lo0
ff02::/16                         ::1                           URS         lo0


And below are the jail's files and command outputs:

Code: 
/etc/rc.conf
----------------
nginx_enable="YES"

Code: 
/etc/hosts
----------------
127.0.1.1               localhost localhost.my.domain


Code: 
/etc/resolv.conf
----------------
nameserver 192.168.50.1

Code: 
ifconfig output
----------------
em0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 6c:4b:90:45:9e:5f
        media: Ethernet autoselect
        status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:24:d6:eb:e8:c1
        inet 192.168.50.181 netmask 0xffffffff broadcast 192.168.50.181
        groups: wlan
        ssid A_Network-5G channel 161 (5805 MHz 11a) bssid 04:d9:f5:91:de:8c
        regdomain FCC country US authmode WPA2/802.11i privacy ON
        deftxkey UNDEF AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 23 bmiss 10
        mcastrate 6 mgmtrate 6 scanvalid 60 wme roaming MANUAL
        parent interface: iwm0
        media: IEEE 802.11 Wireless Ethernet OFDM/54Mbps mode 11a
        status: associated
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 192.168.50.181 netmask 0xffffff00
        inet 127.0.1.1 netmask 0xffffffff
        groups: lo

Code: 
netstat -rn output
----------------
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.1.1          link#4             UH          lo1
192.168.50.181     link#4             UH          lo1
 
You have an IP conflict. In rc.conf you've set lo1 to 192.168.50.181 (it's also missing the netmask there). In your jail configuration you're binding 192.168.50.181 to wlan0. Remove the 192.168.50.181 address from lo1.
 
SirDice said:
You have an IP conflict. In rc.conf you've set lo1 to 192.168.50.181 (it's also missing the netmask there). In your jail configuration you're binding 192.168.50.181 to wlan0. Remove the 192.168.50.181 address from lo1.
Click to expand...
Thank you for the quick reply!

I wasn't sure whether I should remove the alias line outright, or change it to inet 127.0.1.1 netmask 255.255.255.0, so I tested both (and assume that that is the correct netmask -- I'm new at networking :p).

Unfortunately, the jail still looses its ability to connect to the outside internet a few seconds after being started.

Are there any logs that might be relevant? Is it possible a service is coming up after the jails boot that is preventing an internet connection?
 
Spade said:
I wasn't sure whether I should remove the alias line outright, or change it to inet 127.0.1.1 netmask 255.255.255.0, so I tested both (and assume that that is the correct netmask -- I'm new at networking :p).
Click to expand...
You don't need to do this. The IP addresses will be automatically added when the jail starts and removed again when the jail stops.
 
Thank you for the clarification. Unfortunately, the issue persists.

Would it be more workable just to do as this tutorial suggests and inherit the host's network stack?
 
SirDice said:
You don't need to do this. The IP addresses will be automatically added when the jail starts and removed again when the jail stops.
Click to expand...
I have to include that using vnet jails with vnet.interface and the /usr/share/examples/jails/jib scripts in exec.* to add and remove bridged epairs will likely result in a kernel panic (within 10 seconds) and restart if the jail fails to start but the epair is still moved into the jail. Very inconvenient to script that manually :(.
 
Okay, I believe that I've solved the issue.

Apparently, I still had a leftover /etc/jail.conf file from before the manpage recommended ezjail. Confusingly, this file had no networking configuration in it, but still seems to have been the root cause of my problem.

I will report back a couple hours from now to mark this thread as solved if the jail stays accessible. Thank you all for your help.

Just as I posted this, port 80 became inaccessible again and I lost the ability to install packages to the jail. It should be noted that the network was accessible for more than ten minutes.

EDIT 2: The connection revived itself after a short stint, which it has never done, so I think it's an improvement.
 
Spade said:
Just as I posted this, port 80 became inaccessible again and I lost the ability to install packages to the jail. It should be noted that the network was accessible for more than ten minutes.
Click to expand...
You know, it sounds like you may have more IP conflicts. Are you sure the 192.168.50.181 address is free to use?
 
The problem is that he's using Wifi. Usually in wlan in station (client) only one mac address is allowed so if you have virtual machines or other devices connected behind the same wifi they can't communicate at the same time. To avoid this you have to use wifi in bridge mode (WDS) or AP controller which support passive-clients.

If you can avoid using wlan and switch to ethernet if not then config your jail to be behind NAT.
 
VladiBG said:
The problem is that he's using Wifi. Usually in wlan in station (client) only one mac address is allowed so if you have virtual machines or other devices connected behind the same wifi they can't communicate at the same time.
Click to expand...
True, but in this case there's only one MAC address (jail is bound to wlan0 directly).
 
SirDice said:
You know, it sounds like you may have more IP conflicts. Are you sure the 192.168.50.181 address is free to use?
Click to expand...

I was almost certain it was, but lo and behold, I am a fool.

Changing the IP to 192.168.50.182 has seen the issue fixed for >30 minutes now. Thank you for the suggestion.
 
Spade said:
I was almost certain it was, but lo and behold, I am a fool.
Click to expand...
Not a fool, just not familiar with networking. So an IP conflict is an easy mistake to make.

Spade said:
Changing the IP to 192.168.50.182 has seen the issue fixed for >30 minutes now.
Click to expand...
That might work fine until the DHCP server issues that address to something else. Check your DHCP server, most of the time you can configure it to serve addresses in a specific range, 100-200 for example. That means it will never give out addresses 1-99 and 200-254. Then you can use those addresses for static configurations. Or add an address as a reserved IP address for that host. Number one rule of networking, all hosts must have a unique address.
 
You must log in or register to reply here.
Back
Top