Solved No internet access from inside jail!

idude

idude

I've been messing around with jails and I'm able to setup a jail and have access to the net from inside of it using the "inherit" for ip4. Now I want to assign a different IP address for each jail I create but I loose access to the internet and I'm not able to install any packages in them.
I'm not using any jail managers like qjail or ezjail. I'm running FreeBSD 12.2 using UFS. My jail.conf looks like this:
Code: 
exec.start="/bin/sh /etc/rc";
exec.stop="/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

mariadb {
        host.hostname="mariadb.mybsd.io";
        ip4.addr="192.168.1.81";
        path="/usr/jail/mariadb";
}

httpd {
        host.hostname="httpd.mybsd.io";
        ip4.addr="192.168.1.82";
        path="/usr/jail/httpd";
}

I have these lines in the host rc.conf:
Code: 
jail_enable="YES"
jail_list="mariadb httpd"
jail_reverse_stop="YES"

Copied host's /etc/resolv.conf and /etc/localtime to each /etc directory of the jails.

When I try to install a package using pkg -j mariadb install -y mariadb105-server I get this error:

Code: 
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly/Latest/pkg.txz: Non-recoverable resolver failure
Address resolution failed for http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly.
Consider changing PACKAGESITE.

My host ifconfig looks like this:
Code: 
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 78:ac:c0:3f:bc:d2
    inet6 fe80::7aac:c0ff:fe3f:bcd2%re0 prefixlen 64 scopeid 0x1
    media: Ethernet autoselect (none)
    status: no carrier
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 4c:0f:6e:37:9b:10
    inet 192.168.1.78 netmask 0xffffff00 broadcast 192.168.1.255
    groups: wlan
    ssid MEO-62B190 channel 1 (2412 MHz 11g ht/20) bssid 00:06:91:62:b1:90
    regdomain 96 indoor ecm authmode WPA2/802.11i privacy ON
    deftxkey UNDEF TKIP 2:128-bit txpower 20 bmiss 7 scanvalid 60
    protmode CTS ampdulimit 64k ampdudensity 4 shortgi -stbctx stbcrx
    -ldpc -uapsd wme burst roaming MANUAL
    parent interface: ath0
    media: IEEE 802.11 Wireless Ethernet MCS mode 11ng
    status: associated
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

My jail's ifconfig looks like this:
Code: 
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 78:ac:c0:3f:bc:d2
    media: Ethernet autoselect (none)
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    groups: lo
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 4c:0f:6e:37:9b:10
    groups: wlan
    ssid MEO-62B190 channel 1 (2412 MHz 11g ht/20) bssid 00:06:91:62:b1:90
    regdomain 96 indoor ecm authmode WPA2/802.11i privacy ON
    deftxkey UNDEF TKIP 2:128-bit txpower 20 bmiss 7 scanvalid 60
    protmode CTS ampdulimit 64k ampdudensity 4 shortgi -stbctx stbcrx
    -ldpc -uapsd wme burst roaming MANUAL
    parent interface: ath0
    media: IEEE 802.11 Wireless Ethernet MCS mode 11ng
    status: associated

What am I missing here?
 
Yes, remove or comment out "jail_reverse_stop="YES"".
It does the similar thing, but option named as "depend" is more flexible. Especially with a few sets of depended jails.
 
astyle said:
I tried adding that to the jail's /etc/rc.conf after following the instructions in https://wiki.freebsd.org/JailingGUIApplications, but no go... Firefox starts (very slowly), but it can't access the Internet.

Dunno what I'm missing in that setup...
Click to expand...

You add interface = wlan0; to the relevant block in the host's /etc/jail.conf, not the jail's /etc/rc.conf.

Whenever you have network issues, start from the ground up. "Firefox can't access the internet" could be caused by a ton of different things. My basic starting point is:

1. Can I ping my gateway?
2. Can I ping 8.8.8.8?
3. Can I resolve google.com?
 
patmaddox said:
You add interface = wlan0; to the relevant block in the host's /etc/jail.conf, not the jail's /etc/rc.conf.

Whenever you have network issues, start from the ground up. "Firefox can't access the internet" could be caused by a ton of different things. My basic starting point is:

1. Can I ping my gateway?
2. Can I ping 8.8.8.8?
3. Can I resolve google.com?
Click to expand...
Confirmed presence of interface = wlan0; in host's /etc/jail.conf. Double-checked the jail's /etc/rc.conf - it's clean.

The answer to all 3 is no.

pings to any IP just react with ping: ssend socket: Operation not permitted

Firefox responds with "Server not found" when I try to point it to google.com

I'm thinking, one way to troubleshoot is to set up a successful example following the Handbook, and then see what are the differences between my failed setup and that successful example. But if someone can help with basic pointers like yours, that's really nice, because I don't always know what to look for, and why. Even after doing the RTFM myself, sometimes all it takes is someone more knowledgeable than me pointing me in the right direction.
 
Ah, yeah, by default jails don't let you ping from inside them. You can set allow.raw_sockets in the jail's config to allow it.

Yes, setting up a successful example is always a good idea :) I figured you were having trouble doing that...

Have you got a successful configuration going at all? If not - post your /etc/jail.conf and we can start from there.
 
patmaddox said:
Ah, yeah, by default jails don't let you ping from inside them. You can set allow.raw_sockets in the jail's config to allow it.

Yes, setting up a successful example is always a good idea :) I figured you were having trouble doing that...

Have you got a successful configuration going at all? If not - post your /etc/jail.conf and we can start from there.
Click to expand...
Code: 
# /etc/jail.conf

allow.nomount;
exec.clean;
mount.devfs;
host.hostname = "$name.your-host-name.lan";
path = "/zroot/jails/${name}/root";
#securelevel = 3;
# security.jail.allow_raw_sockets=1;
interface = "wlan0";

firefox {
    ip4.addr = "10.0.0.2";
   exec.prestart = "mount | grep ' on /zroot/jails/${name}/root/tmp/.X11-unix` || mount_nullfs /tmp/.X11-unix /zroot/jails/${name}/root/tmp/.X11-unix"
    exec.start = "/bin/sh /home/firefox/run-firefox";
    exec.jail_user = "firefox";
    # persist;
    devfs_ruleset = 5;
    # interface = "wlan0";
}

I tried playing with allow.raw_sockets, but could not get the syntax correct, probably, the jail would not start, so I had to comment it out.

The Handbook does mention a few sysctls related to jails in Section 16.4.1. System Tools for Jail Tuning in FreeBSD . I think those are supposed to be set on the host. I checked mine - all untouched, and they match what's in the Handbook.
 
gotnull said:
In my case with VNET jails, when there is that type of message, the firewall is involved.
Try to add this in your jail's rc.conf, reboot it and see if it does something:
Click to expand...
Tried that with my jail, no go, Same error, even after restarting the jail, and unable to get even to Google on the jailed Firefox.
 
For allow.raw_sockets, put it right in /etc/jail.conf either at top level or per jail. e.g.

Code: 
firefox {
  ip4.addr = "10.0.0.2";
  allow.raw_sockets;
  // all the other stuff
}

Also, I would back off of this firefox stuff for the time being, until you get a simple jail working. There's a lot of stuff going on there - nullfs mounts, a custom startup script, a separate jail user.

Start with something straightforward as described in the handbook. Get that working, and then you can build up to your firefox jail.

A couple other things to look into about your /etc/jail.conf:

1. Is 10.0.0.2 an alias IP on wlan0? If not, I would not expect it to work.
2. You might not want a fully-qualified hostname, e.g. just use host.hostname = $name;
 
patmaddox said:
Is 10.0.0.2 an alias IP on wlan0? If not, I would not expect it to work.
Click to expand...
Running ifconfig wlan0 inside the jail yields 10.0.0.2, as expected.

Running ifconfig wlan0 outside the jail yields 10.10.7.116 (assigned by DHCP server at 10.10.0.1) and 10.0.0.2 (assigned by /etc/jail.conf)

The tutorial wiki that was referenced by me earlier - it only mentions aliases for the cloned lo1 interface, not for wlan0... yeah, I guess I can try and see where I can get with the Handbook first. :/

The fixed sockets syntax in /etc/jail.conf stopped the complaints, jail starts, but now ping reacts with ping: Unknown host after a LONG time.
 
When you say ping: Unknown host - which ping command are you talking about specifically? That sounds like it might be DNS resolution, so ping 8.8.8.8 would work, but ping google.com would not. If that's the case, it seems you don't have a good /etc/resolv.conf in your jail. A typical jail behaves almost exactly like a host, with some important exceptions... so you have to configure things like nameservers in a jail same as you do in a host.
 
I'll check the differences and report back. I appreciate the help here! :)
patmaddox said:
When you say ping: Unknown host - which ping command are you talking about specifically? That sounds like it might be DNS resolution, so ping 8.8.8.8 would work, but ping google.com would not. If that's the case, it seems you don't have a good /etc/resolv.conf in your jail. A typical jail behaves almost exactly like a host, with some important exceptions... so you have to configure things like nameservers in a jail same as you do in a host.
Click to expand...
 
patmaddox said:
When you say ping: Unknown host - which ping command are you talking about specifically? That sounds like it might be DNS resolution, so ping 8.8.8.8 would work, but ping google.com would not. If that's the case, it seems you don't have a good /etc/resolv.conf in your jail. A typical jail behaves almost exactly like a host, with some important exceptions... so you have to configure things like nameservers in a jail same as you do in a host.
Click to expand...
OK, finally set up a basic jail following the Handbook, I called it practice... it shows up in jls, I can jexec into it... /etc/resolv.conf is identical both on host and in the jail. But... within that jail:
Code: 
# ping -c 5 google.com
ping: Unknown host
# ping - c 5 8.8.8.8
ping: ssend socket: Operation not permitted
Outside of the jail, these commands are fine.

Trying to restart the jail after uncommenting the allow.raw_sockets in /etc/jails.conf leaves the jail still unable to resolve google.com, but at least doesn't complain if I try to ping 8.8.8.8... It does take an awfully long time to show anything, though - like more than 2 minutes.

Edit: Looks like I just pinged 8.8.8.8 without the -c 5 flag - that's why nothing showed up. Stopping the process showed 100% packet loss.
 
astyle said:
OK, finally set up a basic jail following the Handbook, I called it practice... it shows up in jls, I can jexec into it... /etc/resolv.conf is identical both on host and in the jail. But... within that jail:
Code: 
# ping -c 5 google.com
ping: Unknown host
# ping - c 5 8.8.8.8
ping: ssend socket: Operation not permitted
Outside of the jail, these commands are fine.

Trying to restart the jail after uncommenting the allow.raw_sockets in /etc/jails.conf leaves the jail still unable to resolve google.com, but at least doesn't complain if I try to ping 8.8.8.8... It does take an awfully long time to show anything, though - like more than 2 minutes.

Edit: Looks like I just pinged 8.8.8.8 without the -c 5 flag - that's why nothing showed up. Stopping the process showed 100% packet loss.
Click to expand...
I think you can't ping from a jail by default. To test you connection try a pkg update -f, if you are able to retrieve the distfiles, you have a connection.

What is the output of sysctl security.jail.allow_raw_sockets from inside the jail?
 
BobSlacker said:
What is the output of sysctl security.jail.allow_raw_sockets from inside the jail?
Click to expand...
Code: 
root@practice:~ # sysctl security.jail.allow_raw_sockets
security.jail.allow_raw_sockets: 1

I have the raw sockets turned on in the /etc/jail.conf on the host...

lo1 interface (the cloned loopback) got assigned 127.0.0.2 IP address...

So my situation is:
  1. I can ping the jail's 10.0.0.2 internal IP address
  2. I can ping wlan0's externally assigned IP address (192.168.1.131)
  3. Cannot ping google.com (Unknown host)
  4. Cannot ping 8.8.8.8 or even 192.168.1.1 - 100% packet loss
1678930774977.png
 
Finally solved the puzzle after reading the Foundation's own tutorial:

I have a default kernel (13.1-RELEASE), but it does have the VIMAGE option.
Code: 
# sysctl kern.conftxt | grep VIMAGE
options VIMAGE
# sysctl kern.conftxt | wc -l
     281

Next up: That same tutorial said that having the VIMAGE kernel option is what allows the jail to have the inherit option, as documented by the jail(8) manpage.

Next up: My previous post clearly showed that I have a routing problem in my practice jail - there were no routes from 10.0.0.2 (alias IP) to 192.168.1.131 (real IP on my home's wifi LAN). I probably could have solved this with a route add after studying the route(8) and the Handbook for adding that via /etc/rc.conf (and making it permanent).

But the tutorial said that the jail can inherit and use the host's entire network stack! So, these are the edits I made to the /etc/jail.conf from the https://wiki.freebsd.org/JailingGUIApplications wiki:

interface = "wlan0"; (In the global part of /etc/jail.conf
ip4 = "inherit" in the section pertaining to the firefox jail
Comment out the ip4.addr = "10.0.0.2"; line

That got the jailed Firefox working -
1679182701973.png


Next up is limiting the jail to 4 GB RAM as per that klarasystems blog! XD
 
You must log in or register to reply here.
Back
Top