Solved ifconfig in a jail shows all interfaces incl. on the host

jazzlover

jazzlover

Hello there,
As I read in Absolute FreeBSD, if one logged in a jail (aka jail /jails/web/ web 10.0.0.10 /bin/tcsh) and run ifconfig there, it should only show the lo0 and an interface assigned to the jail. In my freshly installed and upgraded FreeBSD with a jail ifconfig shows complete list of the interfaces on the host.
I wonder if this is as it should and I misunderstand something or I somehow butched the installation?
 
Some time ago I have stumbled over the same question. In a traditional jail the network stack of the host is used for the jails, too. From my understanding this is why all interfaces are shown.

There is an option to use VNET jails instead. Each VNET jail runs its own network stack. This causes overhead but does not expose all interfaces of the host to that VNET jail.
 
chrbr said:
There is an option to use VNET jails instead. Each VNET jail runs its own network stack
Click to expand...
That's how I stumble upon this :) I'm moving my server with bunch of jails and, since it is in VM now, decided to change networking to VNET.

This is my jail.conf:
Code: 
$j="j";
mount.devfs;
devfs_ruleset="5";
exec.clean;
exec.start="/bin/sh /etc/rc";
exec.stop="/bin/sh /etc/rc.shutdown";
mount.fstab="/etc/fstab.${name}";
exec.prestart="logger trying to start jail $name ...";
exec.created="logger jail $name has started";
exec.prestop="logger shutting down jail $name";
exec.poststop="logger jail $name has shut down";
exec.consolelog="/var/log/j/console_${name}.log";
exec.timeout=30;
allow.raw_sockets=1;    #allow ping-pong

exec.prestart+="/usr/local/sbin/jib addm $name $if";
exec.poststop+="/usr/local/sbin/jib destroy $name";

host.hostname="${name}.${dom}";
path="/$j/${name}";
vnet;
$sl="2";
$if="dmzapps";
$dom="apps";

web {
vnet.interface="e0b_${name}";
securelevel="$sl";
}

If I log into a jail and run ifconfig it shows all interfaces. You just confirmed that I understand correctly how it should work. The next question is why... I did nothing fancy with the host (or a jail).
 
I have never tried the jib thing. Below are links I have got through:

VNET Jail HowTo | Antranig Vartanian

weblog.antranigv.am weblog.antranigv.am

VNET Jail HowTo Part 2: Networking | Antranig Vartanian

weblog.antranigv.am weblog.antranigv.am
forums.freebsd.org

Issues assigning epair interface to vnet jail

Hi, I'm having issues with VNET on my FreeBSD 13.0-RELEASE-p11 host running on a cloud VPS. After some other problems previously discussed in this thread, I'm now stuck with the following issue: I have a very simple VNET jail setup (config see below) and tried to manually assign it an epair...
forums.freebsd.org forums.freebsd.org
it-notes.dragas.net

Boosting Network Performance in FreeBSD's VNET Jails

it-notes.dragas.net it-notes.dragas.net

I have used a bridge on the host and connected the jails by epairs.
 
Now find below my /etc/jail.conf.
Code: 
path = /jails/$name;
host.hostname = $name;
mount.devfs;
allow.raw_sockets = 1;
#ip6.addr = "";
devfs_ruleset = 20;
vnet;persist;
exec.clean;
mount.fstab = "/etc/fstab.$name";
vnet.interface = "epair${jid}b";
exec.prestart =  "ifconfig epair${jid} create up";
exec.prestart += "ifconfig epair${jid}a up";
exec.prestart += "ifconfig bridge0 addm epair${jid}a up";
exec.start  = "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start += "/sbin/ifconfig epair${jid}b 10.0.0.${jid}/24 up";
exec.start += "/sbin/route add default 10.0.0.254";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.prestop   = "ifconfig epair${jid}b -vnet ${name}";
exec.poststop  = "ifconfig bridge0 deletem epair${jid}a";
exec.poststop += "ifconfig epair${jid}a destroy";
fox {
    jid = 2;
}
zdf {
    jid = 3;
}
www {
    jid = 4;
}
This is the relevant part of /etc/rc.conf
Code: 
cloned_interfaces="bridge0"
ifconfig_bridge0="addm wlan0 up"
ifconfig_bridge0="inet 10.0.0.254/24"
gateway_enable="yes"
I am not sure if it is 100% correct, at least it is working. Please verify the stuff before you use it.
 
chrbr said:
Code: 
ifconfig_bridge0="addm wlan0 up" 
ifconfig_bridge0="inet 10.0.0.254/24"
Click to expand...
This isn't going to work. The ifconfig_bridge0 is a shell variable (in essence rc.conf is a shell script that only contains variables) and you're redefining it twice. So only the last one will be set.

Code: 
#!/bin/sh 

foo="bar"
foo="FOOBAR"

echo "${foo}"
 
chrbr said:
I have never tried the jib thing. Below are links I have got through:

VNET Jail HowTo | Antranig Vartanian

weblog.antranigv.am weblog.antranigv.am

VNET Jail HowTo Part 2: Networking | Antranig Vartanian

weblog.antranigv.am weblog.antranigv.am
forums.freebsd.org

Issues assigning epair interface to vnet jail

Hi, I'm having issues with VNET on my FreeBSD 13.0-RELEASE-p11 host running on a cloud VPS. After some other problems previously discussed in this thread, I'm now stuck with the following issue: I have a very simple VNET jail setup (config see below) and tried to manually assign it an epair...
forums.freebsd.org forums.freebsd.org
it-notes.dragas.net

Boosting Network Performance in FreeBSD's VNET Jails

it-notes.dragas.net it-notes.dragas.net

I have used a bridge on the host and connected the jails by epairs.
Click to expand...
Thank you for the tutorial links, I appreciate that.
In my debugging I removed all bridges and epair along with jib and just assigned an interface to the test jail. The most peculiar thing is that with this in jail.conf:
Code: 
vnet;
vnet.interface="vtnet1";
and with this in
jail/etc/rc.conf:
Code: 
ifconfig_vtnet1="10.1.1.111 netmask 255.255.255.0"
defaultrouter="10.1.1.1"
ifconfig in the jail show three interfaces: lo0, vtnet1 and, surprise-surprise - vtnet0, which is host's interface. Have no idea how it got there and, more importantly, how to stop it from being in the jail
 
chrbr said:
This is the relevant part of /etc/rc.conf
Code: 
cloned_interfaces="bridge0"
ifconfig_bridge0="addm wlan0 up"
ifconfig_bridge0="inet 10.0.0.254/24"
gateway_enable="yes"
Click to expand...
SirDice said:
This isn't going to work. The ifconfig_bridge0 is a shell variable (in essence rc.conf is a shell script that only contains variables) and you're redefining it twice. So only the last one will be set.
Click to expand...
Thank you SirDice, that has been a leftover from my first tests connecting the external interface instead of the cloned interface lo1. I cannot understand how I have overlooked that.
 
Hi jazzlover,
here it looks good. Only the local interface and the epair which connects the bridge on the host are visible.
Code: 
# jexec fox ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair2b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:99:6d:f6:16:0b
        inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
May be the output of ifconfig on the host and in the jail would give an indication, at least for the gurus in this forum. I am not sure if I am of help since I have no experience with virtual machines, too.
 
chrbr said:
May be the output of ifconfig on the host and in the jail would give an indication, at least for the gurus in this forum. I am not sure if I am of help since I have no experience with virtual machines, too.
Click to expand...
Here are the outputs in the host
sh: 
 17:15 /root # ifconfig
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether bc:24:11:6a:bf:f4
        inet 172.22.11.25 netmask 0xffffff00 broadcast 172.22.11.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
dmzapps: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether bc:24:11:77:5f:96
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
dmzappsbridge: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:ff:b8
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_test flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: dmzapps flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_test: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:40:d5:77:5f:96
        hwaddr 02:5e:d0:93:f5:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
and in the jail (logged into it jail /jails/web/ web 10.1.1.111 /bin/tcsh):
sh: 
 17:16 / # ifconfig
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether bc:24:11:6a:bf:f4
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
dmzapps: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether bc:24:11:77:5f:96
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
dmzappsbridge: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:ff:b8
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_test flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: dmzapps flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000
        groups: bridge
e0a_test: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:40:d5:77:5f:96
        hwaddr 02:5e:d0:93:f5:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

What's interesting though, jexec provide the expected output:
sh: 
 17:17 /root # jexec test ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_test: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 0e:40:d5:77:5f:96
        hwaddr 02:5e:d0:93:f5:0b
        inet 10.1.1.111 netmask 0xffffff00 broadcast 10.1.1.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Just in case, the /j/test/etc/rc.conf:
Code: 
clear_tmp_enable="YES"
# Disable hal-fopen port for syslog
syslogd_flags="-ss"
sshd_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
ifconfig_e0b_test="10.1.1.111 netmask 255.255.255.0"
defaultrouter="10.1.1.1"

I really hope for some clues, I ran out of ideas and the server I thought I successfully migrated doesn't work due to networking problem.

EDIT: added
/etc/rc.conf on the host:
Code: 
clear_tmp_enable="YES"
syslogd_flags="-ss"
hostname="fbsd-test"
ifconfig_vtnet0="inet 172.22.11.25 netmask 255.255.255.0 -lro -tso"
defaultrouter="172.22.11.1"
gateway_enable="yes"
ifconfig_bridge0="addm  up"
ifconfig_vtnet1_name="dmzapps"
kld_list="if_bridge if_epair"
jail_enable=YES
jail_list="test"
jail_reverse_stop="YES"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
moused_nondefault_enable="NO"
dumpdev="NO"
zfs_enable="YES"
zfskeys_enable="YES"
qemu_guest_agent_enable="YES"
 
In contrast to my output there are no IP addresses in the jails ifconfig output. On the host it is just vnet0 and lo0 with an IP address. I must admit that I have not fully understood how the IP assignment on a FreeBSD bridge works because a simple bridge does not know about IP addresses. I hope someone with more knowledge than me can help.
 
jazzlover said:
Hello there,
As I read in Absolute FreeBSD, if one logged in a jail (aka jail /jails/web/ web 10.0.0.10 /bin/tcsh) and run ifconfig there, it should only show the lo0 and an interface assigned to the jail. In my freshly installed and upgraded FreeBSD with a jail ifconfig shows complete list of the interfaces on the host.
I wonder if this is as it should and I misunderstand something or I somehow butched the installation?
Click to expand...
The command you used here: jail /jails/web/ web 10.0.0.10 /bin/tcsh is for running a command when the jail isn't active (found the explanation in the book "FreeBSD Mastery Jails" chapter_3), so this is not the appropriate way to run this specific command.
Instead as you already found out the jexec command gives you the expected output.

jazzlover said:
In my debugging I removed all bridges and epair along with jib and just assigned an interface to the test jail. The most peculiar thing is that with this in jail.conf:
Code: 
vnet;
vnet.interface="vtnet1";
and with this in
jail/etc/rc.conf:
Code: 
ifconfig_vtnet1="10.1.1.111 netmask 255.255.255.0"
defaultrouter="10.1.1.1"
ifconfig in the jail show three interfaces: lo0, vtnet1 and, surprise-surprise - vtnet0, which is host's interface. Have no idea how it got there and, more importantly, how to stop it from being in the jail
Click to expand...
According to your host's ifconfig output from message #10 you don't have a vtnet1 interface I mean from your real hardware, so it won't work.
VNET works either with real interface or if_bridge/if_epair or netgrah.
 
gotnull said:
The command you used here: jail /jails/web/ web 10.0.0.10 /bin/tcsh is for running a command when the jail isn't active
Click to expand...
Thank you for the clarification, I somehow misinterpreted the statement in the book and though that this is also a way to login into a jail. Actually, this explains a lot.
So, looks like VNET works as expected, still the traffic is routed through the host and I do not understand why, and OPNsense blocking it as coming from unexpected interface. Perhaps, I should add a static route, but I have read in the same book that "VNET has largely obsoleted it in a jail context". Could you please suggest which way to look for that?
gotnull said:
you don't have a vtnet1 interface
Click to expand...
Sorry about that, I have this in the host's rc.conf:
Code: 
ifconfig_vtnet1_name="dmzapps"
It's there, but under another name.
 
jazzlover said:
I somehow misinterpreted the statement in the book and though that this is also a way to login into a jail
Click to expand...
To login into a jail you can use this command instead:
jexec -u root your_jail_name

The jail chapter in the Absolute FreeBSD is more of an introduction, personally I like these two books "FreeBSD Mastery Jails" by Michael W Lucas and "Jails and VNET a guide" by Derik J. Ramirez, the last one is a more update approach for VNET.
Few good blog posts:
vermaden.wordpress.com

FreeBSD Jails Containers

FreeBSD networking and containers (Jails) stacks are very mature and provide lots of useful features … yet for some reason these features are not properly advertised by the FreeBSD project &#…
vermaden.wordpress.com vermaden.wordpress.com

Learning Notes on FreeBSD Jails - genneko

I have heard about jails many times since my early days of FreeBSD life but it was only the last year I began to use it in production. This article is a sort of personal notebook where I summarize what I learned about jails. It would be frequently updated as I learn more. Assumptions The host...
genneko.github.io

And of course the Handbook:
docs.freebsd.org

Chapter 17. Jails and Containers

Jails improve on the concept of the traditional chroot environment in several ways
docs.freebsd.org docs.freebsd.org

jazzlover said:
Sorry about that, I have this in the host's rc.conf:
Code: 
ifconfig_vtnet1_name="dmzapps"
It's there, but under another name.
Click to expand...
My bad I didn't see that.

jazzlover said:
So, looks like VNET works as expected, still the traffic is routed through the host and I do not understand why, and OPNsense blocking it as coming from unexpected interface. Perhaps, I should add a static route, but I have read in the same book that "VNET has largely obsoleted it in a jail context". Could you please suggest which way to look for that?
Click to expand...
Humm I think I've reached my limits, but just in case what do you mean exactly by "still the traffic is routed through the host" and how do you check it exactly?

PS: Do not expect too much help regarding OPNsense, none of the FreeBSD derivatives are allowed around here.
forums.freebsd.org

GhostBSD, pfSense, TrueNAS, and all other FreeBSD Derivatives

Questions about 'derivative FreeBSDs', like GhostBSD DesktopBSD TrueNAS XigmaNAS OPNsense pfSense PacBSD BSD Router Project NomadBSD helloSystem should be asked on the forums and/or mailing lists for these specific products. See below for links. If you still think your questions should be...
forums.freebsd.org forums.freebsd.org
 
gotnull said:
I like these two books "FreeBSD Mastery Jails" by Michael W Lucas and "Jails and VNET a guide" by Derik J. Ramirez, the last one is a more update approach for VNET.
Click to expand...
Thank you for hte suggestions. The former I have already read and found quite illuminating. The latter now is in my read list :)

I figured out the problem: I mistakenly have treated an interface in the FreeBSD as a bridge (I passed a bridge from proxmox to the FreeBSD) and tried to connect to it with epair my jails. No wonder it didn't work. Once I attached the interface directly to the jail it miraculously started working like a charm. So, thank you everyone for the suggestions, your help pushed me in right direction at last.
 
You must log in or register to reply here.
Back
Top