Solved netgraph vnet jail not getting ip address via DHCP

A

avner

Any netgraph gurus out there?

I created two vnet jails with the help of the devin teske's jng scipts and example jail.conf under /usr/share/examples/jails/.

I can enter either jail using jexec

However, I cannot ping any targets from inside the jails. The jails do not appear to have ip addresses. The host machine gets it's ip addresses via DHCP. There is no active firewall on the host.

Here is the result of ifconfig -a on the host:
Code: 
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 78:2b:cb:02:e2:bc
        inet 10.0.0.106 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 78:2b:cb:02:e2:be
        inet 10.0.0.108 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 78:2b:cb:02:e2:c0
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 78:2b:cb:02:e2:c2
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
and on the jails:

(jail one)
Code: 
# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ng0_pgserver: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 02:a4:dc:02:e2:be
        hwaddr 58:9c:fc:00:7c:54
        inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
(jail two)
Code: 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ng0_samba: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 02:90:c0:02:e2:be
        hwaddr 58:9c:fc:10:ff:c0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
output of ngctl graph:
mygraph.jpg

The ng0_pgserver and ng0_samba interfaces are not inheriting ip addresses from bce1bridge. I am unable to ssh into these jails or ping anything from within. Setting ng0_pgserver or ng0_samba to "UP" ifconfig ng0_pgserver up changes the output of ifconfig to show UP but otherwise has no effect.

Here are my jail.conf files under jail.conf.d

pgserver.conf:
Code: 
pgserver {
        host.hostname = "daria.pgserver"; # hostname
        path = "/ja0/pgserver";           # root directory

        exec.clean;
        exec.system_user = "root";
        exec.jail_user = "root";

        #
        # NB: Below 4-lines required
        #
        vnet;
        # netgraph
        vnet.interface = "ng0_pgserver";             # vnet interface(s)
        exec.prestart += "jng bridge pgserver bce1"; # bridge interface(s)
        exec.poststop += "jng shutdown pgserver";    # destroy interface(s)
        # if_bridge
        #vnet.interface = "e0b_xxx";            # vnet interface(s)
        #exec.prestart += "jib addm xxx em0";   # bridge interface(s)
        #exec.poststop += "jib destroy xxx";    # destroy interface(s)

        # Standard recipe
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown jail";
        exec.consolelog = "/var/log/jail_pgserver_console.log";
        mount.devfs;    # mount devfs

        # Optional (default off)
        devfs_ruleset = "11";          # rule to unhide bpf for DHCP
        #allow.mount;                   # mount /etc/fstab.xxx
        #allow.set_hostname = 1;        # Allow hostname to change
        #allow.sysvipc = 1;             # Allow SysV Interprocess Comm.
}
samba.conf:
Code: 
samba {
        host.hostname = "daria.samba";  # hostname
        path = "/ja0/samba";            # root directory

        exec.clean;
        exec.system_user = "root";
        exec.jail_user = "root";

        #
        # NB: Below 4-lines required
        #
        vnet;
        # netgraph
        vnet.interface = "ng0_samba";             # vnet interface(s)
        exec.prestart += "jng bridge samba bce1"; # bridge interface(s)
        exec.poststop += "jng shutdown samba";    # destroy interface(s)
        # if_bridge
        #vnet.interface = "e0b_xxx";            # vnet interface(s)
        #exec.prestart += "jib addm xxx em0";   # bridge interface(s)
        #exec.poststop += "jib destroy xxx";    # destroy interface(s)

        # Standard recipe
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown jail";
        exec.consolelog = "/var/log/jail_samba_console.log";
        mount.devfs;    # mount devfs

        # Optional (default off)
        devfs_ruleset = "11";          # rule to unhide bpf for DHCP
        #allow.mount;                   # mount /etc/fstab.xxx
        #allow.set_hostname = 1;        # Allow hostname to change
        #allow.sysvipc = 1;             # Allow SysV Interprocess Comm.
}
Any help here would be appreciated.
 
When I create new jails using jib (not jng like you though), I set the IP address from jail using ifconfig in rc.conf script. I wonder if it would work also for netgraph?
 
algi said:
When I create new jails using jib (not jng like you though), I set the IP address from jail using ifconfig in rc.conf script. I wonder if it would work also for netgraph
algi said:
When I create new jails using jib (not jng like you though), I set the IP address from jail using ifconfig in rc.conf script. I wonder if it would work also for netgraph?
Click to expand...
From within each vnet jail I issued command
Code: 
dhclient ng0_jailname
. This seems to be the secret sauce. Now I have ip address and jails are able to ping each other as well as other targets on the lan.
Click to expand...
 
avner said:
However, I cannot ping any targets from inside the jails.
Click to expand...
Code: 
             allow.raw_sockets
                     The jail root is allowed to create raw sockets.  Setting
                     this parameter allows utilities like ping(8) and
                     traceroute(8) to operate inside the jail.  If this is
                     set, the source IP addresses are enforced to comply with
                     the IP address bound to the jail, regardless of whether
                     or not the IP_HDRINCL flag has been set on the socket.
                     Since raw sockets can be used to configure and interact
                     with various network subsystems, extra caution should be
                     used where privileged access to jails is given out to
                     untrusted parties.
 
SirDice said:
Code: 
             allow.raw_sockets
                     The jail root is allowed to create raw sockets.  Setting
                     this parameter allows utilities like ping(8) and
                     traceroute(8) to operate inside the jail.  If this is
                     set, the source IP addresses are enforced to comply with
                     the IP address bound to the jail, regardless of whether
                     or not the IP_HDRINCL flag has been set on the socket.
                     Since raw sockets can be used to configure and interact
                     with various network subsystems, extra caution should be
                     used where privileged access to jails is given out to
                     untrusted parties.
Click to expand...
Actually, I added allow.raw_sockets = 1 and then removed it from the jail.conf. In my setup ( FreeBSD 13.1 ) it does not seem necessary. I solved the problem by entering the jail and using
Code: 
dhclient my-jail-interface
in my case for example I used
Code: 
dhclient ng0_samba
. After that the jail obtained an ip address via DHCP. Then I added the dhclient command to the jail.conf so it would run automatically each time the jail started. Now able to ping from jail and ssh into jail.
 
avner said:
Any netgraph gurus out there?
Click to expand...
Basically yes, but I strictly avoid anything dhcp (except ipv6 prefix delegation, which is already bad enough). And also no jng and jip here.
Mine are connected to a zig-zag mesh either with two eiface coupled together, or an eiface attached to a bridge, and carry all static ips. This is rc script for the netgraph: https://gitr.daemon.contact/tools/tree/rc.d/ngbridge

Your graph looks basically like I do it. So the next boring thing to do would be to fire up tcpdump (and take care to run it with the -e switch because you want to see from/to which MAC address packets go) and figure out what actually happens. (That's what I did when crafting my stuff.)

I might assume that your problem is rather with the dhcp - but that may be because I basically hate it... anyway, it might be another viable approach to set it all up with static ips, verify full intended connectivity, and then implement dhcp onto it.

I have this in my jail.conf:

Code: 
allow.set_hostname = "false";
allow.mount.procfs = "false";
allow.mount.devfs = "false";
allow.raw_sockets = "false";
allow.mlock = "true";
enforce_statfs = 1;
mount.devfs;
interface = "lo0";
ip4.saddrsel = "false";

myjail {
        jid = 7;
        devfs_ruleset = 11;
        host.hostname = "myjail.example.com";
        vnet = "new";
        $ifname1l = n${name}1l;
        $ifname1l_mac = 06:19:a2:15:cd:94;
        vnet.interface = "$ifname1l";
        exec.prestart = "
            echo -e \"mkpeer eiface crhook ether\nname .:crhook $ifname1l\" \
                | /usr/sbin/ngctl -f -
            /usr/sbin/ngctl connect ${ifname1l}: lanbridge: ether link18
            ifname=`/usr/sbin/ngctl msg ${ifname1l}: getifname | \
                awk '$1 == \"Args:\" { print substr($2, 2, length($2)-2)}'`
            /sbin/ifconfig \$ifname name $ifname1l
            /sbin/ifconfig $ifname1l link $ifname1l_mac
        ";
        exec.poststop = "
                sleep 1 ;
                /usr/sbin/ngctl shutdown ${ifname1l}: ;
        ";
}
 
PMc said:
Basically yes, but I strictly avoid anything dhcp (except ipv6 prefix delegation, which is already bad enough). And also no jng and jip here.
Mine are connected to a zig-zag mesh either with two eiface coupled together, or an eiface attached to a bridge, and carry all static ips. This is rc script for the netgraph: https://gitr.daemon.contact/tools/tree/rc.d/ngbridge

Your graph looks basically like I do it. So the next boring thing to do would be to fire up tcpdump (and take care to run it with the -e switch because you want to see from/to which MAC address packets go) and figure out what actually happens. (That's what I did when crafting my stuff.)

I might assume that your problem is rather with the dhcp - but that may be because I basically hate it... anyway, it might be another viable approach to set it all up with static ips, verify full intended connectivity, and then implement dhcp onto it.

I have this in my jail.conf:

Code: 
allow.set_hostname = "false";
allow.mount.procfs = "false";
allow.mount.devfs = "false";
allow.raw_sockets = "false";
allow.mlock = "true";
enforce_statfs = 1;
mount.devfs;
interface = "lo0";
ip4.saddrsel = "false";

myjail {
        jid = 7;
        devfs_ruleset = 11;
        host.hostname = "myjail.example.com";
        vnet = "new";
        $ifname1l = n${name}1l;
        $ifname1l_mac = 06:19:a2:15:cd:94;
        vnet.interface = "$ifname1l";
        exec.prestart = "
            echo -e \"mkpeer eiface crhook ether\nname .:crhook $ifname1l\" \
                | /usr/sbin/ngctl -f -
            /usr/sbin/ngctl connect ${ifname1l}: lanbridge: ether link18
            ifname=`/usr/sbin/ngctl msg ${ifname1l}: getifname | \
                awk '$1 == \"Args:\" { print substr($2, 2, length($2)-2)}'`
            /sbin/ifconfig \$ifname name $ifname1l
            /sbin/ifconfig $ifname1l link $ifname1l_mac
        ";
        exec.poststop = "
                sleep 1 ;
                /usr/sbin/ngctl shutdown ${ifname1l}: ;
        ";
}
Click to expand...
Thank you. Actually, my dhcp problem was solved by running dhclient inside jail; seems like a ridiculous oversight now in retrospect. I just figured out how to mark this thread as solved. Much appreciate your sharing your rc.conf and jail.conf with me. Will enjoy studying some of the options which are unfamiliar to me to further my learning.
 
  • Like
Reactions: PMc
You must log in or register to reply here.
Back
Top