Solved No network in jail

B

bxbzq

Hi,
Following instructions from various posts/articles, I'm trying to install Red Hat as bhyve guest in jail. Now I can start the jail successully, and inside jail running install.sh script gives me error:
Code: 
Could not set interface flags
device emulation initialization error: Operation not permitted

Here is the install.sh script
Code: 
bhyve -c 1 -m 15G -w -H \
-s 0,hostbridge \
-s 3,ahci-cd,/bhyve/redhat/iso/rhel-91.iso \
-s 4,ahci-hd,/bhyve/redhat/img/redhat.img \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
redhat

Here is jail.conf
Code: 
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
bhyve {
    path = "/jails";
    host.hostname = "bhyve";
    devfs_ruleset = 20;
    interface = lo0;
    ip4.addr = 127.0.0.10;
    allow.vmm;
    persist;
}

I have these lines in /etc/rc.conf
Code: 
jail_enable="YES"
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0_name="re1bridge"
ifconfig_re1bridge="addm re1 addm tap0 up"
 
Continue...
ifconfig on the host
Code: 
re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 34:f7:16:9d:79:f6
        media: Ethernet autoselect (none)
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
re1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 2c:f0:5d:59:8d:d3
        inet 192.168.1.160 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        inet 127.0.0.10 netmask 0xffffffff
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
re1bridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:8c
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: re1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:dd
        groups: tap
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog

ifconfig inside jail
Code: 
re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 34:f7:16:9d:79:f6
        media: Ethernet autoselect (none)
        status: no carrier
re1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 2c:f0:5d:59:8d:d3
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.10 netmask 0xffffffff
        groups: lo
re1bridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:8c
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: re1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        groups: bridge
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:dd
        groups: tap
        media: Ethernet autoselect
        status: no carrier
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog
 
The similar installation script(with some directory modifications) works ok when bhyve is not in jail. It looks to me the problem is with the jail configuration, especially network related, but I just don't know what goes wrong.
Helps are appreciated.
 
What am I doing wrong?

Code: 
root@redhat:/ # ping 127.0.0.1
ping: ssend socket: Operation not permitted
root@redhat:/ # ping 127.0.0.10
ping: ssend socket: Operation not permitted
 
For the last error into the jail, I believe it's just a matter of configuration and it's probably not related to the error you get with install.sh. You have to add in the jail config: allow.raw_socket.

For your main problem, I think it's linked to the need to unhide some devices in your jail like vmm, vmm.io, nmdm...

See here: https://github.com/lattera/articles/blob/master/freebsd/2018-10-27_jailed_bhyve/article.md

That said, I don't see the point to put a bhyve VM into a jail (except the pleasure to achieve that), but that's another discussion.
 
Emrion said:
For the last error into the jail, I believe it's just a matter of configuration and it's probably not related to the error you get with install.sh. You have to add in the jail config: allow.raw_socket.
Click to expand...
Thanks. Adding allow.raw_sockets in jail.conf did the trick. I also changed the interface to "re1" and ip4.addr to the same subnet as re1's. Now I can ping and have internet access. I put LAN's router ip to jail's rc.conf as well, not sure if it has any effect though.

Emrion said:
For your main problem, I think it's linked to the need to unhide some devices in your jail like vmm, vmm.io, nmdm...
See here: https://github.com/lattera/articles/blob/master/freebsd/2018-10-27_jailed_bhyve/article.md
Click to expand...
I did refer to this article and creaeted /etc/devfs.rules as instructed. Just that the author used dev mode volume while I have default mode.
I tried to run the /usr/share/examples/bhyve/vmrun.sh and it gives me "vm_open: No such file or directory".
Keep digging.

Emrion said:
That said, I don't see the point to put a bhyve VM into a jail (except the pleasure to achieve that), but that's another discussion.
Click to expand...
My impression is running bhyve vm inside jail adds another layer of security. Does it not?
 
The vm_open error is proabably because I didnot use # bhyvectl --destroy --vm=redhat
Anyway, now that is fixed. I'm back to the original error:
Code: 
Could not set interface flags
device emulation initialization error: Operation not permitted

Could this bea privilege problem. For example, when I try to add default router in jail, route add default 192.168.1.1, it returns
Code: 
route: writing to routing socket: Operation not permitted
Even it is not related, there seems something wrong with jail's user privilege.
 
The principle of a jail is to drastically limit what is possible to do from them, so it's not a surprise that you cannot do certain things by default. You have others settings like allow.socket_af which may or may not allow what you try.

I note that step by step, you're making the fence more permeable between this jail and the main system. That's not a good presage.

The referenced article was written in 2018. It's probable that something has changed in FreeBSD and certainly it lacks one or more hole in your jail. The error message makes me think of a problem with write access to a device interface (network maybe?), but I'm not sure at all.
 
Solving the problem would require more knowledge than I have. Moving on to use bhyve vm outside of jail.
I wish there was similar tool as debootrap for Red Hat family distro...
 
bxbzq said:
Solving the problem would require more knowledge than I have. Moving on to use bhyve vm outside of jail.
I wish there was similar tool as debootrap for Red Hat family distro...
Click to expand...
This is why you have posted a thread here. This is why many people post in the forum.
Be patient, and maybe someone will shed a light on this.
 
You must log in or register to reply here.
Back
Top