jails Wifi socks Hurd!

BaronBS

BaronBS

So, I know that network doesn't work with bhyve and wifi, I'm using a lagg (wlan0 + em0) setup. But does anyone here was able to make jails+vnet work with wifi(in a lagg setup)?
 
Uhm, I don't see how one or the other should have anything to do with Wifi or lagg?

In a nutshell, put whatever your "real" host interface is (e.g. the lagg interface) in a bridge, together with all the virtual ends (tap for VMs, epair for vnet jails), and this should work.

So, please describe your scenario and how it doesn't work for you in detail, thanks.
 
zirias@ said:
Uhm, I don't see how one or the other should have anything to do with Wifi or lagg?

In a nutshell, put whatever your "real" host interface is (e.g. the lagg interface) in a bridge, together with all the virtual ends (tap for VMs, epair for vnet jails), and this should work.

So, please describe your scenario and how it doesn't work for you in detail, thanks.
Click to expand...

I used sysutils/bastille and sysutils/cbsd, both of them do the bridge setup automatically.

Here are some info:

Host /etc/rc.conf:
Code: 
{beastie}  FreeBaSeD-T430 > /home/beastie
 → cat /etc/rc.conf                                     
# Auto

clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="FreeBaSeD-T430"
sshd_enable="YES"
moused_enable="YES"
ntpd_enable="YES"
ntpdate_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
zfskeys_enable="YES"
kld_list="i915kms linux linux64 fusefs coretemp sysctlinfo sysctlbyname_improved mac_priority if_bridge nmdm"

################################################################

# Boot

rc_info="NO"
rc_startmsgs="NO"

################################################################
# Firewall

## IPFW
firewall_enable="YES"
firewall_quiet="YES"
firewall_type="workstation"
firewall_logdeny="YES"
firewall_myservices="80/tcp 443/tcp 22/tcp"
firewall_allowservices="any"

## PF
pf_enable="yes"
pflog_enable="YES"

#################################################################

# Network

## Trunk
ifconfig_em0="up"
wlans_iwn0="wlan0"
ifconfig_wlan0="WPA"
create_args_wlan0="wlanaddr 00:21:cc:d9:fd:75 up"
cloned_interfaces="lagg0"
ifconfig_lagg0="up laggproto failover laggport em0 laggport wlan0 DHCP"
rtsold_enable="YES"

## Nat Network
gateway_enable="YES"

## VPN
#openvpn_enable="YES"

## DNSMASQ
#dnsmasq_enable="YES"

#################################################################

# DBUS

dbus_enable="YES"

#################################################################

# Webcamd

webcamd_enable="YES"
webcamd_0_flags="-d ugen1.5"

#################################################################

# Enabling drive monitoring

smartd_enable="YES"

#################################################################

# Enabling Microcode updates

microcode_update_enable="YES"

#################################################################

# mixertui, sound and mic stuff

mixer_enable="YES"
sysctlinfo_load="YES"
sysctlbyname_improved_load="YES"
jackd_enable="YES"
jackd_user="beastie"
jackd_rtprio="YES"
jackd_args="-r -d oss -r44100 -p1024 -n2 -w16 -i4 -o8 -C /dev/dsp0 -P /dev/dsp0"
alsa_seq_server_enable="YES"

#################################################################

# Jails & Virtualization

# VM-Bhyve

vm_enable="YES"
vm_dir="zfs:tank0/vm"

# CBSD
cbsd_workdir="/tank0/cbsd"

#################################################################

# Loading /etc/devfs.rules

devfs_system_ruleset="system"

#################################################################

# Linux Compat
linux_enable="YES"

#################################################################

# GELI
geli_devices="ada1 ada2"
#geli_ada1_flags="-k /root/ada1.key"

Host /etc/sysctl.conf
Code: 
{beastie}  FreeBaSeD-T430 > /home/beastie
 → cat /etc/sysctl.conf
# $FreeBSD$
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Security
# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
## Kernel Hardening
kern.randompid=1
kern.elf32.aslr.enable=1
kern.elf32.aslr.pie_enable=1
kern.elf32.aslr.honor_sbrk=0
kern.elf64.aslr.enable=1
kern.elf64.aslr.pie_enable=1
kern.elf64.aslr.honor_sbrk=0
vfs.zfs.min_auto_ashift=12
###############################################
# Boost performance
kern.sched.preempt_thresh=224
kern.ipc.shm_allow_removed=1
kern.ipc.shmmax=67108864
kern.ipc.shmall=32768
kern.threads.max_threads_per_proc=4096
#kern.sched.slice=5
kern.coredump=0
###############################################
# FUSEfs Samba
vfs.usermount=1
###############################################
# Network
net.local.stream.recvspace=65536
net.local.stream.sendspace=65536
net.inet.ip.forwarding=1
net.inet.ip.random_id=1
net.link.tap.up_on_open=1
###############################################
# Suspend on lid close
hw.acpi.lid_switch_state=S3
###############################################
# Sound
hw.snd.default_unit=1
###############################################
# Jail ping
security.jail.allow_raw_sockets=1

Host /etc/devfs.rules
Code: 
{beastie}  FreeBaSeD-T430 > /home/beastie
 → cat /etc/devfs.rules
[system=10]
add path 'ad[0-9]\*'        mode 666 group operator
add path 'ada[0-9]\*'        mode 666 group operator
add path 'da[0-9]\*'        mode 666 group operator
add path 'acd[0-9]\*'        mode 666 group operator
add path 'cd[0-9]\*'        mode 666 group operator
add path 'mmcsd[0-9]\*'        mode 666 group operator
add path 'pass[0-9]\*'        mode 666 group operator
add path 'xpt[0-9]\*'        mode 666 group operator
add path 'ugen[0-9]\*'        mode 666 group operator
add path 'usbctl[0-9]\*'    mode 666 group operator
add path 'usb/\*'        mode 666 group operator
add path 'fd[0-9]\*'        mode 666 group operator
add path 'uscan[0-9]\*'        mode 666 group operator
add path 'video[0-9]\*'        mode 666 group operator
add path 'lpt[0-9]\*'        mode 666 group cups
add path 'ulpt[0-9]\*'        mode 666 group cups
add path 'unlpt[0-9]\*'        mode 666 group cups

# Devices usually found in a jail.
[devfsrules_jail=4]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path zfs unhide
add path 'bpf*' unhide

Host /etc/pf.conf
Code: 
{beastie}  FreeBaSeD-T430 > /home/beastie
 → cat /etc/pf.conf    
ext_if="lagg0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

Host ifconfig output:
Code: 
{beastie}  FreeBaSeD-T430 > /home/beastie
 → ifconfig                     
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=800000<>
    ether 00:21:cc:d9:fd:75
    media: Ethernet autoselect
    status: no carrier
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 00:21:cc:d9:fd:75
    groups: wlan
    ssid Montanas channel 48 (5240 MHz 11a ht/40-) bssid 50:a5:dc:de:8c:23
    regdomain FCC country US authmode WPA2/802.11i privacy ON
    deftxkey UNDEF AES-CCM 2:128-bit txpower 17 bmiss 10 mcastrate 6
    mgmtrate 6 scanvalid 60 ampdulimit 64k ampdudensity 4 -amsdutx amsdurx
    shortgi -stbc -ldpc -uapsd wme roaming MANUAL
    parent interface: iwn0
    media: IEEE 802.11 Wireless Ethernet MCS mode 11na
    status: associated
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 00:21:cc:d9:fd:75
    inet 192.168.0.176 netmask 0xffffff00 broadcast 192.168.0.255
    laggproto failover lagghash l2,l3,l4
    laggport: em0 flags=1<MASTER>
    laggport: wlan0 flags=4<ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: lagg0
    ether 58:9c:fc:10:ff:af
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
epair1a: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: i2pd-eth0
    options=8<VLAN_MTU>
    ether 02:f7:fe:35:86:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Jail initialization:
Code: 
{beastie}  FreeBaSeD-T430 > /home/beastie
 → doas cbsd jstart i2pd   
create epair: epair1:lagg0 
Default NIC automatically selected: lagg0
set resource limit: [ ]
jail renice: 1
Starting jail: i2pd, parallel timeout=5
i2pd: created
eth0
late_start in progress...
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg
32-bit compatibility ldconfig path: /usr/lib32
Starting Network: lo0 eth0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
eth0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 00:a0:98:d0:2d:41
    hwaddr 02:f7:fe:35:86:0b
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Creating and/or trimming log files.
Updating motd:.
Updating /var/run/os-release done.
Clearing /tmp (X related).
Starting syslogd.
Starting cron.
Generating RSA host key.
3072 SHA256:9yC7dX8fEoAxTiou20tpYksWhat/9CAIX+Pu/Pp0zdQ root@i2pd.example.com (RSA)
Generating ECDSA host key.
256 SHA256:oTMNKgHFm6nD3UDzUtPJU/XE1XkPzm4xUqDNfstnoWo root@i2pd.example.com (ECDSA)
Generating ED25519 host key.
256 SHA256:8ArbhmV1D8pU4A3slhsdCYcNZVo5lz+XK/KQdkWi6tU root@i2pd.example.com (ED25519)
Performing sanity check on sshd configuration.
Starting sshd.

Fri May 12 21:25:52 CEST 2023
CBSD setup: jail ipfw counters num: 99/101
jstart done in 5 seconds

Jail internet test;
Code: 
{beastie}  FreeBaSeD-T430 > /home/beastie
 → doas cbsd jlogin i2pd
FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC
i2pd:/root@[21:29] # pkg update
Updating FreeBSD repository catalogue...
pkg: http://pkg.freebsd.org/FreeBSD:13:amd64/quarterly/meta.txz: No address record
repository FreeBSD has no meta file, using default settings
pkg: http://pkg.freebsd.org/FreeBSD:13:amd64/quarterly/packagesite.pkg: No address record
pkg: http://pkg.freebsd.org/FreeBSD:13:amd64/quarterly/packagesite.txz: No address record
Unable to update repository FreeBSD
Error updating repositories!
i2pd:/root@[21:30] # ping -c 3 freebsd.org
ping: Unknown host
i2pd:/root@[21:30] # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
eth0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 00:a0:98:d0:2d:41
    hwaddr 02:f7:fe:35:86:0b
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 
First thing standing out, you have
Code: 
firewall_enable="YES"
pf_enable="yes"
Enabling ipfw and pf at the same time certainly is begging for (networking) trouble.

Then, I'm pretty sceptical doing nat would EVER work when you're not routing. Even if so, you seem to use some dynamic table for it, is it ever filled? Can't see that above.

Why don't you try the straight-forward thing, just add the jail to the bridge and configure the interface inside the jail for DHCP as well, so it would get some valid address from the same subnet as your host (and have the correct routes automatically)? (that works, but probably only when using the LAN interface)

Edit: just thought about how Wifi authenticates stations (based on MAC addresses) and googled a bit, confirming the issue: Well, the bridge with your lagg device will certainly work. But even when you configure it correctly, as soon as the active interface in the lagg is the wifi interface, the access point receiving the package will reject it because it has the wrong source MAC.

The simple solution if you want to share a wifi connection is routing. That's doable with VMs and VNET jails as well of course.

What could also work is using the wds mode for your Wifi, but both AP and station must have it enabled, and I never tried that. It offers one more MAC field, so it can differentiate between the sender of the package (could be your jail) and the sending wifi station (your host's wifi interface).
 
BTW, I would probably opt for the routing as this will certainly work. I can't tell you how to configure it using your tools, I don't know them. But in a nutshell, just don't add your lagg interface to the bridge, instead assign it an address from the same subnet all your jails/vms use, and add gateway_enable="YES" to your /etc/rc.conf. Then you will probably also need nat, but no need for a table there, just nat everything originating from that subnet when going out over the lagg interface.
 
zirias@ said:
Enabling ipfw and pf at the same time certainly is begging for (networking) trouble.
Click to expand...
Owk dowk. Witch one do you recommend to use with VNET Jails?

zirias@ said:
is it ever filled? Can't see that above.
Click to expand...
How can I show you?

zirias@ said:
The simple solution if you want to share a wifi connection is routing. That's doable with VMs and VNET jails as well of course.
Click to expand...
Could you kindly point to the documentation for me to do that in FreeBSD please?

zirias@ said:
What could also work is using the wds mode for your Wifi, but both AP and station must have it enabled, and I never tried that. It offers one more MAC field, so it can differentiate between the sender of the package (could be your jail) and the sending wifi station (your host's wifi interface).
Click to expand...
I will try the routing solution first.

zirias@ said:
I can't tell you how to configure it using your tools, I don't know them.
Click to expand...
FreeBSD and any tool that it is in the ports tree.

zirias@ said:
Then you will probably also need nat, but no need for a table there, just nat everything originating from that subnet when going out over the lagg interface.
Click to expand...
Could you kindly point to the documentation for me to do that in FreeBSD please?²
 
You must log in or register to reply here.
Back
Top