FreeBSD 15 Bridges, VLANs and Jails - Nice!

pprocacci said:
From what I gather from the information you provided em0 is a trunk.... so

1) Add em0 to bridge0 (w/out it having an address)
2) Create a vlan interface off of bridge0 (bridge0.10 for example)
3) DHCP on bridge0.10 (or whatever)

When you are done, it should look something like this (except I'm using ixl0 here):
ixl0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS>
ether 3c:fd:fe:aa:df:a8
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=10<VLAN_HWTAGGING>
member: ixl0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 1 priority 128 path cost 55 vlan protocol 802.1q untagged 1 tagged 10,20,30,40,50,60,70
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>

bridge0.10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1496
options=0
ether 58:9c:fc:10:6f:fa
inet 192.168.10.2 netmask 0xffffff00 broadcast 192.168.10.255
groups: vlan
vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: bridge0
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


If you want an rc.conf configuration here's mine. Substitute ixl0 and vlans as necessary:
ifconfig_ixl0="-txcsum -txcsum6 up"
ifconfig_bridge0="vlanfilter addm ixl0 tagged 10,20,30,40,50,60,70 untagged 1 up"
vlans_bridge0="10"
ifconfig_bridge0_10="SYNCDHCP up"

The above will send traffic on vlan 10 as you desire.
Click to expand...

pprocacci Thank you!

Unfortunately, bridge0.10 is not acquiring an ip address.

Here's my system. Note that I'm only using VLANS in the virtualbox guest os to manage jails. There is no upstream switch with VLAN capabilities in my setup.

0. LAPTOP : Freebsd 15.0-RELEASE : (my laptop DHCP's an address from my home router)
LAPTOP network
Bash: 
re0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether d4:93:90:3c:84:69
        inet 192.168.0.230 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::d693:90ff:fe3c:8469%re0 prefixlen 64 scopeid 0x1
        inet6 2601:602:8100:d9a0:d693:90ff:fe3c:8469 prefixlen 64 autoconf pltime 304912 vltime 304912
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
arp -na
Bash: 
? (192.168.0.1) at a4:01:de:2f:38:95 on re0 expires in 1200 seconds [ethernet]
? (192.168.0.2) at 08:00:27:7f:a5:81 on re0 expires in 619 seconds [ethernet]
? (192.168.0.230) at d4:93:90:3c:84:69 on re0 permanent [ethernet]
netstat -arn
Bash: 
Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.0.1        UGS             re0
127.0.0.1          link#2             UH              lo0
192.168.0.0/24     link#1             U               re0
192.168.0.230      link#2             UHS             lo0

So the laptop looks good. The default route to my gateway is correct. Networking works fine. I can ping and ssh into the guest VM JAILHOST (server_a) at 192.168.0.2.

1. VirtualBox : Virtual Box is running on my laptop
2. JAILHOST (server_a): Freebsd 15.0-RELEASE : running inside virtualbox
JAILHOST network (with ifconfig_em0="DHCP")
Bash: 
hostname="server_a"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
pf_enable="YES"
pf_fallback_rules_enable="YES"
pf_fallback_rules="block drop log all
pass quick on em4"        # multi-rule
pflog_enable="YES"
node_exporter_enable="YES"

syslogd_flags="-ss"
defaultrouter="192.168.0.1"

# jails
jail_enable="YES"
# TODO: is parallel start a good idea?
jail_parallel_start="YES"
gateway_enable="YES"
ipv6_gateway_enable="NO"

# <<<<<<<<<<<<<< INTERFACE SET TO USE DHCP >>>>>>>>>>>>>>>>>>>>>
ifconfig_em0="DHCP"

# <<<<<<<<<<<<<< VLAN STUFF DISABLED >>>>>>>>>>>>>>>>>>>>>
# Create bridge
#cloned_interfaces="bridge0"
#ifconfig_em0="-txcsum -txcsum6 up"
#ifconfig_bridge0="vlanfilter addm em0 tagged 10,20,30,40,50,60,70 untagged 1 up"
#vlans_bridge0="10"
#ifconfig_bridge0_10="DHCP"
#dhcp_timeout="60"
Bash: 
em0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=48505b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
    ether 08:00:27:7f:a5:81
    inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
arp -na
Bash: 
? (192.168.0.1) at a4:01:de:2f:38:95 on em0 expires in 1181 seconds [ethernet]
? (192.168.0.2) at 08:00:27:7f:a5:81 on em0 permanent [ethernet]
? (192.168.0.230) at d4:93:90:3c:84:69 on em0 expires in 506 seconds [ethernet]
netstat -arn
Bash: 
Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.0.1        UGS             em0
127.0.0.1          link#2             UH              lo0
192.168.0.0/24     link#1             U               em0
192.168.0.2        link#2             UHS             lo0

Looks good so far. The guest VM has arp entries for the default gateway and has full internet connectivity. Default gateway looks good.

But when I change JAILHOST to use VLANS, the ip address is not dhcp'd into bridge0.10 and there is no network and no connectivity.

3. JAILHOST with suggested changes
/etc/rc.conf
Bash: 
# ... <snip>
# 
# <<<<<<<<<<<<<< INTERFACE COMMENTED OUT >>>>>>>>>>>>>>>>>>>>>
# ifconfig_em0="DHCP"

# <<<<<<<<<<<<<< VLAN STUFF ENABLED >>>>>>>>>>>>>>>>>>>>>
# Create bridge
cloned_interfaces="bridge0"
ifconfig_em0="-txcsum -txcsum6 up"
ifconfig_bridge0="vlanfilter addm em0 tagged 10,20,30,40,50,60,70 untagged 1 up"
vlans_bridge0="10"
ifconfig_bridge0_10="DHCP"
dhcp_timeout="60"

Bash: 
sudo service netif restart

Networking is now broken:
* arp has no entries
* netstat just shows the localhost
*ifconfig shows that the bridge0.10 is missing an ipaddress (below)

Bash: 
em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=48501b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
    ether 08:00:27:7f:a5:81
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33152
    options=0
    groups: pflog
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=10<VLAN_HWTAGGING>
    ether 08:00:27:7f:a5:81
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=1<VLANFILTER>
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 1 priority 128 path cost 20000 vlan protocol 802.1q untagged 1 tagged 10,20,30,40,50,60,70
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0.10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1496
    options=0
    ether 08:00:27:7f:a5:81
    groups: vlan
    vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: bridge0
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

I must be missing something simple here. Any thoughts?
 
There is no upstream switch with VLAN capabilities in my setup.
Click to expand...

Right, that changes things. ;)

So talking strictly about the port connected to your switch, since it's not vlan aware, you just stick that port in bridge0 without any crazy options, and DHCP off of bridge0.
That'll give bridge0 an address that was gotten from your switch. You should be able to hit that ip without issue. That's the first part. If that doesn't work, don't continue until it does. ;)

Assuming that works, then you `tussle' with the remaining parts.

The first part to keep in mind is that your switch/router probably won't answer to broadcasts for which a vlan is set. The switch/router is "dumb".

So what you'll need to do is create a bridge0.10 (for example) and run dhcp on that interface for anything within that broadcast domain.
That'll allow anything sitting within vlan 10 to acquire addresses within that specific domain.
Add interfaces from your VMs on bridge0.10.

Once VMs are able to ping each other, it's time to get them out to the internet.
You need a way to forward packets to the "dumb" switch/router.
That can be done with PF and nat, forwarding packets that are not destined for the subnet you have associated with the VLAN.

That should be it.

If you're expecting VMs that are in a specific VLAN to be able to reach the switch and the switch respond to packets when a tag is set, my guess is it's just going to drop it entirely -- though this would be manufactuer specific.
This also means you need to setup the FreeBSD host (where the DHCP is) to be setup as a gateway. (gateway_enable)

---

There's an alternative to the above and in your case it might be the better choice.

Step one remains the same. Create bridge0, stick em0, and DHCP off of bridge0.

Step two however is creating a separate bridge for each domain you want to host behind it.
So bridge10, bridge20, etc. (note, no periods here)
You'd assign the first address of each broadcast domain on these interfaces, 192.168.10.1 on bridge10 and 192.168.20.1 on bridge20, etc....
Setup dhcp to respond to requests on bridge10, bridge20, etc....
Then setup the FreeBSD host to forward accordingly to the upstream as described above.

~Paul
 
pprocacci Thanks Paul! I'll work through this and get back to you. BTW - can you recommend any books on this stuff? I have books about FreeBSD, Jails, Networking, etc. But nothing seems to cover VLANs and the sort of things we are talking about here.
 
I don't have any book recommendations unfortunately. I've been doing this on FreeBSD for 30 years roughly now and 'just know'. Sorry about that.
But what I can do is try to explain it a tad.

A vlan is for all intents and purposes a separate network. -- lots of weeds here I'm ignoring.
The introduction of vlans into what you've been describing means an introduction to a new network.

Because your switch is "dumb", your em0 interface is one network and only ever one network.
The adding of one or more vlans on the host is another (one or more) network(s) that your "dumb" switch cannot handle on it's own.
The "dumb" switch likely drops packets when the vlan tag is set. Again this is manufacturer specific, but generally w/ those extra 2 bytes present they're like 'nah man'.
This means you need the intermediary (the host) to do packet translation for you (bridge/pf/nat).

This is why when NOT using vlans if you stick everything into bridge0 it should just work w/out much modification.
The bridge acts as a switch and anything connected to it is within the same broadcast domain. That means everything works.

With vlans however a bit of, call it "trickery", needs to happen.
The easiest way imho to accomplish that trickery is, as described, creating separate bridges for your unique broadcast domains.

~Paul
 
pprocacci Thank you, that helps. One thing that is stumping me is the first part:

So talking strictly about the port connected to your switch, since it's not vlan aware, you just stick that port in bridge0 without any crazy options, and DHCP off of bridge0.

I've been stumped by this before. Can you provide a simple example of adding an inet or DHCP to a bridge?
 
toddg said:
pprocacci Thank you, that helps. One thing that is stumping me is the first part:

So talking strictly about the port connected to your switch, since it's not vlan aware, you just stick that port in bridge0 without any crazy options, and DHCP off of bridge0.

I've been stumped by this before. Can you provide a simple example of adding an inet or DHCP to a bridge?
Click to expand...

Sure.

# First create the bridge
ifconfig bridge create

That outputs something akin to `bridge0'.
Once you have the bridge name you can:

# Add the em0 interface to the bridge interface (assuming bridge0)
ifconfig bridge0 addm em0

# Bring up em0 and the bridge interface (assuming bridge0)
ifconfig em0 up
ifconfig bridge0 up

# For static assignment (use whatever your network is - assuming bridge0)
ifconfig bridge0 inet 192.168.0.2/24

# For dynamic assignment
dhclient bridge0

--
----
--

An applicable rc.conf for the above is:

cloned_interfaces="bridge0"
ifconfig_em0="up"
ifconfig_bridge0="addm em0 SYNCDHCP up" # if dynamically assigning addresses
ifconfig_bridge0="addm em0 inet 192.168.0.2/24 up" # if statically assigning addresses
 
toddg said:
BTW - can you recommend any books on this stuff? I have books about FreeBSD, Jails, Networking, etc. But nothing seems to cover VLANs and the sort of things we are talking about here.
Click to expand...
I was looking myself for VLANs documentation recently and found the following blog tutorial with detailed configuration examples. (it also includes VLAN and Jails). Perhaps you find it useful as I have.

How to Implement VLANs on FreeBSD

Step-by-step guide to implementing VLANs on FreeBSD. Covers VLAN concepts, if_vlan interface creation, rc.conf configuration, inter-VLAN routing, trunk…
www.freebsdsoftware.org www.freebsdsoftware.org

Table of Contents​

  1. VLAN Fundamentals
  2. FreeBSD VLAN Support
  3. Creating VLAN Interfaces
  4. Persistent Configuration in /etc/rc.conf
  5. Multiple VLANs on One Physical Interface
  6. Inter-VLAN Routing
  7. PF Firewall Rules Between VLANs
  8. VLAN and DHCP
  9. VLAN and Jails
  10. Switch Configuration Notes
  11. Troubleshooting
  12. FAQ
 
pprocacci Your instructions worked great, thank you. I'll attach my ifconfig output. I was a little confused at first bridge0 dhcp'd a different address. But now I see that, and can ssh to the host. Progress!

T-Daemon Thank you for the link! I'll give that a read next.

Bash: 
em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=48501bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
    ether 08:00:27:7f:a5:81
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33152
    options=0
    groups: pflog
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=10<VLAN_HWTAGGING>
    ether 58:9c:fc:10:7e:44
    inet 192.168.0.40 netmask 0xffffff00 broadcast 192.168.0.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 1 priority 128 path cost 20000 vlan protocol 802.1q
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
You have new mail in /var/mail/root
 
toddg said:
pprocacci Your instructions worked great, thank you. I'll attach my ifconfig output. I was a little confused at first bridge0 dhcp'd a different address. But now I see that, and can ssh to the host. Progress!

T-Daemon Thank you for the link! I'll give that a read next.

Bash: 
em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=48501bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
    ether 08:00:27:7f:a5:81
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33152
    options=0
    groups: pflog
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=10<VLAN_HWTAGGING>
    ether 58:9c:fc:10:7e:44
    inet 192.168.0.40 netmask 0xffffff00 broadcast 192.168.0.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 1 priority 128 path cost 20000 vlan protocol 802.1q
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
You have new mail in /var/mail/root
Click to expand...
.. and what you said about getting a new address makes sense.

When you get an ip address from a router or whatever other source is handing them out, they're generally tied to the mac address of the interface.
Once handed out it's called a lease and that lease is applicable for a specific mac until said lease expires.
Not going to go much further into details about the inner workings of DHCP but rather just a small FYI.

em0 in this case is 08:00:27:7f:a5:81
bridge0 in this case is 58:9c:fc:10:7e:44
 
You must log in or register to reply here.
Back
Top