Netgraph Bridge

Phishfry

Phishfry

I really think a good article on Netgraph bridges is needed. This seems to be a good primer from the author..

Daemon News '200003' : '"All About Netgraph "'

people.freebsd.org people.freebsd.org

The netgraph (4) manual is really busy and not real helpful for beginners.

netgraph(4)

www.freebsd.org www.freebsd.org

I did find these useful:

Klara has a useful article with Netgraph for Bhyve
klarasystems.com

Using Netgraph for FreeBSD’s Bhyve Networking

FreeBSD 13 adds new support for a netgraph backend for virtual network devices under bhyve. Netgraph is a modular networking framework that allows for arbitrary stacking of protocols and transports, along with filtering, tunneling, redirection, inspection, injection and more—fast and...
klarasystems.com klarasystems.com

Basic. Not real helpful.

Content here is Jail oriented ng_bridge.
ngctl dot | dot -Tsvg -o vgraph.svg

I changed the focus of the post to something more positive. I had disparaged Klara and that was not appropriate.
Netgraph is the subject. Any articles are helpful.
 
Last edited:
Several of our members have written helpful posts on Netgraph bridge.

Good example on last post
forums.freebsd.org

Ethernet interfaces on bridge using netgraph

Hello, I have some trouble while trying to implement 2 ethernet interfaces connected via a bridge using netgraph. I am using FreeBSD 10.1 release. What I want to do seems to be as simple as shown in the code section below, however I can't make sending/receiving broadcast messages...
forums.freebsd.org forums.freebsd.org

User benoitc has been struggling with this too:
forums.freebsd.org

failing to mix bridge, vlans and lagg with netgraph

I am trying touse netgraph for the bhyve vms with the following goals: I setup 2 bridge using an LACP bond one dedicated bridge for the vlan 1002 one bridge for all others vlans The network coming on the bridge that has untagged vlan 1002 is working, but it seems I can only get the vlan 1 on...
forums.freebsd.org forums.freebsd.org
forums.freebsd.org

netgraph ng_bridge: can't ge an IP using DHCP when creating a vlan

I am using a bridge `bridge0` that will contain a native nic interface ixl3 and add a tap interface in it: ifconfig bridge0 create ifconfig bridge0 addm ixl3 addm tap0 ifconfig bridge0 bridge0: flags=8843 metric 0 mtu 1500 ether 58:9c:fc:10:e7:20 id 00:00:00:00:00:00 priority 32768...
forums.freebsd.org forums.freebsd.org

The manual for ng_bridge(4) was unhelpful as well.

I have found these instructions that look promising:
Code: 
• Create the virtual ethernet interface
• # ngctl mkpeer . eiface hook ether
• Verify the interface exists, observe the MAC address is zeroed out
• # ifconfig ngeth0
• ngeth0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
• ether 00:00:00:00:00:00
• Bring up the virtual ethernet interface
• # ifconfig ngeth0 up
• ngeth0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
• inet6 fe80::2d0:9ff:fe4c:9e5f%ngeth0 prefixlen 64 scopeid 0x4
• ether 00:00:00:00:00:00
• Create the bridge and connect the lower link of the virtual interface
# ngctl mkpeer ngeth0: bridge lower link0
• Name the bridge
# ngctl name ngeth0:lower mybridge
• Connect the lower link to the physical interface
# ngctl connect dc0: mybridge: lower link1
• Connect the upper link to the physical interface
# ngctl connect dc0: mybridge: upper link2
• Connect the upper link to the virtual interface
# ngctl connect ngeth0: mybridge: upper link3
• Set the physical interface to not overwrite its source route
# ngctl msg dc0: setautosrc 0
• Set the physical interface into promiscous mode
# ngctl msg dc0: setpromisc 1
• Set the MAC address of the virtual interface
• # ifconfig ngeth0 link 00:5c:16:10:dd:79
• Set the IP address of the virtual interface
• # dhclient ngeth0
• # ifconfig ngeth0
• ngeth0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
• inet6 fe80::2de:adff:fe12:1212%ngeth0 prefixlen 64 scopeid 0x4
• inet 192.168.1.21 netmask 0xffffff00 broadcast 192.168.1.255
• ether 00:5c:16:10:dd:aa

Most all netgraph bridge instructions seem to miss the most basic first step.
Make the virtual interface ngeth0
 
Last edited:
Phishfry said:
Create the virtual ethernet interface
• # ngctl mkpeer . eiface hook ether
• Verify the interface exists, observe the MAC address is zeroed out
• # ifconfig ngeth0
• ngeth0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
• ether 00:00:00:00:00:00
• Bring up the virtual ethernet interface
• # ifconfig ngeth0 up
• ngeth0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
• inet6 fe80::2d0:9ff:fe4c:9e5f%ngeth0 prefixlen 64 scopeid 0x4
• ether 00:00:00:00:00:00
• Create the bridge and connect the lower link of the virtual interface
# ngctl mkpeer ngeth0: bridge lower link0
...
Click to expand...
Doesn't seem to work this way. It's a minor issue that MAC doesnt' zero out following this command.
But the last bridge command ngctl mkpeer ngeth0: bridge lower link0 doesn't work ...
I can only get ng to bridge with real NIC's ng node (like igb0) Though than one works fine. And since I bridge my igb1.10 iface, the VM connected to that has vlan ID 10 automatically.
 
free-and-bsd said:
Doesn't seem to work this way. It's a minor issue that MAC doesnt' zero out following this command.
But the last bridge command ngctl mkpeer ngeth0: bridge lower link0 doesn't work ...
I can only get ng to bridge with real NIC's ng node (like igb0) Though than one works fine. And since I bridge my igb1.10 iface, the VM connected to that has vlan ID 10 automatically.
Click to expand...
Seeing how easily this works, I can't see why ng_vlan is needed at all... As per man, it must be linked to a real nic.
Does it have to be a NIC supporting vlan tags? Well if it does, then nothing prevents one from creating a $vlan_tag interface, then you won't need ng_vlan to create a vlan. But if the real NIC does not support vlan tagging, then I suppose it won't support ng_vlan based VLAN either (if that is its purpose)?
As you said, very little information about netgraph at all. Shame, too, as this feature has been around for quite a while... Does this lack of documentation mean it isn't much used? If so, then why is it kept?...
Though I must say, in my setup a simple ng bridge to real NIC works fine to connect to my VM.
 

ngbridge « rc.d - tools - assorted systems management scripts

gitr.daemon.contact gitr.daemon.contact

This is a working implementation, you can start from this.
It builds bridges and point-to-point connections.

To build a bridge (in /etc/rc.conf.local):
Code: 
ngbridge_names="... lan ..."
# physical interfaces to connect (if applicable):
ngbridge_lan_physifs="alc0 fxp2"
# physical interface you want to see from the OS (not recommended, because
# then the bridge is offline when this iface is unplugged):
ngbridge_lan_hostif="alc0"
# virtual interfaces and their configurations:
ngbridge_lan_eifaces="nge_1u nge_2u"
ngbridge_nge_1u_mac="00:37:92:01:02:02"
ngbridge_nge_1u_addr_num="1"
ngbridge_nge_1u_addr_1="inet 192.168.0.17/28"
ngbridge_nge_2u_mac="00:37:92:01:02:01"
ngbridge_nge_2u_addr_num="2"
ngbridge_nge_2u_addr_1="inet 192.168.0.19/28"
ngbridge_nge_2u_addr_2="inet6 fd00::9876 prefixlen 120 auto_linklocal"
# interfaces you want to persist during "service ngbridge restart":
ngbridge_lan_eifaces_keep="nge_1u"
# routes that can only be configured after the bridge is up (invoked in descending seq):
ngbridge_lan_route_num=2
ngbridge_lan_route_2="-6 -net fd00::/64 fd00::1234"
ngbridge_lan_route_1="-net default 192.168.33.1"
ngbridge_lan_vlans="NO"

To connect a jail to this bridge via an additional eiface, put this in /etc/jail.conf:
Code: 
myjail {
        vnet = "new";
        $ifname1l = n${name}1l;
        $ifname1l_mac = 00:37:92:01:01:02;
        vnet.interface = "$ifname1l";
        exec.prestart = "
            echo -e \"mkpeer eiface crhook ether\nname .:crhook $ifname1l\" \
                | /usr/sbin/ngctl -f -
            /usr/sbin/ngctl connect ${ifname1l}: lanbridge: ether link17
            ifname=`/usr/sbin/ngctl msg ${ifname1l}: getifname | \
                awk '$1 == \"Args:\" { print substr($2, 2, length($2)-2)}'`
            /sbin/ifconfig \$ifname name $ifname1l
            /sbin/ifconfig $ifname1l link $ifname1l_mac
        ";
        exec.poststop = "
                sleep 1 ;
                /usr/sbin/ngctl shutdown ${ifname1l}: ;
        ";
}

To build a point-to-point connection:
(A point-to-point connection has exactly two virtual ifaces and nothing else, so there is no point in creating a bridge for that)
Code: 
ngbridge_names="... tele ..."
# two virtual interfaces and their configurations:
ngbridge_tele_eifaces="ngtele1u ngtele1l"
ngbridge_ngtele1u_mac="00:37:92:01:03:01"
ngbridge_ngtele1l_mac="00:37:92:01:03:02"
ngbridge_ntele1u_addr_num=2
ngbridge_ntele1u_addr_1="inet 192.168.0.129/30"
ngbridge_ntele1u_addr_2="inet6 fd00::abcd prefixlen 120 auto_linklocal"
ngbridge_tele_vlans="NO"

To use this for a jail:
Code: 
tele {
        vnet = "new";
        vnet.interface = "n${name}1l";
}

If you need vlans, you can create like this:
Code: 
ngbridge_names="... vl ..."
ngbridge_vl_physifs="igb0"
ngbridge_vl_vlans="YES"

This creates a bridge, and a ng_vlan(4) node attached to it. You can then wire to that node from anywhere else and add a filter for some vlan - like it's done in my bhyve rc.d.

Code: 
  Name: igb0            Type: ether           ID: 00000002   Num hooks: 1
  Local hook      Peer name       Peer type    Peer ID         Peer hook     
  ----------      ---------       ---------    -------         ---------     
  lower           vlbridge      bridge       0000000c        link0         

  Name: vlbridge      Type: bridge          ID: 0000000c   Num hooks: 2
  Local hook      Peer name       Peer type    Peer ID         Peer hook     
  ----------      ---------       ---------    -------         ---------     
  link1           vl_vlan       vlan         00000011        downstream    
  link0           igb0            ether        00000002        lower         

  Name: vl_vlan       Type: vlan            ID: 00000011   Num hooks: 2
  Local hook      Peer name       Peer type    Peer ID         Peer hook     
  ----------      ---------       ---------    -------         ---------     
  vlan7           tap1            ether        00000042        lower         
  downstream      vlbridge       bridge       0000000c        link1
 
Can I please get a sanity check on my Firewall plans.
I want to bridge two Chelsio T540 cards for a NFS Network from my networks firewall. A high speed backbone.
All my Chelsio interfaces are named 'cxl'
ifconfig -l
Code: 
cxl0 cxl1 cxl2 cxl3 cxl4 cxl5 cxl6 cxl7 cxl8 cxl9 cxl10 cxl11 em0 em1 lo0 lagg0 pflog0

Currently I am using cxl0,cxl1,cxl2,cxl3 in a LAGG arrangement to top of rack Cisco SG550X with 48 copper ports
Code: 
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    options=6e800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP>
    ether 00:07:43:35:8b:80
    inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
    laggproto roundrobin lagghash l2,l3,l4
    laggport: cxl0 flags=4<ACTIVE>
    laggport: cxl1 flags=4<ACTIVE>
    laggport: cxl2 flags=4<ACTIVE>
    laggport: cxl3 flags=4<ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

WAN is provided by cable modem over motherboard em0 interface.

I have Chelsio interfaces cxl4-cxl11 unused.

I have 5 servers in a rack with 10G I want to make a NFS Network with.

Seeing how these rack machines are serviced by the Cisco switch for copper ethernet I was thinking of creating a secondary network for just NFS file transfers.

Penciled in /etc/rc.conf
Code: 
#### Chelsio 10G network ######
#cloned_interfaces="bridge0"
#ifconfig_bridge0="inet 10.0.0.1 netmask 255.0.0.0 addm cxl4 addm cxl5 addm cxl6 addm cxl7 addm cxl8 addm cxl9 addm cxl10 addm cxl11"
#ifconfig_cxl4="up"
#ifconfig_cxl5="up"
#ifconfig_cxl6="up"
#ifconfig_cxl7="up"
#ifconfig_cxl8="up"
#ifconfig_cxl9="up"
#ifconfig_cxl10="up"
#ifconfig_cxl11="up"
#

I would much rather use Netgraph Bridge for this secondary high-speed NFS network.

Is this feasible? A secondary network for NFS ?? Would a Netgraph Bridge be more appropriate here?
 
PMc said:

ngbridge « rc.d - tools - assorted systems management scripts

gitr.daemon.contact gitr.daemon.contact

This is a working implementation, you can start from this.
It builds bridges and point-to-point connections.

To build a bridge (in /etc/rc.conf.local):
Code: 
ngbridge_names="... lan ..."
# physical interfaces to connect (if applicable):
ngbridge_lan_physifs="alc0 fxp2"
# physical interface you want to see from the OS (not recommended, because
# then the bridge is offline when this iface is unplugged):
ngbridge_lan_hostif="alc0"
# virtual interfaces and their configurations:
ngbridge_lan_eifaces="nge_1u nge_2u"
ngbridge_nge_1u_mac="00:37:92:01:02:02"
ngbridge_nge_1u_addr_num="1"
ngbridge_nge_1u_addr_1="inet 192.168.0.17/28"
ngbridge_nge_2u_mac="00:37:92:01:02:01"
ngbridge_nge_2u_addr_num="2"
ngbridge_nge_2u_addr_1="inet 192.168.0.19/28"
ngbridge_nge_2u_addr_2="inet6 fd00::9876 prefixlen 120 auto_linklocal"
# interfaces you want to persist during "service ngbridge restart":
ngbridge_lan_eifaces_keep="nge_1u"
# routes that can only be configured after the bridge is up (invoked in descending seq):
ngbridge_lan_route_num=2
ngbridge_lan_route_2="-6 -net fd00::/64 fd00::1234"
ngbridge_lan_route_1="-net default 192.168.33.1"
ngbridge_lan_vlans="NO"

To connect a jail to this bridge via an additional eiface, put this in /etc/jail.conf:
Code: 
myjail {
        vnet = "new";
        $ifname1l = n${name}1l;
        $ifname1l_mac = 00:37:92:01:01:02;
        vnet.interface = "$ifname1l";
        exec.prestart = "
            echo -e \"mkpeer eiface crhook ether\nname .:crhook $ifname1l\" \
                | /usr/sbin/ngctl -f -
            /usr/sbin/ngctl connect ${ifname1l}: lanbridge: ether link17
            ifname=`/usr/sbin/ngctl msg ${ifname1l}: getifname | \
                awk '$1 == \"Args:\" { print substr($2, 2, length($2)-2)}'`
            /sbin/ifconfig \$ifname name $ifname1l
            /sbin/ifconfig $ifname1l link $ifname1l_mac
        ";
        exec.poststop = "
                sleep 1 ;
                /usr/sbin/ngctl shutdown ${ifname1l}: ;
        ";
}

To build a point-to-point connection:
(A point-to-point connection has exactly two virtual ifaces and nothing else, so there is no point in creating a bridge for that)
Code: 
ngbridge_names="... tele ..."
# two virtual interfaces and their configurations:
ngbridge_tele_eifaces="ngtele1u ngtele1l"
ngbridge_ngtele1u_mac="00:37:92:01:03:01"
ngbridge_ngtele1l_mac="00:37:92:01:03:02"
ngbridge_ntele1u_addr_num=2
ngbridge_ntele1u_addr_1="inet 192.168.0.129/30"
ngbridge_ntele1u_addr_2="inet6 fd00::abcd prefixlen 120 auto_linklocal"
ngbridge_tele_vlans="NO"

To use this for a jail:
Code: 
tele {
        vnet = "new";
        vnet.interface = "n${name}1l";
}

If you need vlans, you can create like this:
Code: 
ngbridge_names="... vl ..."
ngbridge_vl_physifs="igb0"
ngbridge_vl_vlans="YES"

This creates a bridge, and a ng_vlan(4) node attached to it. You can then wire to that node from anywhere else and add a filter for some vlan - like it's done in my bhyve rc.d.

Code: 
  Name: igb0            Type: ether           ID: 00000002   Num hooks: 1
  Local hook      Peer name       Peer type    Peer ID         Peer hook    
  ----------      ---------       ---------    -------         ---------    
  lower           vlbridge      bridge       0000000c        link0        

  Name: vlbridge      Type: bridge          ID: 0000000c   Num hooks: 2
  Local hook      Peer name       Peer type    Peer ID         Peer hook    
  ----------      ---------       ---------    -------         ---------    
  link1           vl_vlan       vlan         00000011        downstream   
  link0           igb0            ether        00000002        lower        

  Name: vl_vlan       Type: vlan            ID: 00000011   Num hooks: 2
  Local hook      Peer name       Peer type    Peer ID         Peer hook    
  ----------      ---------       ---------    -------         ---------    
  vlan7           tap1            ether        00000042        lower        
  downstream      vlbridge       bridge       0000000c        link1
Click to expand...
Wow thank you very much. I've already used Klara manual for a simple bridge and it's working for my Win10 bhyve VM.

But I have this small testing project, OpenBSD in bhyve VM acting as router, which has all 3 host's NICs bridged to it (no passthrough on that machine). Now using if_bridge, but want to see if ng_bridge is better.
 
free-and-bsd said:
But I have this small testing project, OpenBSD in bhyve VM acting as router, which has all 3 host's NICs bridged to it (no passthrough on that machine). Now using if_bridge, but want to see if ng_bridge is better.
Click to expand...
I have no idea if it is "better" (by what means?); I prefer it because it keeps these things out of the -already bloated- ifconfig setups. It is something like a second cardboard where you can start wiring things independently. For very simple setups this is another layer of complexity, for more elaborate ones it becomes fun.
Also since R.13 you can plug bhyve directly into netgraph. And you can route flows from ipfw into netgraph and back.
 
Oh I read somewhere that they may be faster... Or was it about netmap? Anyway, in my case it's at least "simpler" because my VM only uses "lower hook" to connect to bridged igb1.10 -- no additional tapX interface. Looks kinda simpler.
 
You must log in or register to reply here.
Back
Top