Multiple Nics, Networks, VNET and Iocage

I recently did some fairly complex networking for a project and thought .. hey maybe someone else may find it useful....

I assume the following:

multiple network cards
freebsd 13
iocage with Vnets
---
This KB scales with how ever many network cards you have.. it assumes (2) igb0 and igb1 .. if you have 6 then just add more .. likewise if you only have 1.. remove anything wiht igb1 in it.

You should end up with 4 jails.
igb0-wan with vnet0 using DHCP and vnet1 configured as the 10.10.10.1 gateway and 1 jail igb0-application at 10.10.10.100
igb1-wan with vnet0 using DHCP and vnet1 configured as the 172.16.24.1 gateway and 1 jail igb1-application at 172.16.24.100


/etc/sysctl.conf
Code:
net.inet.ip.forwarding=1
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0
adds some back end plumbing

/etc/rc.conf
Code:
iocage_enable="YES"

# enable pf and configure a path for your rule set
pf_enable="YES"
pflog_enable="YES"
pf_rules="/etc/pf.conf"

# allow the host to act as a gateway
gateway_enable="YES"

# rename your network cards
ifconfig_igb0_name="igb0wan"
ifconfig_igb1_name="igb1wan"
# bring them up
ifconfig_igb0wan="up"
ifconfig_igb1wan="up"

# create your internal and external bridges
cloned_interfaces="bridge0 bridge1 bridge2 bridge3"

# name your bridges so you can attach them properly ie public vs private
ifconfig_bridge0_name="igb0wanbridge"
ifconfig_bridge1_name="igb1wanbridge"
ifconfig_bridge2_name="igb0pbridge"
ifconfig_bridge3_name="igb1pbridge"

# bring your bridges up
ifconfig_igb0wanbridge="addm igb0wan up"
ifconfig_igb1wanbridge="addm igb1wan up"

NOTE: if you had 3 network cards, just copy these 4 lines and change the interface id.. ir igb4
Code:
ifconfig_igb0_name="igb0wan"
ifconfig_igb0wan="up"
cloned_interfaces="bridge0 ...
ifconfig_bridge0_name="igb0wanbridge"
ifconfig_bridge2_name="igb0pbridge"

normally just a service restart will do, but I would reboot at this point to make sure.

upon restart you should see something like this ...

ifconfig
interfaces should be renamed to igb0wan/igb1wan
Code:
igb0wan: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
ether xxxxxxxxxxxx
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1wan: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
ether xxxxxxxxxxxxx
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
... cut
4 new bridges, igb0wanbridge, igb0pbridge, igb1wanbridge, igb1pbridge
aka 2 public sides and 2 private sides
Code:
igb0wanbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xxxxxxxxxxxxx
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb0wan flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
igb1wanbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xxxxxxxxxxxx
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb1wan flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
igb0pbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether xxxxxxxxxxxxxx
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
igb1pbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xxxxxxxxxxxxx
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>

igb0wanbridge -- the front side of the bridge
igb0pbridge -- the private side
at this point there are no jails associated to any of the bridges

Let’s create the WAN/LAN gateway

the gateway jails igb0-wan/igb1-wan basically have 2 network cards in them, the first network adaptor connects to the WAN and the other to the LAN. Just like any normal firewall device. This allows you to route, block, forward or scrub traffic just like any other firewall. It also allows you to tag your internal application traffic.

.172 Network WAN
Code:
iocage create -r 13.0-RELEASE priority=50 boot=on allow_raw_sockets=1 vnet=on bpf=1 defaultrouter=10.100.10.1 ip4_addr="vnet0|dhcp,vnet1|172.16.24.1/24" interfaces="vnet0:igb0wanbridge,vnet1:igb0pbridge" resolver=10.100.10.1 -n igb0-wan

priority=50 raises the boot up priority to ensure this jail comes up first
boot=on tells the host to bring the jail up on boot up
allow_raw_sockets=1 enables the gateway jail to use pf
bpf = required to use dhcp
defaultrouter= (generally your network gateway)
ip4_addr = vnet0 uses dhcp to fetch an ip, vnet1 assigns the gateway ip you want to use for your internal network
interfaces = vnet0: bind to bridge device (in this case its the wan/internet side of the bridge) vnet1 is the other end that attaches to the internal network
resolver = what ever ip you want to use for dns
-n name of jail

In my case I am using to physical cables, one plugged into igb0 that goes to network 1 and the other into igb1 that goes to network 2... they are totally separate so that I can destory and rebuild them at will without touching anything else.

In theory if you had 10 nics, you could aslo build lags more bridges and tunnels galore.

.10 Network WAN
Code:
iocage create -r 13.0-RELEASE priority=51 boot=on allow_raw_sockets=1 vnet=on bpf=1 defaultrouter=10.100.10.1 ip4_addr="vnet0|dhcp,vnet1|10.10.10.1/24" interfaces="vnet0:igb1wanbridge,vnet1:igb1pbridge" resolver=10.100.10.1 -n igb1-wan


this is the command for the wan jail on igb1.. I’ve included it for comparison , if you want more .. up arrow and change the good bits (assuming you created more bridges in the /etc/rc.conf above

Now lets add a jail to the private network

.172 Network Application

Code:
iocage create -r 13.0-RELEASE priority=60 boot=on vnet=on defaultrouter=172.16.24.1 ip4_addr="vnet0|172.16.24.100/24" interfaces="vnet0:igb0pbridge" resolver=172.16.24.1 -n igb0-application

.10 Network Application
Code:
iocage create -r 13.0-RELEASE  priority=61 boot=on vnet=on defaultrouter=10.10.10.1 ip4_addr="vnet0|10.10.10.100/24" interfaces="vnet0:igb1pbridge" resolver=10.10.10.1 -n igb1-application

there are some minor differences, such as the boot priority, the routers use the wan-jails ip address and the local lan ip is 124.100 likewise the resolver is set to the upstream wan's internal ip.

Lets bring it up!

iocage list
Code:
+-----+------------------+-------+--------------+------------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+==================+=======+==============+==================+
| 7 | igb0-application | up | 13.0-RELEASE | 172.16.24.100 |
+-----+------------------+-------+--------------+------------------+
| 5 | igb0-wan | up | 13.0-RELEASE | dhcp,172.16.24.1 |
+-----+------------------+-------+--------------+------------------+
| 8 | igb1-application | up | 13.0-RELEASE | 10.10.10.100 |
+-----+------------------+-------+--------------+------------------+
| 6 | igb1-wan | up | 13.0-RELEASE | dhcp,10.10.10.1 |
+-----+------------------+-------+--------------+------------------+
when you do an ifconfig on the host you should see all of your vnet devices..

ifconfig
.... cut sample.
Code:
vnet0.8: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: igb1-application as nic: epair0b
options=8<VLAN_MTU>
ether xxxxxxxxxxx
hwaddr 02:23:95:84:97:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

presto..

Testing

iocage console igb0-wan (aka the .172 network)
the WAN host can ping the internet,its gateway, or hosts on the 172 network

Code:
igb0-wan:~ # ping 10.100.10.1
PING 10.100.10.1 (10.100.10.1): 56 data bytes
64 bytes from 10.100.10.1: icmp_seq=0 ttl=255 time=0.256 ms
root@igb0-wan:~ # ping cnn.com
PING 151.101.65.67 (151.101.65.67): 56 data bytes
64 bytes from 151.101.65.67: icmp_seq=0 ttl=255 time=0.230 ms
root@igb0-wan:~ # ping 172.16.24.1
PING 172.16.24.1 (172.16.24.1): 56 data bytes
64 bytes from 172.16.24.1: icmp_seq=0 ttl=64 time=0.145 ms
root@igb0-wan:~ # ping 172.16.24.100
PING 172.16.24.100 (172.16.24.100): 56 data bytes
64 bytes from 172.16.24.100: icmp_seq=0 ttl=64 time=0.252 ms


Additional Notes:

In my case this entire project was to simply make something that can be built, managed and destroyed by ansible without touching anything else. I thought it would be a good learing tool to share and may help others.

please feel free to reply with your thoughts & comments, updates, or additions.

Cheers!
 
Back
Top