9.3-R - Set up LACP lagg, run VLANs over it and add route tables for each jail (setfib, fib)

This is my first HowTo ever so bear with me ...

This setup was something that in the end was very simple and should have taken only 15 minutes but due to my limited FreeBSD experience and online documentation that for me and my distro was lacking, it took me almost a solid week. I write this so you don't have to go through what I did... My actual configuration will be posted in the end as a working example.

In my scenario I have a G5 HP server that I wanted to double as a Drupal development server AND an egroupware server, servicing different types of endusers through the subnets on my network they have access to. The embedded NIC is used for host administration on a subnet only sysadmins have access to. Nothing is accessible from the internet but all have internet access through their respective interface; Jail traffic passes over its vlan through the lagg exclusively

This howto assumes the ports on your switch are setup correctly for a LACP LAGG and your vlans tags are tagged over the trunk. It also assumes you have basic working knowledge of FreeBSD / *nix commands

Let's get to it ...

First define in /boot/loader.conf how many distinct route tables your box will serve and not to share tables across FIBs

Code:
net.fibs=5
net.add_addr_allfibs=0
FreeBSD 9.3 DOES NOT need a kernel recompile to add extra routing tables!

declare how many routing tables you want. Max =16
Code:
net.fib=x

Determine whether or not routing information gets shared across FIBs. 0 = disabled
Code:
net.add_add_allfibs=x

Time to setup your networks and adapters in /etc/rc.conf

First setup the host interface (static IP)
Code:
hostname="YOUR-HOSTNAME-HERE"
ifconfig_(your_nic_id)="inet (your_local_static_ip) netmask (your_netmask)"
defaultrouter="(your_local_gateway_ip"
IMPORTANT! FreeBSD applies rc.conf() configurations in the order it reads them - make sure to layer your commands in the order you want them to be applied @ boot time. eg., the lag needs to come up before its vlans can come up otherwise you get errors in dmesg

Setup the LAGG and Vlans
https://www.freebsd.org/doc/handbook/network-aggregation.html

Code:
ifconfig_(your_1st_lag_nic_id)="up"
ifconfig_(your_2nd_lag_nic_id)="up"
ifconfig_(your_3rd_lag_nic_id)="up"
ifconfig_(your_4th_lag_nic_id)="up"
cloned_interfaces="(lagg_id) vlan(id) vlan(id)"
ifconfig_(lagg_id)="laggproto lacp laggport (1st_nic_id) laggport (2nd_nic_id) laggport (3rd_nic_id) laggport (4th_nic_id)"
ifconfig_(vlan_id)="vlan (id) vlandev (lagg_id)"
ifconfig_(vlan_id)="vlan (id) vlandev (lagg_id)"

Because the interfaces used to build the lagg are not assigned an ip address, you have to bring them up from the down state before you can use them
Code:
ifconfig_(each_nic_used_for_lagg)="up"

define which virtual interfaces will be passing traffic
Code:
cloned_interfaces="(lagg_id) vlan(id) vlan(id)"

build the lagg using lacp protocol
Code:
ifconfig_(lagg_id)="laggproto lacp laggport (1st_nic_id) laggport (2nd_nic_id) laggport (3rd_nic_id) laggport (4th_nic_id)"

Define the Vlans and assign them to the LAGG interface

Code:
ifconfig_(vlan_id)="vlan (id) vlandev (lagg_id)"
ifconfig_(vlan_id)="vlan (id) vlandev (lagg_id)"
Almost every online example assigns an IP address / netmask to the LAGG and / or Vlan interfaces at this stage but in my configuration that is unwanted; I only want IPs assigned to the jail itself

OK! If you save the /etc/rc.conf now and reboot without doing anything else, your lagg should be up and ready to pass traffic.

Let's get back into /etc/rc.conf to finish up ...

Define static routes for each FIB
Code:
static_routes="(route_1_network_name)(route_1_gateway_name)(route_2_network_name)(route_2_gateway_name)"
route_(route_1_network_name)="-net (network_address_1/mask) -interface vlan(id) -fib (id)"
route_(route_2_network_name)="-net (network_address_2/mask) -interface vlan(id) -fib (id)"
route_(route_1_gateway_name)="default (network_1_gateway) -fib (id)"
route_(route_2_gateway_name)="default (network_2_gateway) -fib (id)"
Make sure to declare the network first before the gateway path otherwise you'll get errors on boot!

Every FIB howto I found online used rc.local to host the routes for the FIBs - However using rc.local to run commands at boot is a throwback from yester-year and you can see from my config, is not necessary to assign routes to FIBs. Everything can be done in rc.conf

Save /etc/rc.conf and reboot. Everything should look normal in dmesg.

I used Ezjail for my jails, after reading it seemed easier to administer the jails with no detriment to my particular setup. Install ezjail and create your jails before heading to the next step. It's very easy to do and well documented so no need for me to include in this howto: http://erdgeist.org/arts/software/ezjail/

Find each specific jail config file under /usr/local/etc/ezjail (this should be the default location)

Add jail IP information and assign jail to a FIB by changing the following settings

Code:
export jail_www_drupal_devel_ip="(vlan-interface|ip_address_for_jail/mask)"
export jail_www_drupal_devel_fib="(fib_id)"
Make sure the jail's IP information matches the FIB you assigned it otherwise you won't have network access!!!

--- EXTRA ---
by default jails do NOT allow raw sockets use for security reasons - this mean no ping! if you want your jail to use ping, modify this setting;

from;
Code:
export jail_www_drupal_devel_parameters=""
to:
Code:
export jail_www_drupal_devel_parameters="allow.raw_sockets=1"
Now restart your jails ezjail-admin restart (your_jail_name) and you should be good to go! Good Luck


********************************* My Config Files ***********************************

loader.conf
Code:
zfs_load="YES"
net.fibs=5
net.add_addr_allfibs=0

rc.conf
Code:
##
# Host Interface
##
hostname="local-wwwdevel"
ifconfig_bce0="inet 10.0.1.5 netmask 0xffffff00"
defaultrouter="10.0.1.1"
#
##
#
sshd_enable="YES"
moused_enable="YES"
#
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
#
zfs_enable="YES"
#
##
# Lagg / vlan config
##
#
ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_em2="up"
ifconfig_em3="up"
cloned_interfaces="lagg0 vlan100 vlan60"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 laggport em2 laggport em3"
ifconfig_vlan100="vlan 100 vlandev lagg0"
ifconfig_vlan60="vlan 60 vlandev lagg0"
#
##
# Static routes
##
# 0: default = host
# 1: devel = production users
# 2: egroup = office users
# 3:
# 4:
##
static_routes="develnet develgate egroupnet egroupgate"
route_develnet="-net 10.0.10.0/29 -interface vlan100 -fib 1"
route_egroupnet="-net 10.0.6.0/28 -interface vlan60 -fib 2"
route_develgate="default 10.0.10.1 -fib 1"
route_egroupgate="default 10.0.6.1 -fib 2"
#
##
# Ezjail Config
##
ezjail_enable="YES"
ezjail_use_zfs="YES"
ezjail_use_zfs_for_jails="YES"
ezjail_jailzfs="zroot/usr/jails"

my_jail_conf_file
Code:
# To specify the start up order of your ezjails, use these lines to
# create a Jail dependency tree. See rcorder(8) for more details.
#
# PROVIDE: standard_ezjail
# REQUIRE:
# BEFORE:
#

export jail_www_drupal_devel_hostname="www.drupal.devel"
export jail_www_drupal_devel_ip="vlan100|10.0.10.2/29"
export jail_www_drupal_devel_rootdir="/usr/jails/www.drupal.devel"
export jail_www_drupal_devel_exec_start="/bin/sh /etc/rc"
export jail_www_drupal_devel_exec_stop=""
export jail_www_drupal_devel_mount_enable="YES"
export jail_www_drupal_devel_devfs_enable="YES"
export jail_www_drupal_devel_devfs_ruleset="devfsrules_jail"
export jail_www_drupal_devel_procfs_enable="YES"
export jail_www_drupal_devel_fdescfs_enable="YES"
export jail_www_drupal_devel_image=""
export jail_www_drupal_devel_imagetype=""
export jail_www_drupal_devel_attachparams=""
export jail_www_drupal_devel_attachblocking=""
export jail_www_drupal_devel_forceblocking=""
export jail_www_drupal_devel_zfs_datasets=""
export jail_www_drupal_devel_cpuset=""
export jail_www_drupal_devel_fib="1"
export jail_www_drupal_devel_parentzfs=""
export jail_www_drupal_devel_parameters="allow.raw_sockets=1"
export jail_www_drupal_devel_post_start_script=""
export jail_www_drupal_devel_retention_policy=""
 
Back
Top