Clarification on vlans, interfaces, clones, and behaviour.

[connchri]

I have a server with 5 interfaces. All 1Gbps, 4 of which are on an intel 4 port card.

I have set up LACP with the 4 Intel ports - working fine with a procurve switch, ifconfig shows it having 4Gbps connection. I also have numerous vlans running over this without issue, all for servicing jails.

The remaining 1Gbps port is on a Broadcom interface - and I use this for connecting to the host for admin.

One thing I can't seem to understand, in my configuration I have the following line:

cloned_interfaces="lagg0"

I cannot find the reason for this, but included it because numerous other tutorials have. What is the purpose of cloned_interfaces? Is it required? I can't find anything about it in ifconfig(8) man page.

In addition, for security, I wanted to disconnect the Broadcom ethernet connection - just to keep the host installation isolated from the network. I can connect it when I need to admin. When I do pull this patch cable though, the entire server (lagg0 and jails) also disconnects. I don't know if this is because FreeBSD is freaking out, or if there's some behavior I'm unaware of. But the result is I can't do what I want without complete disconnection.

Here's the setup for my lagg:

############# 4Gbps Lagg connection #########

# Bring up interfaces:
ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_em2="up"
ifconfig_em3="up"

# Create lagg0
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 laggport em2 laggport em3"

# Set up Vlans
vlans_lagg0="1 2 3 4 5 6"

# Create vlan interfaceson Lagg0
ifconfig_lagg0_1="inet 192.168.1.41/24"
ifconfig_lagg0_1_alias0="inet 192.168.1.45/32"
ifconfig_lagg0_3="inet 192.168.3.45/24"

For my Broadcom interface I have:

ifconfig_bge0="inet 192.168.1.40 netmask 255.255.255.0"
defaultrouter="192.168.1.1"

Nothing special at all. I just don't understand why pulling this connection makes the rest go down as well.

Cheers.

 

[phoenix]

The cloned_interfaces line causes the RC script to call ifconfig create lagg0 to create the pseudo-interface. Then it can be configured via the ifconfig_* lines just like any other interface.

What do the logs say when you unplug the Broadcom NIC?

 

[`Orum]

In addition, for security, I wanted to disconnect the Broadcom ethernet connection - just to keep the host installation isolated from the network. I can connect it when I need to admin.

Are you sure this interface isn't used for something else, like IPMI? If so, I strongly recommend you don't use it at all within FreeBSD (if it's even visible). What I do for machines like this is simply put the IPMI interface on a separate, secure VLAN as it really shouldn't be accessible to anyone you wouldn't trust within your server room.

Netmask 255.255.255.0 means : first adress, 192.168.1.0, last address 192.168.1.254 = CIDR 24

Well, the first usable address is 192.168.1.1, not .0

 

[sko]

I suspect your default route is attached to the bge0 interface, therefore it gets removed when the interface goes down. Have a look at the "netif" column at the output of netstat -nr4.

The route should be linked to the lagg0 interface if you specify it right after the lagg0 configuration and before the ifconfig_bge0 line in your /etc/rc.conf

This of course would also route all host traffic via the lagg0 interface (because the host also has an address on the lagg0 interface!), so the better solution would be:
1) use a dedicated VLAN for management interfaces. VLAN1 like in your configuration is exceptionally bad, as everything plugged into an unconfigured switchport ends up in this default VLAN! The management-VLAN should be completely separated and not be accessible from any other network; usually you have to connect either via access ports or VPN to that network for administrative tasks.
and
2) use seperate FIBs for jails so they can have their own default gateway specified. This is a requirement if you have jails in different VLANs as they need to have their gateway within the same VLAN to be able to reach it. (or you have to deal with very ugly and hard-to-debug NAT and RDRs from/into the various VLANs on the host).

 

[SirDice]

The route should be linked to the lagg0 interface if you specify it right after the lagg0 configuration and before the ifconfig_bge0 line in your /etc/rc.conf

The order of things in rc.conf is completely irrelevant.

 

[sko]

The order of things in rc.conf is completely irrelevant.

Thanks, I was not aware of this.
So there is no way to influence the order in which network interfaces are brought up / configured?


However, I just had a closer look at route(8) and found what I initially looked for but couldn't find (because it is hidden nearly at the end of the description...):

In a change or add command where the destination and gateway are not
sufficient to specify the route (as in the ISO case where several
interfaces may have the same address), the -ifp or -ifa modifiers may be
used to determine the interface or interface address.

so setting defaultrouter="192.168.1.1 -ifp lagg0_1" should attach the route to the lagg0 interface.

 

[SirDice]

Thanks, I was not aware of this.
So there is no way to influence the order in which network interfaces are brought up / configured?

Nope. The rc.conf is in essence a shell script (only containing variables) that gets sourced multiple times by various rc(8) scripts.

 

[kpa]

If you want to see the order in which various bits are configured you have to use rcorder(8):

rcorder /etc/rc.d/* /usr/local/etc/rc.d/*

That gives the order of execution for rc(8) scripts on system boot. To get a better idea how the network interfaces get configured you have to read the /etc/rc.d/netif script.