jail cause mac address change (flip-floping)

Hello, I have made two jailed vms (jid 1235 and 1236) using epair interface and bridged that to physical. I have very limited FreeBSD experience (but growing interest) and extended Linux experience. Hence did it in the same manner I would do it on Linux. Please look at some outputs of the testing stand (bge1 unused):

Bash:
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                           
        options=80099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>                                                                                                   
        ether 00:1c:c4:81:bf:c2                                                                                                                                               
        inet 192.168.0.160 netmask 0xffffff00 broadcast 192.168.0.255                                                                                                         
        media: Ethernet autoselect (1000baseT <full-duplex,master>)                                                                                                           
        status: active                                                                                                                                                       
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>                                                                                                                 
bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                                               
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>                                                                                           
        ether 00:1c:c4:81:bf:c3                                                                                                                                               
        media: Ethernet autoselect                                                                                                                                           
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>                                                                                                                 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384                                                                                                             
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>                                                                                                       
        inet6 ::1 prefixlen 128                                                                                                                                               
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3                                                                                                                           
        inet 127.0.0.1 netmask 0xff000000                                                                                                                                     
        groups: lo                                                                                                                                                           
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                                                                                                                             
ab0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                                     
        ether 02:d1:88:e6:66:00                                                                                                                                               
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15                                                                                                           
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200                                                                                                             
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0                                                                                                             
        member: epair1236a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                                                                                     
                ifmaxaddr 0 port 6 priority 128 path cost 2000                                                                                                               
        member: epair1235a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                                                                                     
                ifmaxaddr 0 port 5 priority 128 path cost 2000                                                                                                               
        member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                                                                                           
                ifmaxaddr 0 port 1 priority 128 path cost 55                                                                                                                 
        groups: bridge                                                                                                                                                       
        nd6 options=9<PERFORMNUD,IFDISABLED>                                                                                                                                 
epair1235a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                     
        options=8<VLAN_MTU>                                                                                                                                                   
        ether 02:6f:e4:30:b7:0a                                                                                                                                               
        inet6 fe80::6f:e4ff:fe30:b70a%epair1235a prefixlen 64 scopeid 0x5                                                                                                     
        groups: epair                                                                                                                                                         
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                                                                                                                   
        status: active                                                                                                                                                       
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                                                                                                                             
epair1236a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                     
        options=8<VLAN_MTU>                                                                                                                                                   
        ether 02:9d:d1:69:c1:0a                                                                                                                                               
        inet6 fe80::9d:d1ff:fe69:c10a%epair1236a prefixlen 64 scopeid 0x6                                                                                                     
        groups: epair                                                                                                                                                         
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                                                                                                                   
        status: active                                                                                                                                                       
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The problem is I get constant messages about mac change between epair and bridge interface:

Code:
arp: 192.168.0.160 moved from 00:1c:c4:81:bf:c2 to 02:6f:e4:30:b7:0a on epair1235b                                                                                            
arp: 192.168.0.160 moved from 02:6f:e4:30:b7:0a to 00:1c:c4:81:bf:c2 on epair1235b

Mainly this happens for one of the vms which has more network activity.
I really need some way to stop this behaviour (not to hide the log message!).
Any explanation of why FreeBSD is doing this is appreciated!
 
Well, the solution seems to be simple. I assigned the hopping ip (192.168.0.160) to bridge itself (exactly as I did it on linux). Easier said but done.:( For some reason FreeBSD 12 scripts exclusively allowed only one of two things: either assign ip to the bridge or bring bge0 up. As soon as I fix one, script resets the other. After debuging network scripts I'm still unsure. It maybe just me being ignorant about BSD rc.conf craft or this is a bug:

Code:
# from /etc/rc.d/netif
        if [ -f /etc/rc.d/bridge -a -n "$cmdifn" ] ; then
                # I had to add quotes below. no way it can pass a list of interfaces otherwise.:(
                /etc/rc.d/bridge start "$cmdifn"
        fi

rc.conf if somebody cares:
Code:
network_interfaces="lo0 bge0 bridge0 ab0"
autobridge_interfaces="ab0"
cloned_interfaces="bridge0"
ifconfig_bridge0_name="ab0"
autobridge_ab0="bge0"
ifconfig_bge0="up"
ifconfig_ab0="192.168.0.160 netmask 255.255.255.0"
 
Code:
network_interfaces="lo0 bge0 bridge0 ab0"
autobridge_interfaces="ab0"
autobridge_ab0="bge0"
Remove those.

And use this:
Code:
ifconfig_ab0="addm bge0 inet 192.168.0.160 netmask 255.255.255.0"
 
No, you're right. I was sure I had that configured myself at some point. But I've used a completely different method (I'm using sysutils/vm-bhyve to create/manage them). It's definitely possible to add an IP address to a bridge(4) interface but getting it correct in rc.conf is a bit tricky and doesn't work the way I expected it to work.
 
In the end there are two sets of directives, that will work:

Code:
network_interfaces="lo0 bge0 bridge0 ab0"
autobridge_interfaces="ab0"
cloned_interfaces="bridge0"
ifconfig_bridge0_name="ab0"
autobridge_ab0="bge0"
ifconfig_bge0="up"
ifconfig_ab0="192.168.0.160 netmask 255.255.255.0"

and

Code:
network_interfaces="lo0 bge0 bridge0 ab0"
ifconfig_bge0="up"
cloned_interfaces="bridge0"
ifconfig_bridge0_name="ab0"
ifconfig_ab0="addm bge0 192.168.0.160 netmask 255.255.255.0"

In both cases network_interfaces list is mandatory. Quotation fix I used is not, though I probably raise a bug cause it's more like a coincidence.
 
Back
Top