ARP who-has leaking?

tarkhil

tarkhil

Hello

I'm running a FreeBSD box with lots of vnet jails on private address, hosted in Hetzner.

They constantly complained (and blocked me) on wrong ARP addresses appears on their router.

tcpdumping those addresses, I've found that they leaks on ARP who-has requests from vnet jails.

All of them are running on bridge not connected to external interface; I use em0.100 for .1 in my jail network. However, who-has still appears on em0.

I've tried to mark all interface on internal bridge as private; I've lost connectivity to all my jails.

Currently, I've stopped leaking by writing permanent arp tables in each jail, but that's not a permanent solution. How can L2 packet escape from bridge?
 
Please post the ifconfig output of the various interfaces, particularly the bridge and em0.100.

tarkhil said:
All of them are running on bridge not connected to external interface; I use em0.100 for .1 in my jail network. However, who-has still appears on em0.
Click to expand...
That's not entirely correct. Your bridge is connected to the external interface, assuming em0 is the external interface. You only wrapped the packets with a VLAN tag, and they're going out on em0 with that VLAN tag.
 
SirDice said:
Please post the ifconfig output of the various interfaces, particularly the bridge and em0.100.


That's not entirely correct. Your bridge is connected to the external interface, assuming em0 is the external interface. You only wrapped the packets with a VLAN tag, and they're going out on em0 with that VLAN tag.
Click to expand...
Damn, it's reasonable! So I have to wait until net/vether-kmod gets fixed. Is there any other way to completely isolate jails and prevent arp who-has to appear on the external interface?
 
rootbert said:
just use a bridge without a physical interface. Add the epair interfaces of the jails to that bridge, add an ip address to the bridge interface acting as gateway and enable nat/forwarding of packets
Click to expand...
Assigning IP to bridge lead to connectivity stop.
 
Ifconfig with ARP who-has going outside:
Code: 
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
        ether b4:2e:99:6a:81:80
        inet xxx.251.xxx.212 netmask 0xffffffe0 broadcast 148.251.68.223
        inet xxx.251.xxx.217 netmask 0xffffffff broadcast 148.251.68.217
        inet xxx.251.xxx.247 netmask 0xffffffff broadcast 148.251.68.247
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0.100: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=1<RXCSUM>
        ether 00:a0:98:54:ce:b1
        inet 10.65.38.1 netmask 0xffffff00 broadcast 10.65.38.255
        groups: vlan
        vlan: 100 vlanpcp: 0 parent interface: em0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::b62e:99ff:fe6a:8180%tun0 prefixlen 64 scopeid 0x4
        inet 10.65.39.1 --> 10.65.39.2 netmask 0xffffff00
        groups: tun
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 994
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: em0
        ether 02:e3:d1:20:a0:01
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000000
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: CBSDSYSTEM0
        options=80000<LINKSTATE>
        ether 00:bd:68:a4:f7:01
        inet6 fe80::2bd:68ff:fea4:f701%tap1 prefixlen 64 tentative scopeid 0xd
        groups: tap
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: em0.100
        ether 02:e3:d1:20:a0:02
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em0.100 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 20000
        member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 35 priority 128 path cost 2000000
        member: epair26a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 34 priority 128 path cost 2000
...
        member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        groups: bridge
        nd6 options=1<PERFORMNUD>
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: mysql-eth0
        options=8<VLAN_MTU>
        ether 02:19:2c:37:ae:0a
        inet6 fe80::19:2cff:fe37:ae0a%epair1a prefixlen 64 tentative scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
...
epair26a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: whitesharks-eth0
        options=8<VLAN_MTU>
        ether 02:18:84:dd:45:0a
        inet6 fe80::18:84ff:fedd:450a%epair26a prefixlen 64 tentative scopeid 0x22
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: tron-node-nic0
        options=80000<LINKSTATE>
        ether 00:bd:de:1d:4d:02
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 64464
tap3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: tron-node-nic1
        options=80000<LINKSTATE>
        ether 00:bd:2b:1e:4d:03
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 64464

I've entered

Code: 
ifconfig em0.100 delete 10.65.38.1 ; ifconfig bridge2 10.65.38.1/24 ; ifconfig bridge2 deletem em0.100
ifconfig is now
Code: 
em0.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=1<RXCSUM>
        ether 00:a0:98:54:ce:b1
        groups: vlan
        vlan: 100 vlanpcp: 0 parent interface: em0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
...
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: em0.100
        ether 02:e3:d1:20:a0:02
        inet 10.65.38.1 netmask 0xffffff00 broadcast 10.65.38.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 35 priority 128 path cost 2000000
        member: epair26a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 34 priority 128 path cost 2000
        member: epair25a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 33 priority 128 path cost 2000
        member: epair24a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 32 priority 128 path cost 2000
...
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        groups: bridge
        nd6 options=1<PERFORMNUD>
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: mysql-eth0
        options=8<VLAN_MTU>
        ether 02:19:2c:37:ae:0a
        inet6 fe80::19:2cff:fe37:ae0a%epair1a prefixlen 64 tentative scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
...
epair26a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: whitesharks-eth0
        options=8<VLAN_MTU>
        ether 02:18:84:dd:45:0a
        inet6 fe80::18:84ff:fedd:450a%epair26a prefixlen 64 tentative scopeid 0x22
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: tron-node-nic0
        options=80000<LINKSTATE>
        ether 00:bd:de:1d:4d:02
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 64464
tap3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: tron-node-nic1
        options=80000<LINKSTATE>
        ether 00:bd:2b:1e:4d:03
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 64464

and nothing works
Code: 
www2:/root@[16:40] # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
eth0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:a0:98:06:f5:89
        hwaddr 02:72:b6:94:d8:0b
        inet 10.65.38.17 netmask 0xffffff00 broadcast 10.65.38.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
www2:/root@[16:40] # ping 10.65.38.1
PING 10.65.38.1 (10.65.38.1): 56 data bytes
^C
--- 10.65.38.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
 
ifconfig does not show it is up. furthermore, your em0 is in the same network as your bridge2 - that needs to be changed
 
tarkhil said:
Sorry? Ifconfig shows it UP. And em0 is in the bridge1, not 2.
Click to expand...
I mean your ifconfig in the post above did not show any bridge. And your em0 shows IP after "and nothing works" 10.65.38.17, before you give your bridge an "ifconfig bridge2 10.65.38.1/24" - they are both on the same network, that does not work. maybe I misread your information, it is quite confusing.
 
rootbert said:
I mean your ifconfig in the post above did not show any bridge. And your em0 shows IP after "and nothing works" 10.65.38.17, before you give your bridge an "ifconfig bridge2 10.65.38.1/24" - they are both on the same network, that does not work. maybe I misread your information, it is quite confusing.
Click to expand...
em0.100, not em0. em0 never has 10.65.38.1
 
You must log in or register to reply here.
Back
Top