Hey all,
While this could be in Virtualization it, at it's core, is a networking issue, so I wanted to post it here first. My post on Proxmox forums got little to no traction but here's all the info:
forum.proxmox.com
I do expect this to be a Proxmox issue and not a FreeBSD issue, but figured I'd post here on chance someone has ideas. I will start digging into it soon and will post results.
Some info copied here, see the Proxmox post for more.
Overview: I have a 10Gb SFP+ trunk to a server. When I have two vms set to the same vmbr using virtio, I lose DNS. I can ping, route, connect to services, just can't access udp/tcp 53 on any device internal or external.
Considerations:
1. If I switch VMS to independent vmbrs using different nics, everything works, but speed is slow (this is likely by design).
2. If I put both VMS on the same vmbr but change to Realtek RTL8139, everything works, but speed is slow (this is likely by design).
3. Clients connected to the same switch in a VLAN configured port work as intended.
Layout:
Proxmox VE 8.1.3 Host
- X10SDV-TP8F SuperMicro Motherboard
- 128GB RAM
- 4 Cores, 8 threads
- pve-firewall off
- vmbr0 - Linux Bridge - eno8 - D1500 SFP+ SoC
- vmbr0.1 - Linux Lan 192.168.10.10/24 | gateway 192.168.10.1
- vmbr1 - Linux Bridge - eno1 - RTL8211E 1GbE
VM1 - 101 - FreeBSD 14 Router [4 Cores/8GB Ram]
- vtnet0 - EXT.TER.NAL.IP/24 - virtio,bridge=vmbr0,tag=1
- PF firewall and Nat
- vtnet1 - 10.0.20.1/24 INTERNAL - virtio,bridge=vmbr0,tag=1003
VM2 - 103 - Ubuntu 22.04 Test Box [4 Cores/16GB Ram]
- ens18 - 10.0.20.10/24 - virtio,bridge=vmbr0,tag=1003
Notes:
1. I think the above screenshots (See Proxmox Thread) single out routing concerns as everything appears to work, so long as both VMs are not on the same vmbr with the same model.
2. It should be noted that, in working scenarios, iperf results between machines are accurate to the physical connection abilities, but to the internet they can be very slow around Fast Ethernet speeds (10Mbits) -- I include this in case it is relevant.
-- iperf between 10.0.20.10 and 10.0.20.1 is around 8Gb/s on vtnet1 via vmbr0 (SFP+)
-- iperf between 10.0.20.10 and 10.0.20.1 is around 750Mb/s on vtnet1 via vmbr1 (1GbE)
-- iperf between VM1 EXTERNAL and EXTERNAL IP is around 650Mb/s on vtnet0 via vmbr0 (500Mb/s Fiber Service)
-- iperf between 10.0.20.10 and EXTERNAL IP is around 9Mb/s on ens18 via vmbr0 or (Nat'd through vtnet1 to vtnet0)
While this could be in Virtualization it, at it's core, is a networking issue, so I wanted to post it here first. My post on Proxmox forums got little to no traction but here's all the info:
Multiple VMs on the same bridge no DNS
Hey guys -- this is a weird one, I'm going to do my best to describe it. Overview: I have a 10Gb SFP+ trunk to a server. When I have two vms set to the same vmbr using virtio, I lose DNS. I can ping, route, connect to services, just can't access udp/tcp 53 on any device internal or external...
forum.proxmox.com
I do expect this to be a Proxmox issue and not a FreeBSD issue, but figured I'd post here on chance someone has ideas. I will start digging into it soon and will post results.
Some info copied here, see the Proxmox post for more.
Overview: I have a 10Gb SFP+ trunk to a server. When I have two vms set to the same vmbr using virtio, I lose DNS. I can ping, route, connect to services, just can't access udp/tcp 53 on any device internal or external.
Considerations:
1. If I switch VMS to independent vmbrs using different nics, everything works, but speed is slow (this is likely by design).
2. If I put both VMS on the same vmbr but change to Realtek RTL8139, everything works, but speed is slow (this is likely by design).
3. Clients connected to the same switch in a VLAN configured port work as intended.
Layout:
Proxmox VE 8.1.3 Host
- X10SDV-TP8F SuperMicro Motherboard
- 128GB RAM
- 4 Cores, 8 threads
- pve-firewall off
- vmbr0 - Linux Bridge - eno8 - D1500 SFP+ SoC
- vmbr0.1 - Linux Lan 192.168.10.10/24 | gateway 192.168.10.1
- vmbr1 - Linux Bridge - eno1 - RTL8211E 1GbE
VM1 - 101 - FreeBSD 14 Router [4 Cores/8GB Ram]
- vtnet0 - EXT.TER.NAL.IP/24 - virtio,bridge=vmbr0,tag=1
- PF firewall and Nat
- vtnet1 - 10.0.20.1/24 INTERNAL - virtio,bridge=vmbr0,tag=1003
VM2 - 103 - Ubuntu 22.04 Test Box [4 Cores/16GB Ram]
- ens18 - 10.0.20.10/24 - virtio,bridge=vmbr0,tag=1003
Notes:
1. I think the above screenshots (See Proxmox Thread) single out routing concerns as everything appears to work, so long as both VMs are not on the same vmbr with the same model.
2. It should be noted that, in working scenarios, iperf results between machines are accurate to the physical connection abilities, but to the internet they can be very slow around Fast Ethernet speeds (10Mbits) -- I include this in case it is relevant.
-- iperf between 10.0.20.10 and 10.0.20.1 is around 8Gb/s on vtnet1 via vmbr0 (SFP+)
-- iperf between 10.0.20.10 and 10.0.20.1 is around 750Mb/s on vtnet1 via vmbr1 (1GbE)
-- iperf between VM1 EXTERNAL and EXTERNAL IP is around 650Mb/s on vtnet0 via vmbr0 (500Mb/s Fiber Service)
-- iperf between 10.0.20.10 and EXTERNAL IP is around 9Mb/s on ens18 via vmbr0 or (Nat'd through vtnet1 to vtnet0)