Solved No Internet Access In Guest VM

Hi all,

I have been trying to find a solution to my networking problem since hours, but i am running out of ideas and hope that someone here can help.

To create a VM, i followed the steps in the handbook. The guest system is running, but i am not happy yet with the networking.
After i assigned an ip address to the bridge, i was able to ping the host from the guest, SSH her and vice versa. If i try this for a node on the internet like ping 9.9.9.9, it is unreachable. Firewall is disabled.

My netstat looks not like as if it would restrict any ip packets from leaving the host to the internet (vpn-gr-72).

Destination Gateway Flags Netif Expire
0.0.0.0/1 link#5 US vpn-gr-72
default 192.168.1.1 UGS wlan0
10.2.0.2 link#1 UH lo0
unn-11-111-11-111. 192.168.1.1 UGHS wlan0
localhost link#1 UH lo0
128.0.0.0/1 link#5 US vpn-gr-72
169.254.0.0/16 link#7 U bridge0
169.254.154.15 link#1 UHS lo0
192.168.1.0/24 link#2 U wlan0
192.168.1.250 link#1 UHS lo0

In the attachment i put the scripts i used to initialize the network and start the guest vm.

Thanks for any advices!
 

Attachments

Hi there.
My netstat looks not like as if it would restrict any ip packets from leaving the host to the internet (vpn-gr-72).
Rich (BB code):
169.254.0.0/16 link#7 U bridge0
Isn't the 169.254. address block reserved for "link-local" addresses, "valid only for communications on a local link, i.e., within a subnetwork", from which network packets with a link-local source or destination address must not be forwarded beyond the local link?

Link-local addresses are not guaranteed to be unique beyond their network segment. Therefore, routers do not forward packets with link-local source or destination addresses.

RFC 3927: Dynamic Configuration of IPv4 Link-Local Addresses, 2.7. Link-Local Packets Are Not Forwarded
Code:
...
   An IPv4 packet whose source and/or destination address is in the
   169.254/16 prefix MUST NOT be sent to any router for forwarding, and
   any network device receiving such a packet MUST NOT forward it,
   regardless of the TTL in the IPv4 header.
...
 
@M
Hi there.

Isn't the 169.254. address block reserved for "link-local" addresses, "valid only for communications on a local link, i.e., within a subnetwork", from which network packets with a link-local source or destination address must not be forwarded beyond the local link?



RFC 3927: Dynamic Configuration of IPv4 Link-Local Addresses, 2.7. Link-Local Packets Are Not Forwarded
Code:
...
   An IPv4 packet whose source and/or destination address is in the
   169.254/16 prefix MUST NOT be sent to any router for forwarding, and
   any network device receiving such a packet MUST NOT forward it,
   regardless of the TTL in the IPv4 header.
...
Yes it is. But in my understanding the guest is connected to the tap interface and the bridge forwards all kinds of packets to wlan0 and vice versa.
 
You can't bridge with a wlan interface, the wlan interface essentially has to spoof source MAC addresses, and the driver won't allow it.
 
if I correctly understood your configuration and the use of wlan0, then you need to use the PF to create a NAT to access the Internet.
You can't bridge with a wlan interface, the wlan interface essentially has to spoof source MAC addresses, and the driver won't allow it.
because...
 
As a next step, i tried to prevent the guest assigning himself the link-local address (169.254.154.9).

So i was trying to follow this qemu wiki article here, which is basically 1) adding a bridge interface bridge0 2) assign that bridge an ip like 10.0.2.1 3) set up a DHCP server (using dnsmasq) listening on bridge0 4) start qemu.

So i configured the network on my host as follow:
ifconfig tap0 create
ifconfig tap0 up

sysctl net.link.tap.up_on_open=1
sysctl net.link.tap.user_open=1

ifconfig bridge0 create
ifconfig bridge0 addm tap0
ifconfig bridge0 10.0.2.1/32 up


and configured /usr/local/etc/dnsmasq.conf like this:
interface=bridge0
dhcp-range=10.0.2.2,10.0.2.99
dhcp-option=3,10.0.2.1

dnsmasq is then started as a service.

qemu is started like this:
/usr/local/bin/qemu-system-x86_64 -monitor none \
-cpu qemu64 \
-vga std \
-m 4096 \
-smp 4 \
-cdrom ../ISO/debian.iso \
-boot order=cd,menu=on \
-blockdev driver=file,aio=threads,node-name=imgleft,filename=../VM/left.img \
-blockdev driver=raw,node-name=drive0,file=imgleft \
-device virtio-blk-pci,drive=drive0,bootindex=1 \
-netdev tap,id=nd9,ifname=tap0,script=no,downscript=no,br=bridge0 \
-device e1000,netdev=nd9,mac=02:20:6c:65:66:74 \
-name \"left\"


On the guest, the ethernet interface is still assigned a link local address instead of 10.0.2.x. Maybe any hints here what i could have missed?
 
You mentioned a firewall in your opening post, is it still disabled?

I've set up a test qemu VM similar to your setup ( bridge, dnsmasq) but used FreeBSD as qemu guest instead. No problem here getting an IP lease from the DHCP server. The VM has internet traffic over pf NAT, wlan0.
 
Two suggestions. 1. Use a wired connection. Remove the wlan0 interface from the bridge and replace it with with your ethernet device (em0, igbe0, etc.) which should be connected to your local LAN. 2. You have a typo in left.sh on this line, should be id=nd0
-netdev tap,id=nd9,ifname=tap0,script=no,downscript=no,br=bridge0 \

Also, the bridge does not need an address. That's a linux thing.

If you are still stuck, let us know!
 
For a comprehensive look at this problem see here:

Jim Thompson explained this problem here:


ChatGPT explains it like this:
Yes—but with an important limitation: a FreeBSD host generally cannot transparently bridge a VM's tap0 interface onto a Wi-Fi interface when that Wi-Fi interface is operating as a normal client/station.

The topology you probably have in mind is:
Code:
                 Wi-Fi AP
                    |
                 wlan0
                    |
              +-----------+
              |  bridge0  |
              +-----------+
                    |
                   tap0
                    |
                    VM

With Ethernet this works perfectly:

ifconfig bridge0 create
ifconfig bridge0 addm em0 addm tap0 up
But replacing em0 with a station-mode wlan0 normally does not give you a functional transparent bridge.

Why Wi-Fi is different​

Ordinary Wi-Fi client mode uses the 802.11 three-address frame format. The access point expects Ethernet traffic coming from the wireless station to use the MAC address belonging to that associated station.

Your FreeBSD host might have:

wlan0 MAC: aa:bb:cc:dd:ee:ff
while the VM has:

vtnet0 MAC: 02:12:34:56:78:9a
If tap0 is simply bridged to wlan0, frames originating from the VM carry the VM's MAC address. A normal Wi-Fi association does not provide a transparent Ethernet bridge for arbitrary MAC addresses behind the station.

So although FreeBSD may allow something resembling:

ifconfig bridge0 create
ifconfig bridge0 addm wlan0
ifconfig bridge0 addm tap0
ifconfig bridge0 up

you'll usually find that things such as DHCP, ARP, or return traffic don't work correctly.

The normal solution: route/NAT the VM​

For a wireless-only host, I'd use:

Code:
Internet / LAN
                   |
                 Wi-Fi
                   |
                 wlan0
                   |
              FreeBSD host
             routing + NAT
                   |
                bridge0
                   |
                 tap0
                   |
                   VM

For example:

VM: 192.168.200.2/24
Host: 192.168.200.1/24
Gateway: 192.168.200.1
Create the VM-side bridge:

ifconfig bridge0 create
ifconfig bridge0 inet 192.168.200.1/24
ifconfig tap0 create
ifconfig bridge0 addm tap0
ifconfig bridge0 up
ifconfig tap0 up
Enable forwarding:

gateway_enable="YES"
in /etc/rc.conf.

Then NAT 192.168.200.0/24 through wlan0 using whichever firewall you're already using—pf, ipfw, etc.


There is one exception​

A true Wi-Fi bridge can work when the Wi-Fi infrastructure supports 4-address/WDS mode:

Code:
AP <---- 802.11 4-address/WDS ----> FreeBSD
                                      |
                                   bridge0
                                      |
                                    tap0
                                      |
                                     VM
That lets the wireless link carry MAC addresses belonging to machines behind the wireless station.

But support has to exist in both the AP and the FreeBSD wireless driver, and it isn't something I'd assume will work with arbitrary consumer Wi-Fi hardware.

One other trick you may encounter​

Linux has things like macvtap arrangements and various proxy-ARP / MAC-NAT approaches. Those aren't the same thing as a true layer-2 bridge. You can accomplish similar effects under FreeBSD with routing, NAT, proxy ARP, etc., but at that point routing the VM network is simpler and more predictable.

Code:
DHCP address
                               |
Wi-Fi AP ~~~~~~~~~~~~~~~~ wlan0
                               |
                         FreeBSD host
                         192.168.200.1
                               |
                            bridge0
                           /       \
                        tap0       tap1
                          |          |
                        VM 1       VM 2
                   192.168.200.2  .3

Then NAT or route 192.168.200.0/24 through wlan0.



So the short version is:​

​

Ethernet NIC + bridge0 + tap0: yes, true bridge.​

​

Wi-Fi STA + bridge0 + tap0: generally no, not as a transparent L2 bridge.​

​

Wi-Fi STA + routed/NAT VM network: yes, and that's the setup I'd recommend.​

 
I finally managed to assign an ip address (not a link-local one) to the guest vm! I also did the routing by NAT as suggested and the guest now has access to the internet. Instead of dnsmasq, I used isc-dhcpd to configure my DHCP server. Also the bridge0 seems to need an ip address assigned, isc-dhcpd would otherwise complain:
Bash:
> service isc-dhcpd start
Starting dhcpd.
Internet Systems Consortium DHCP Server 4.4.3-P1
..
No subnet declaration for bridge0 (no IPv4 addresses).

I would like to share my configuration:

Network interfaces:
Code:
#!/bin/sh
ifconfig tap0 create
ifconfig tap0 up

sysctl net.link.tap.up_on_open=1
sysctl net.link.tap.user_open=1
sysctl net.inet.ip.forwarding=1

ifconfig bridge0 create
ifconfig bridge0 addm tap0
ifconfig bridge0 10.0.2.1/27 up

DHCP (isc-dhcpd) resp. /usr/local/etc/dhcpd.conf
Code:
option domain-name-servers 9.9.9.9;

default-lease-time 600;
max-lease-time 7200;

subnet 10.0.2.0 netmask 255.255.255.224 {
  range 10.0.2.10 10.0.2.20;
  option routers 10.0.2.1;
}

Firewall configuration resp. /etc/pf.conf
Code:
nat on vpn-gr-72 from 10.0.2.0/24 to any -> vpn-gr-72
block in all
pass out all keep state
pass from 10.0.2.0/24 to any keep state

/etc/rc.conf
Code:
pf_enable="yes"
gateway_enable="yes"
dhcpd_enable="YES"
dhcpd_ifaces="bridge0"

VM start script
Code:
#!/bin/sh
/usr/local/bin/qemu-system-x86_64  -monitor none \
  -cpu qemu64 \
  -vga std \
  -m 4096 \
  -smp 4   \
  -cdrom ../ISO/debian.iso \
  -boot order=cd,menu=on \
  -blockdev driver=file,aio=threads,node-name=imgleft,filename=../VM/debian.img \
  -blockdev driver=raw,node-name=drive0,file=imgleft \
  -device virtio-blk-pci,drive=drive0,bootindex=1  \
  -netdev tap,id=nd0,ifname=tap0,script=no,downscript=no,br=bridge0 \
  -device e1000,netdev=nd0,mac=02:20:6c:65:66:74 \
  -name \"debian\"

Thanks a lot for your support :)!
 
Back
Top