Need help fixing the bridge connection between Linux Devuan 5 and FreeBSD 13.2 (virtualized with qemu + kvm + libvirt)

Status
Not open for further replies.
on Linux :

Code: 
# iptables -t nat -A POSTROUTING -o mlan0 -j MASQUERADE 
# ip tuntap add tap0 mode tap 
# ip link set dev tap0 up 
# ifconfig tap0 192.168.99.1/24 
# echo 1 > /proc/sys/net/ipv4/ip_forward


# iptables -L -v -t nat 

# Warning: iptables-legacy tables present, use iptables-legacy to see them 
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) 
 pkts bytes target     prot opt in     out     source               destination          

Chain INPUT (policy ACCEPT 0 packets, 0 bytes) 
 pkts bytes target     prot opt in     out     source               destination          

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) 
 pkts bytes target     prot opt in     out     source               destination          

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) 
 pkts bytes target     prot opt in     out     source               destination          
   13  1203 MASQUERADE  all  --  any    mlan0   anywhere             anywhere

# ping 8.8.8.8 

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 
From 192.168.99.1 icmp_seq=1 Destination Host Unreachable 
From 192.168.99.1 icmp_seq=2 Destination Host Unreachable 
From 192.168.99.1 icmp_seq=3 Destination Host Unreachable 
^C 
--- 8.8.8.8 ping statistics --- 
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4144ms 
pipe 4 

# ping google.com 
ping: google.com: Nome o servizio sconosciuto
 
let the ping run and while it runs
look at
iptables-legacy -L -v
and
iptables-legacy -L -v -t nat
see if the chains get any traffic
i set a simple gre tunnel between a bsd box and a linux vm i have here and i can route some ips via gre and nat on linux
the single rule i added is
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
 
covacat said:
let the ping run and while it runs
look at
iptables-legacy -L -v
and
iptables-legacy -L -v -t nat
see if the chains get any traffic
i set a simple gre tunnel between a bsd box and a linux vm i have here and i can route some ips via gre and nat on linux
the single rule i added is
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
Click to expand...

your experiment is not well thought out. You should try with all the wi-fi adapters that you have.
 
anyway,the ping is going :

# ping -I 192.168.99.1 8.8.8.8

PING 8.8.8.8 (8.8.8.8) from 192.168.99.1 : 56(84) bytes of data.
From 192.168.99.1 icmp_seq=1 Destination Host Unreachable
From 192.168.99.1 icmp_seq=2 Destination Host Unreachable
From 192.168.99.1 icmp_seq=3 Destination Host Unreachable
From 192.168.99.1 icmp_seq=5 Destination Host Unreachable

and :

# iptables-legacy -L -v

Chain INPUT (policy ACCEPT 22833 packets, 1358K bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 46 packets, 3468 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 15668 packets, 45M bytes)
pkts bytes target prot opt in out source destination


# iptables-legacy -L -v -t nat

Chain PREROUTING (policy ACCEPT 19 packets, 1524 bytes)
pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 5 packets, 620 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 23 packets, 2859 bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 22 packets, 2222 bytes)
pkts bytes target prot opt in out source destination
 
A really stupid question, just for clarification and to sum it up:
  • the Linux host is your KVM hypervisor which is running
  • your virtual FreeBSD system?
If so, the guide you followed is quite cumbersome and complicated. There should not be any need to fiddle around with manual setup of tap interfaces, let alone Netfilter rules. libvirt will setup everything you need.
 
Linux host = Devuan 5
os guest = FreeBSD 13.2

Unfortunately no,my friend. I tried that libvirt does the dirty work for me,but I've got the same error. But when I tried I haven't added the netfilter rule. Write it here and I will try. Do you mean to issue only this command :

iptables -t nat -A POSTROUTING -o mlan0 -j MASQUERADE

?
 
I caught the precise moment when it stops pinging the outside :

on Linux :

Code: 
# iptables -t nat -A POSTROUTING -o mlan0 -j MASQUERADE 
# ip tuntap add tap0 mode tap 
# ip link set dev tap0 up 
# ifconfig tap0 192.168.99.1/24 
# echo 1 > /proc/sys/net/ipv4/ip_forward

# ping google.com

PING google.com (142.251.209.14) 56(84) bytes of data.
64 bytes from mil04s50-in-f14.1e100.net (142.251.209.14): icmp_seq=1 ttl=115 time=17.7 ms
64 bytes from mil04s50-in-f14.1e100.net (142.251.209.14): icmp_seq=2 ttl=115 time=47.7 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 17.656/32.697/47.738/15.041 ms

Now,I start the FreeBSD vm and when the its boot messages come to this point,more or less :


vtnet0: link state changed to UP
Starting Network: lo0 vtnet0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 52:54:00:12:34:55
inet 192.168.99.2 netmask 0xffffff00 broadcast 192.168.99.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Starting devd.
eval: cannot open /dev/ttyv0: No such file or directory
eval: cannot open /dev/ttyv0: No such file or directory
Configuring vt: keymapeval: cannot open /dev/ttyv0: No such file or directory
.
Starting ums0 moused.
eval: cannot open /dev/ttyv*: No such file or directory
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add net default: gateway 192.168.99.1
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Starting local_unbound.
Waiting for nameserver to start... good


it stops pinging google.com

# ping google.com (from Linux)
ping: google.com: Name or service unknown.

So,it depends about how I have configured the networking on FreeBSD.
 
Well, I've been running KVM, libvirtd with bridged networking for ages. It's the simplest networking setup you can have.
Considering that you explicitly want bridged networking:
  • Just set up a bridge on the host and add your "normal" interface to the brige. In your particular case with the wifi interface using hostapd.
  • Move the IP address from the "normal" interface to the bridge interface.
  • Setup bridged networking for your BSD VM and configure the VM to use the bridge interface:
    Code: 
    <interface type='bridge'>
<mac address='52:54:00:f1:8c:0b'/>
<source bridge='br0'/>
  <target dev='vnet0'/>
  <model type='virtio'/>
  <alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    <source bridge='br0'/> is the relevant part.
  • Test result by assigning an IP to the VM which is in the same subnet as br0 on the hypervisor.
If you want a different subnet for your VM, you need to set up routing and MASQUERADING / SNAT afterwards. But in this case, you should consider a different network topology for libvirt (that is, a NATted network).

Keep in mind that manual on-the-fly creation of network interfaces might interfere with network management components such a Network Manager which might shut down your manually set up interfaces at any time. I'd try the entire setup with a "normal" copper interface instead the wifi interface (and try adding this to the bridge afterwards), just to make sure the problem is not related to wifi.
 
When it does not work :

# ip route

0.0.0.0 dev tap0 scope link
default dev tap0 scope link
default via 192.168.1.1 dev mlan0 proto dhcp src 192.168.1.7 metric 600
169.254.0.0/16 dev tap0 proto kernel scope link src 169.254.167.92
192.168.1.0/24 dev mlan0 proto kernel scope link src 192.168.1.7 metric 600
192.168.1.1 dev mlan0 scope link
192.168.99.0/24 dev tap0 proto kernel scope link src 192.168.99.1

# iptables-nft -L -v

# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
 
Now I closed the vm and I can ping google.com :

# ip route

0.0.0.0 dev tap0 scope link linkdown
default via 192.168.1.1 dev mlan0
default via 192.168.1.1 dev mlan0 proto dhcp src 192.168.1.7 metric 600
192.168.1.0/24 dev mlan0 proto kernel scope link src 192.168.1.7 metric 600
192.168.1.1 dev mlan0 scope link
192.168.99.0/24 dev tap0 proto kernel scope link src 192.168.99.1 linkdown
 
This is the difference between the two situations and where is the error is very clear :

169.254.0.0/16 dev tap0 proto kernel scope link src 169.254.167.92

Despite that,it says that 192.168.99.1 has been assigned ?


tap0: flags=-28669<UP,BROADCAST,MULTICAST,DYNAMIC> mtu 1500
inet 192.168.99.1 netmask 255.255.255.0 broadcast 192.168.99.255
ether 1e:3e:92:e1:89:df txqueuelen 1000 (Ethernet)
RX packets 15 bytes 1006 (1006.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 135 bytes 49767 (48.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
 
This how-to you followed is no good. Really ;-)
You just don't fiddle around manually with cumbersome qemu command or tap interfaces. You dont't have to. Everything is done using virsh.

Basically, you don't even need TAP for the bridge interface your VMs are using. Here is mine:
Code: 
root@nb19106042:~# ethtool -i br0
driver: bridge
version: 2.3
firmware-version: N/A
expansion-rom-version:
bus-info: N/A
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

TAP would show up as bus-info: tap

In any case, your problem is obviously related to your hypervisor setup (running Linux), not FreeBSD. Expect the same problem to show up with any other guest OS.
 
Gyros Komplett said:
Well, I've been running KVM, libvirtd with bridged networking for ages. It's the simplest networking setup you can have.
Considering that you explicitly want bridged networking:
  • Just set up a bridge on the host and add your "normal" interface to the brige. In your particular case with the wifi interface using hostapd.
  • Move the IP address from the "normal" interface to the bridge interface.
  • Setup bridged networking for your BSD VM and configure the VM to use the bridge interface:
    Code: 
    <interface type='bridge'>
<mac address='52:54:00:f1:8c:0b'/>
<source bridge='br0'/>
  <target dev='vnet0'/>
  <model type='virtio'/>
  <alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    <source bridge='br0'/> is the relevant part.
  • Test result by assigning an IP to the VM which is in the same subnet as br0 on the hypervisor.
If you want a different subnet for your VM, you need to set up routing and MASQUERADING / SNAT afterwards. But in this case, you should consider a different network topology for libvirt (that is, a NATted network).

Keep in mind that manual on-the-fly creation of network interfaces might interfere with network management components such a Network Manager which might shut down your manually set up interfaces at any time. I'd try the entire setup with a "normal" copper interface instead the wifi interface (and try adding this to the bridge afterwards), just to make sure the problem is not related to wifi.
Click to expand...

very thanks,but your explanations is not complete. There are some points that aren't clear for me. Unfortunately you have been too generic. And I'm not able to cover by myself the parts that you haven't explained,in particular these points :

  • Just set up a bridge on the host and add your "normal" interface to the brige. In your particular case with the wifi interface using hostapd.

  • Move the IP address from the "normal" interface to the bridge interface.

I think I'm not able to work on this.
 
Gyros Komplett said:
This how-to you followed is no good. Really ;-)
You just don't fiddle around manually with cumbersome qemu command or tap interfaces. You dont't have to. Everything is done using virsh.

Basically, you don't even need TAP for the bridge interface your VMs are using. Here is mine:
Code: 
root@nb19106042:~# ethtool -i br0
driver: bridge
version: 2.3
firmware-version: N/A
expansion-rom-version:
bus-info: N/A
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

TAP would show up as bus-info: tap

In any case, your problem is obviously related to your hypervisor setup (running Linux), not FreeBSD. Expect the same problem to show up with any other guest OS.
Click to expand...

No,I'm not sure. Everything works good before to start FreeBSD. Probably when FreeBSD starts it adds some network configuration settings that freezes the config. settings of the host os.
 
I've asked some help on the devuan ML and he asked me :

Do you have a default route set? Since 192.168.99.1 and 192.168.1.7 are both IP addresses on your local machine, they are pingable, but where does the traffic go when it leaves your local machine? What's your next hop on the network? I assume it's your local gateway. Can you ping that? You're missing a lot of information here to be able to troubleshoot this problem.
 
your default route is 192.168.1.1
what is the output of ip route when ping is not working (when vm is active)
 
When it does not work :

# ip route

0.0.0.0 dev tap0 scope link
default dev tap0 scope link
default via 192.168.1.1 dev mlan0 proto dhcp src 192.168.1.7 metric 600
169.254.0.0/16 dev tap0 proto kernel scope link src 169.254.167.92
192.168.1.0/24 dev mlan0 proto kernel scope link src 192.168.1.7 metric 600
192.168.1.1 dev mlan0 scope link
192.168.99.0/24 dev tap0 proto kernel scope link src 192.168.99.1

when it works :

# ip route

0.0.0.0 dev tap0 scope link linkdown
default via 192.168.1.1 dev mlan0
default via 192.168.1.1 dev mlan0 proto dhcp src 192.168.1.7 metric 600
192.168.1.0/24 dev mlan0 proto kernel scope link src 192.168.1.7 metric 600
192.168.1.1 dev mlan0 scope link
192.168.99.0/24 dev tap0 proto kernel scope link src 192.168.99.1 linkdown
 
when not working try
ip route del default dev tap0 scope link
ip route del 0.0.0.0 dev tap0 scope link
see if it fixes it
 
# ip route del default dev tap0 scope link
# ip route del 0.0.0.0 dev tap0 scope link
# ping google.com

ping: google.com: Name or service unknown

# ip route

default dev tap0 scope link
169.254.0.0/16 dev tap0 proto kernel scope link src 169.254.236.170
192.168.1.0/24 dev mlan0 proto kernel scope link src 192.168.1.7 metric 600
192.168.1.1 dev mlan0 scope link
192.168.99.0/24 dev tap0 proto kernel scope link src 192.168.99.1
 
Status
Not open for further replies.
Back
Top