qemu 12.2-RELEASE and QEMU w/ tap networking

[ceteris]

Hello all,
I am trying to use QEMU to emulate a x86_64 machine to run OpenBSD 6.9 inside FreeBSD 12.2-RELEASE with two tap interfaces. Hardware is an Intel Xeon E5450, 8GB DDR2, SATA SSD. This CPU doesn't have EPT support so I can't use bhyve and I don't want to use VirtualBox. I would like, if at all possible, use QEMU.

I added the following to /boot/loader.conf:

if_tap_load="YES"
if_bridge_load="YES"

to /etc/sysctl.conf:

net.link.tap.user_open=1
net.link.tap.up_on_open=1

to /etc/devfs.conf:

own     tap0    root:wheel
own     tap1    root:wheel

to /etc/rc.conf:

cloned_interfaces="bridge0"
autobridge_interfaces="bridge0"
autobridge_bridge0="tap* em0"

It's my desire to have both tap0 and tap1 connected to the QEMU-emulated OpenBSD instance in order to set it up as a gateway, firewall and so on. tap0 should connect to bridge0 (LAN) and tap1 should connect to bridge1 (WAN).

I first tried installing QEMU 5.x from ports but later settled for the generic pkg install to save on time. I also pkg install(ed) qemu-utils.

Playing around with QEMU demonstrated that qemu-system-x86_64 was fully capable of setting the desired network configuration when suitable scripts were present under /usr/local/etc/qemu-ifup and /usr/local/etc/qemu-ifdown. I copied the scripts from here:

Automagic bridged networking with QEMU (tap)

As you'll no doubt notice, I relied heavily on that guide from 2009.

#!/bin/sh
#
#/usr/local/etc/qemu-ifup

IFNAME=em0

for BRIDGE in $(ifconfig -a | grep '^bridge' | cut -d: -f1)
do
        if [ -n "$(ifconfig "$BRIDGE" | grep -w "member: $IFNAME")" ]
        then
                echo "${##*/}: Adding $1 as a member of $BRIDGE"
                /sbin/ifconfig "$BRIDGE" addm "$1" up
                exit
        fi
done

BRIDGE="$(/sbin/ifconfig bridge create)"
/sbin/ifconfig "$BRIDGE" addm "$IFNAME" addm "$1" up
echo "${0##*/}: Created $BRIDGE and added $1 as a member"

#!/bin/sh
#
# /usr/local/etc/qemu-ifdown
#

for BRIDGE in $(ifconfig -a | grep '^bridge' | cut -d: -f1)
do
        if [ -n "$(ifconfig "$BRIDGE" | grep -w "member: $1")" ]
        then
                if [ "$(ifconfig "$BRIDGE" | grep -c -w "member:")" -le 2 ]
                then
                        echo "${0##*/}: Destroying $BRIDGE"
                        /sbin/ifconfig "$BRIDGE" destroy
                fi
                echo "${0##*/}: Destroying $1"
                /sbin/ifconfig "$1" destroy
        fi
done

The closest I've come to anything resembling success so far is by running the machine with the config files as noted above, manually removing the bridges and tap interfaces after boot (redundant, I know) and invoking QEMU with:

qemu-system-x86_64 \
-m 1024 \
-cdrom install69.iso \
-drive if=virtio,format=raw,file=obsd-x64.raw \
-netdev tap,id=nd0,ifname=tap0 -device virtio-net,netdev=nd0 \
-netdev tap,id=nd1,ifname=tap1 -device virtio-net,netdev=nd1 \
-nographic \
-serial tcp::4450,server,telnet,wait

To view the QEMU machine I run:

telnet localhost 4450

I then input "set tty com0" at the OpenBSD boot terminal to enable the serial console and everything seems great, I can install OpenBSD, list network interfaces with ifconfig and set IPs for example, but I can't reach the host network, the internet or even ping outside the emulated machine. Running a jail tied to em0 (the interface attached to bridge0, beside tap0) and trying to ping from either side (jail -> obsd or the other way around), with IPs set to the same subnet inside the jail and OpenBSD, produces no results.

What am I missing?

Thank you very much.

EDIT:

Note - if you are using the (default) SLiRP user networking, then ping (ICMP) will not work, though TCP and UDP will. Don't try to use ping to test your QEMU network configuration!

I don't think I am using SLiRP though.

 

[ceteris]

I believe I've gotten the FreeBSD host side of things sorted out, since I'm perfectly able to use tap(s) and bridge(s) with the aid of ifconfig and the two sysctl tweaks I mentioned early on but I'm still unable to convince QEMU (or the OpenBSD guest) to connect to it

I'm now trying to find out if there's something in the ports version of QEMU that, a config option I'd have to tick, for QEMU to work right.

There once was a kernel module called kqemu that one could load to gain kernel-level support for QEMU, but it seems to have been left broken and forgotten with the advent of bhyve.

I got a lot of help from this:

networking qemu virtual bsd systems

EDIT: There's nothing in the ports config options to indicate the default configuration might lack tap network support. I've looked under version 5.0 and 6.1 (qemu-devel), release is r366954 GENERIC.

I'll save my work with bectl create and likely sacrifice a year of my life, waiting on it to compile, to give 6.1 a try. pkg also offers 5.1.

 

[ceteris]

Well, this has been nothing but PAINFUL, but I've finally cracked it: I've managed to ping out of the QEMU OpenBSD VM and, not unlike most annoying issues, this too turned out to be trivial in the end.

Here's a short HOW-TO:
I) Install FreeBSD-12.2-STABLE-amd64
II)

pkg install qemu-devel

III)

ifconfig tap0 create; ifconfig bridge0 create; ifconfig bridge0 addm tap0; ifconfig tap0 up, ifconfig bridge0
up;

IV)

ifconfig bridge0 addm $NIC

$NIC = a NON-wireless network interface you wish to add to the bridge, along with tap0 (it will also have to be "up").

V)

pkg install wget; wget https://cdn.openbsd.org/pub/OpenBSD/6.9/amd64/install69.img

VI) Create .raw image:

qemu-img create -f raw obsd-x64.raw 10G

VII)

qemu-system-x86_64 \
-m 1024 \
-cdrom install69.iso \
-drive if=virtio,format=raw,file=obsd-x64.raw \
-netdev tap,id=nd0,ifname=tap0,script=no -device virtio-net,netdev=nd0  \
-nographic \
-serial tcp::4450,server,telnet,wait

Careful observers will notice I added "script=no", which turned out to be the key part. You can either fully configure tap interfaces and bridges on the host, or you can allow them to be set up by QEMU, which, by default, calls qemu-ifup/down scripts, located under /usr/local/etc/. Not using the script at all seems to work.

VIII) run

telnet localhost 4450

in a separate terminal window to connect to the QEMU VM

IX) at the OpenBSD boot prompt ("boot>"), enter

set tty com0

to divert terminal output to the QEMU virtual serial adapter (the telnet server)

X) run OpenBSD LiveCD in Shell mode and assign a complementary IP address to the OpenBSD vio0 interface with

ifconfig vio0 10.11.12.13/24; ifconfig vio0 up;

and ping out to the host

ping $NICIP

XI) install OpenBSD inside QEMU, set the output to default to serial.
(I've jet to find out why DHCP doesn't seem to work, might be a pf issue.)

XII) you can now run the QEMU VM without using the "-cdrom install69.iso" line.

*This only applies to running the QEMU VM with the root user. Other users might require futher configuration.
**If the filesystem you're using on the host FreeBSD machine is ZFS, stick to .raw QEMU disk images.
***The above QEMU VM configuration only employs a single network interface. Feel free to add more in the same manner.

 

[ceteris]

Testing has shown that I can only pass ICMP traffic through bridge0, to which everything in this setup connects. I've also managed to connect to the QEMU VM via ssh/scp, so TCP seems to work as well, but DHCP, DNS traffica are a no-go and they both, obviously, rely on UDP.

 

[ceteris]

Scrap that!

After going through every possible manual and trying the most desperate things, I finally upended my network configuration, switched some things around, and it now seems the culprit responsible for my limited network connectivity was an OpenSUSE that served as (one of the) DHCP server(s).

The procedure I described in post #3 is correct and should work, but please notice that the QEMU invocation I gave will provide your VM with only a single core CPU.

 

[ceteris]

As it turns out, It's not so easy to get QEMU to emulate to a multi-core CPU, at least in my case, which would, obviously, be:

root@fb122s:~ # pkg info qemu-devel | head
qemu-devel-6.1.0.20210928
Name           : qemu-devel
Version        : 6.1.0.20210928

running on

root@fb122s:~ # uname -a
FreeBSD fb122s 12.2-STABLE FreeBSD 12.2-STABLE r370528 GENERIC  amd64

Here are two working single core QEMU configurations for anyone aiming to emulate (no pun intended) my efforts:

- Alpha -

qemu-system-x86_64 \
-M q35 \
-smp cpus=4,maxcpus=4,sockets=1,dies=2,cores=2,threads=1 \
-m 2048 \
-cdrom install69.iso \
-drive if=virtio,format=raw,file=qob69.raw \
-netdev tap,id=nd0,ifname=tap0,script=no \
-device virtio-net,netdev=nd0 \
-nographic \
-serial tcp::4450,server,telnet,wait

What the above invocation means, described with casual computer parlance is:
"Run me an emulated machine consisting of:"

• a 'standard PC (Q35 + ICH9, 2009)',
• a 'QEMU TCG CPU version 2.5+' w/ 2 cores (OpenBSD 6.9 shows a single core only),
• 2 GBs of RAM,
• a QEMU-default-configured CD-ROM device, inserted with OpenBSD's .iso image,
• a "virtio" disk drive, employing the image file "qob69.raw" (created with 'qemu-img create' beforehand),
• a "virtio" network interface that's attached to the host's "tap0" interface,
• the last two lines simply tell the OpenBSD guest not to bother outputting video, but to rather direct console output to the serial device, which is also virtual, provided QEMU. To make it accessible, QEMU then provides serial output in the form of a telnet server bound to the host's loopback address and waits for a client to connect before starting the guest.

- Beta -

qemu-system-x86_64 \
-M q35 \
-cpu max \
-smp cpus=4,maxcpus=4,sockets=1,dies=2,cores=2,threads=1 \
-m 2048 \
-cdrom install69.iso \
-drive if=virtio,format=raw,file=qob69.raw \
-netdev tap,id=nd0,ifname=tap0,script=no \
-device e1000,netdev=nd0 \
-netdev tap,id=nd1,ifname=tap1,script=no \
-device e1000,netdev=nd1\
-nographic \
-serial tcp::4451,server,telnet,wait

Changes:
By issuing "-cpu max" I've instructed QEMU to use everyhing the physical CPU has to offer in terms of instructions etc.; I've also added another NIC, but this time it will appear to the guest machine as an Intel gigabit adapter. I've also upped the telnet port number by one, allowing you to run this instance of QEMU alongside the one described previously.

Despite reading through a bunch of documentation and trying just about every CPU configuration QEMU has to offer, I can't get it to provide my OpenBSD guest with more than a single CPU (thus far).

 

[ceteris]

There is a simple solution to the single core problem, kind of:

QEMU will deduplicate the RAM used by instances run in parallel, but it seems that doesn't happen on the fly. For example: htop will report use 4G+ on the host when I run two identical OpenBSD guests (assigned 2G RAM each), but, as time passes, that number will drop significantly, despite the lack of a memory balloon device. This behaviour seems to come by default, but there's no harm in noting it explicitly:

qemu-system-x86_64 qemu-system-x86_64 -M q35,mem-merge=on -cpu max
-smp 4,sockets=1,dies=2,cores=2,threads=1 -m 2048 -drive if=virtio,format=raw,file=qob69.raw -netdev tap,id=nd1,ifname=tap1,script=no -device virtio-net,netdev=nd1 -nographic -serial tcp::4451,server,telnet,wait

@-M q35,mem-merge=on

 

[ceteris]

I've found a way to get OpenBSD to recognize more CPUs. The solution was, as I've foreshadowed in the first sentence, to reinstall the OpenBSD QEMU guest. I had gone through at least 50 different emulated CPU configurations and was almost ready to give up. It turns out that OpenBSD chooses a different kernel when it detects It's running on a multi-processor system (bsd.mp vs. bsd), so switching an installation that had been done on a single-processor to a multi-processor machine, will detect 4 CPUs (hw.ncpufond=4), but will only employ a single processor (hw.ncpu=1 vs. hw.ncpu=4). OpenBSD also fetches and installs the appropriate CPU microcode support/update from the internet upon first bootup, so It's beneficial to conduct the install in an environment that has internet access (to have the process complete automatically, at least).

qobAB# sysctl -a | grep "hw"
hw.machine=amd64
hw.model=QEMU TCG CPU version 2.5+
hw.ncpu=4
hw.byteorder=1234
hw.pagesize=4096
hw.disknames=sd0:169e9d929fd8ae43,cd0:
hw.diskcount=2
hw.cpuspeed=3506
hw.vendor=QEMU
hw.product=Standard PC (Q35 + ICH9, 2009)
hw.version=pc-q35-6.2
hw.physmem=2130558976
hw.usermem=2130542592
hw.ncpufound=4
hw.allowpowerdown=1
hw.smt=0
hw.ncpuonline=4

And the QEMU invocation:

qemu-system-x86_64 -M q35,mem-merge=on -m 2048 -cpu max -smp 4,cpus=4,maxcpus=4,cores=4 -drive if=virtio,format=raw,file=qob69_4.raw -netdev tap,id=nd0,ifname=tap0,script=no -device virtio-net,netdev=nd0 -nographic -serial tcp::4450,server,telnet,wait