bhyve Switch does not survive reboot and other network changes

freezr

freezr

I changed the title since the main issue has changed in me trying to keep the switch alive.
Click to expand...

Hi guys,

I am hitting my head with virtualization.

I followed the guide to use vmrun.sh to create a vm, but it looks like this method is for creating headless vm.

Now I am trying with Virtualbox, but it doesn't have any specific FreeBSD acceleration... Perhaps Libvirt?
I read that I need to built from ports to activate bhyve... What else? Qemu?

I need a vm to test out random software that might run on FreeBSD but I don't want to touch my real system, thanks... 🙏
 
Last edited:
freezr said:
Hi guys,

I am hitting my head with virtualization.

I followed the guide to use vmrun.sh to create a vm, but it looks like this method is for creating headless vm.

Now I am trying with Virtualbox, but it doesn't have any specific FreeBSD acceleration... Perhaps Libvirt?
I read that I need to built from ports to activate bhyve... What else? Qemu?

I need a vm to test out random software that might run on FreeBSD but I don't want to touch my real system, thanks... 🙏
Click to expand...
bhyve is included with the base system, it has nothing to do with ports.
I would recommend sysutils/vm-bhyve to easily manage your bhyve VMs.
 
Generally I find a headless setup to work best for virtualization. Then if I want graphics:
  • If the UI I am using is decent, X11/ssh forward.
  • If the UI is bloated / crap, I use Xvnc (tigervnc).
If you want accelerated 3D / gpu passthrough then VirtualBox is easiest if using Win/Lin guests. Qemu doesn't have a Bhyve backend so is software CPU accel only. Bhyve I believe allows you to pass through the entire GPU if you have multiple on the host.

Perhaps just LLVMpipe is good enough for you for a 3D software renderer?
 
freezr said:
Perhaps Libvirt?
Click to expand...
No that won't get you a GUI.
Problem is Bhyve is geared to command line.

This as a good place to start but I don't prefer that method. Jump ahead in the manual to 24.7.4 and checkout UEFI guest. That is probably the most practical example.
/usr/share/examples/bhyve/vmrun.sh -h

Libvirt has a GUI program for setting up VM's but not running one.

When I did GUI in VM I had separate video cards for each VM. Along with USB card for each VM.
And you need VT-d//IOMMU instructions on your CPU for PCI Passthrough.

So its pretty complex going hardware route. TigerVNC route might be better for you.
 
This is one little tweak if you are interested in a single VM GUI Bhyve experience.

If you run a NVidia/AMD Graphics card on your host machine you may have a spare Intel GPU that you are not using.

There is driver to send that through to a Bhyve VM.

⚙ D26209 GVT-d support for bhyve

reviews.freebsd.org reviews.freebsd.org

Mark McBride (@markmcb@mas.to)

Turns out #FreeBSD is an excellent #virtualization host for iGPU passthrough. I had a #Linux #Incus (#lxc) container running Plex on an Ubuntu 24.04 host. Probably because 24.04 is pre-release, the Intel drivers started failing and I could no longer use the GPU. I tried reserving my GPU to pass...
mas.to mas.to
 
Is the VNC server provided by bhyve or do I have to install a vnc server into the VM?

I don't need GPU passthrough, I only need to test some software in a safe way.
 
Bhyve does provide a VNC server but it is tied to their virtual GPU driver and scrapes the framebuffer from it. This is way more layers.

So, as long as you have CPU accel (and virtio network drivers), I believe tigervnc's Xvnc installed on the guest will still be faster. Its also much smaller than a full fat Xlibre/Xorg (though you likely want one of them too for the libs / utilities).
 
kpedersen said:
Bhyve does provide a VNC server but it is tied to their virtual GPU driver and scrapes the framebuffer from it. This is way more layers.

So, as long as you have CPU accel (and virtio network drivers), I believe tigervnc's Xvnc installed on the guest will still be faster. Its also much smaller than a full fat Xlibre/Xorg (though you likely want one of them too for the libs / utilities).
Click to expand...

This is true... It is just convoluted, I haven't figured it out in my head yet...
 
Phishfry said:
www.sharpwriting.net

Install Virtual Machines Using Bhyve on FreeBSD | Sharp Writing

Setting up virtual machines with Bhyve on FreeBSD is simple and quick. I create a Fedora VM, but these steps should work for most distros.
www.sharpwriting.net www.sharpwriting.net
Click to expand...

The guide you gave made the process more straightforward, however I can't connect the VNC to the VM:

Code: 
doas vm list
NAME     DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO  STATE
freebsd  default    bhyveload  2    4G      -    No    Running (35875)

1767194096754.png


This is not the IP of my VM though, but I don't have a clue to solve this... 😓

Thanks... 🙏
 
If use "UEFI" ; I get errors and I am unable to connect:

Code: 
doas vm list
NAME     DATASTORE  LOADER  CPU  MEMORY  VNC           AUTO  STATE
freebsd  default    uefi    2    4G      0.0.0.0:5900  No    Locked (gpc)
vm ) 
doas vm console freebsd
/usr/local/sbin/vm: ERROR: unable to locate console device for this virtual machine

The switch is working:

Code: 
doas vm switch list
NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     ue0

Perhaps is an error on the template?

Code: 
loader="uefi"
graphics="yes"
graphics_res="1280x720"
xhci_mouse="yes"
cpu=2
memory=4G
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

Can I run a FreeBSD graphical session?
 
My script
%> cat /usr/local/bin/vmvnc
#!/bin/sh
# vmvnc — ouvre un bureau Openbox via TigerVNC dans une VM FreeBSD (bhyve) via SSH (tunnel)
# - Pas d’auth VNC (SecurityTypes None) : on se repose sur le tunnel SSH
# - Le serveur VNC dans la VM écoute uniquement en localhost (-localhost)
# - Force /bin/sh côté VM (pas de csh), évite les redirections csh ambiguës

VM_HOST="xxx.xxx.xx.x" # IP/nom de la VM
VM_USER="OP" # utilisateur dans la VM
GEOM="1920x1080" # résolution
LPORT="5905" # port local (hôte)
RPORT="5901" # port VM (display :1)
VIEWER="vncviewer" # tigervnc-viewer
SSH_OPTS='-o BatchMode=yes -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes -o ConnectTimeout=5'

set -eu

say() { printf '%s\n' "$*"; }
die() { printf 'ERREUR: %s\n' "$*" >&2; exit 1; }

wait_ssh() {
i=0
while ! ssh $SSH_OPTS -o ConnectTimeout=3 "${VM_USER}@${VM_HOST}" true 2>/dev/null; do
i=$((i+1)); [ $i -ge 20 ] && return 1
sleep 1
done
}

say "[0/4] Attente SSH sur ${VM_HOST}…"
wait_ssh || die "SSH injoignable sur ${VM_HOST} (réseau/sshd ?)"

say "[1/4] Vérification/démarrage VNC côté VM…"
# IMPORTANT : on exécute /bin/sh directement côté VM via -s (stdin), pour ne JAMAIS passer par csh.
# On injecte ${GEOM} depuis l’hôte (expansion locale dans le heredoc non-quoté).
ssh $SSH_OPTS "${VM_USER}@${VM_HOST}" /bin/sh -s <<EOF || die "Impossible de démarrer vncserver dans la VM."
PATH=/usr/local/bin:/usr/bin:/bin

# Nettoyage d’un ancien :1 (verrous + pid)
vncserver -kill :1 >/dev/null 2>&1 || true
rm -f /tmp/.X1-lock "\$HOME"/.vnc/*:1.pid /tmp/.X11-unix/X1 >/dev/null 2>&1 || true

# xstartup minimal (Openbox) si absent
if [ ! -x "\$HOME/.vnc/xstartup" ]; then
mkdir -p "\$HOME/.vnc"
printf "%s\n" "#!/bin/sh" "exec openbox-session" > "\$HOME/.vnc/xstartup"
chmod +x "\$HOME/.vnc/xstartup"
fi

# Si un serveur tourne déjà, ne pas échouer
if vncserver -list 2>/dev/null | grep -qE '^[[:space:]]*:1[[:space:]]'; then
exit 0
fi

# Démarrage TigerVNC (sans mot de passe VNC; on se repose sur le tunnel SSH)
vncserver :1 -geometry ${GEOM} -localhost -SecurityTypes None
EOF

say "[2/4] Ouverture du tunnel SSH localhost:${LPORT} → ${VM_HOST}:localhost:${RPORT}…"
if sockstat -4 -l | awk '{print $6}' | grep -q "^127\.0\.0\.1:${LPORT}\$"; then
say " (déjà ouvert)"
TUN_PID=""
else
ssh -f -N -L "${LPORT}:localhost:${RPORT}" $SSH_OPTS "${VM_USER}@${VM_HOST}" || die "Tunnel SSH échoué."
sleep 0.3
TUN_PID="$(pgrep -f "ssh -f -N -L ${LPORT}:localhost:${RPORT} ${VM_USER}@${VM_HOST}" || true)"
fi

say "[3/4] Lancement du viewer → localhost:${LPORT}"
command -v "$VIEWER" >/dev/null 2>&1 || die "Viewer VNC introuvable: pkg install -y tigervnc-viewer"
"$VIEWER" "localhost:${LPORT}"

say "[4/4] Nettoyage côté VM et fermeture tunnel…"
# On nettoie :1 proprement (sans passer par csh) ;
# ne pas échouer si déjà stoppé
ssh $SSH_OPTS "${VM_USER}@${VM_HOST}" /bin/sh -s <<'CLEAN_SH' || true
PATH=/usr/local/bin:/usr/bin:/bin
vncserver -kill :1 >/dev/null 2>&1 || pkill -f "Xvnc :1" >/dev/null 2>&1 || true
CLEAN_SH

# Fermer le tunnel si créé par ce script
if [ -n "${TUN_PID:-}" ]; then
kill "$TUN_PID" >/dev/null 2>&1 || true
fi

say "Terminé."
Click to expand...
Sorry, comments in French.
 
I confirm the issue is with the UEFI loader.

It looks vm-bhyve is not meant to run FreeBSD graphical session...

Jim L.

bhyve Thread 'How to boot a bhyve UEFI FreeBSD guest w/framebuffer boot menu graphics?'

I'd like to boot a UEFI vm-bhyve guest and see the FreeBSD orb graphic logo rather than the ASCII stick-art logo.

This does not accomplish that:

/bhyve/freebsd-guest/my.conf:
Code: 
loader="uefi"
graphics="yes"
graphics_res="1280x1024"
graphics_wait="yes"
xhci_mouse="yes"
cpu=2
memory=4G
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="784989bb-2f14-11ef-ace3-90e2ba78ae40"
network0_mac="58:9c:fc:0b:f6:f0"

Guest's /boot/loader.conf:
Code: 
zfs_load=yes
vesa_load=yes
videomode_load=yes
splash_bmp_load=yes
kern.geom.label.gpt=1...

☹️
 
elgrande said:
Try to connect via VNC to 127.0.0.1:5900 instead of 0.0.0.0:5900 - that fixed it for me.
Click to expand...

I can't even perform the installation, unless I install the OS in bhyveload, and when I finish the installation I switch to UEFI[/code]. Pretty weird...
 
freezr said:
I can't even perform the installation, unless I install the OS in bhyveload, and when I finish the installation I switch to UEFI[/code].

Pretty weird...
Click to expand...
Really weird... For me I can install directly into UEFI just fine.
What vm command do you exactly use for the installation?
 
Hi guys,

I don't understand why the switch doesn't not survive the reboot...
I mean, if I reboot the host the guest can't reach internet anymore.

EDIT:
Mixed scottro and vermaden recommendations:

On the host:

Code: 
rc.conf
hostname="gpc"
ifconfig_re0="inet 10.0.10.1 netmask 255.255.255.0"
ifconfig_ue0="DHCP"
gateway_enables="YES"
defaultrouter="10.0.10.10"

sysctl.conf
# vm-bhyve
net.inet.ip.forwarding=1
net.link.tap.up_on_open=1
 
I've also been seeing trouble with bhyve lately. For me, the last time I rebooted, I lost the switch. My bhyves are on a machine that I don't reboot very often, but if I remember last time I had to reboot, I wound up doing pkg remove vm-bhyve then pkg install vm-bhyve then re-creating the switch. As none of my bhyves are that important to me, I haven't thoroughly tested, investigated, or even reboot again to see if it's still an issue. Here's my thread on it.

S

bhyve Thread 'Network seems to stopped working in vm-bhyve (failed to find virtual switch 'public')'

I've been using vm-bhyve for awhile with Linux, Windows, and FreeBSD guests. Probably around a month ago, but I'm not exactly sure, the guest will seem to start, but then if I try to connect, I can't. If I then do vm list, it shows that it has stopped. The log will say (failed to find virtual switch.
'public' ).
I have always just created a virtual switch, then run vm switch add public re0. (My ethernet card.) Having looked at vermaden's pages, he also says that one should
Code: 
sysrc gateway_enable=YES
sysrc gateway_enable=YES
sysctl net.inet.ip.forwarding=1
host  echo...

Various folks kindly tried to help, and shared their working configs, but none of said configs gave me any improvement.
 
It is not working again. ☹️

I made it working by accident, don't know what to do to make the connection working at every boot...

EDIT:

I got the bridge doesn't have an IP address, I guess if I makes the bridge working it work the network on the VM as well:

Code: 
doas vm switch list
NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     ue0

Code: 
vm-public: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=10<VLAN_HWTAGGING>
    ether 72:34:42:48:1d:7f
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 4 priority 128 path cost 2000000 vlan protocol 802.1q
    groups: bridge vm-switch viid-4c918@
    nd6 options=9<PERFORMNUD,IFDISABLED>
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=0
    ether 06:69:00:37:d5:41
    inet 10.84.151.100 netmask 0xffffff00 broadcast 10.84.151.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 
I found a partial reply here: https://gitmemories.com/churchers/vm-bhyve/issues/435

Me-too comment. As soon as the vm is started, bridge's IP address is gone, and guests's access to the network with it, Once added again via ,icode.vm switch address public <IP/mask>[/icode] everything is back to normal.
Click to expand...

And this worked out:

Code: 
doas vm switch address public 10.84.151.100/24
vm ) doas vm switch list
NAME    TYPE      IFACE      ADDRESS           PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  10.84.151.100/24  no       -    -     ue0

You need to poweroff or restart your VM if the bridge continues to not work.

If you assign the IP manually you don't need to make any changes to rc.conf and sysctl.com.
 
Unfortunately the steps I used yesterday are not working today.

And even though the switch is assigned properly is not reaching the virtual machine:

Host ifconfig:
Code: 
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vm-public: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=10<VLAN_HWTAGGING>
    ether 1e:b1:d5:82:b8:4e
    inet 10.84.151.100 netmask 0xffffff00 broadcast 10.84.151.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 5 priority 128 path cost 2000000 vlan protocol 802.1q
    groups: bridge vm-switch viid-4c918@
    nd6 options=9<PERFORMNUD,IFDISABLED>
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=0
    ether 62:5c:3c:17:16:69
    inet 10.84.151.100 netmask 0xffffff00 broadcast 10.84.151.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tap0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: vmnet/sp/0/public
    options=4080000<LINKSTATE,MEXTPG>
    ether 58:9c:fc:10:b1:1a
    groups: tap vm-port
    media: Ethernet 1000baseT <full-duplex>
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 48775

vm-bhyve[icode] switch: [code] doas vm switch list NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS public standard vm-public 10.84.151.100/24 no - - ue0 [/code] Guest ifconfig: [code] vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=880028<VLAN_MTU,JUMBO_MTU,LINKSTATE,HWSTATS> ether 58:9c:fc:01:54:0f media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> [/code] The bridge is not working the [icode]vtnet0 does not have any ip assigned and it cannot connect to internet.

I have not idea if the problem is the tap0 interface, however if I try to force a poweroff at the next reboot the text of the vm becomes completely scrambled.

This situation is driving me crazy… 😖
 
You must log in or register to reply here.
Back
Top