Solved Shutdown multiple Bhyve VM's

How can I shutdown multiple VM's with Bhyve?
Now I do it with multi line commands:
Code:
bhyvectl --destroy --vm=freebsd1
bhyvectl --destroy --vm=freebsd2
bhyvectl --destroy --vm=freebsd3

I tried several iterations:
Code:
bhyvectl --destroy --vm=freebsd1 --vm=freebsd2 --vm=freebsd3
bhyvectl --destroy --vm=freebsd1,freebsd2,freebsd3
How can I tear down multiple instances in one line?
 
Right now I am using 4 FreeBSD VM's and one Devuan with Grub2.

Setting up my Poudriere server in one now. Ended up using a 120GB image

I opted for the /etc/rc.local file and stuffed it with my 5 VM commands and their loaders too.

Any more I could see the utility of an helper.

I could have gone the rc.d route but just messing around for now.

/etc/rc.local
Code:
#!/bin/sh
#bhyveload -m 4G -S -d /vm/freebsd/freebsd1.img freebsd1;
#bhyveload -m 4G -S -d /vm/freebsd/freebsd2.img freebsd2;
#bhyveload -m 4G -S -d /vm/freebsd/freebsd3.img freebsd3;
#bhyveload -m 8G -S -d /vm/freebsd/freebsd4.img freebsd4;
#grub-bhyve -m /vm/devuan/device.map -r hd0,msdos1 -M 2048M -S devuan1 -d /boot/grub/;
#/usr/sbin/daemon -f bhyve -S -m 4G -c 4 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 5:0,passthru,6/0/0 -s30,xhci,tablet -l com1,/dev/nmdm1A freebsd1;
#/usr/sbin/daemon -f bhyve -S -m 4G -c 4 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd2.img -s 5:0,passthru,6/0/1 -s30,xhci,tablet -l com1,/dev/nmdm2A freebsd2;
#/usr/sbin/daemon -f bhyve -S -m 4G -c 4 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd3.img -s 5:0,passthru,7/0/0 -s30,xhci,tablet -l com1,/dev/nmdm3A freebsd3;
#/usr/sbin/daemon -f bhyve -S -m 8G -c 24 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd4.img -s 5:0,passthru,7/0/1 -s30,xhci,tablet -l com1,/dev/nmdm4A freebsd4;
#/usr/sbin/daemon -f bhyve -S -m 2048M -c 4 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/devuan/devuan1.img -s 5:0,passthru,132/0/0 -s30,xhci,tablet -l com1,/dev/nmdm5A devuan1;
 
Setting up my Poudriere server in one now.
I've tried this too. It's dead slow, mainly due to slow I/O. Building 200 or so packages on my 'simple' Core i5 (2 core, 4 threads) server takes around 3-4 hours. Building the same list on the VM (4 cores) took more than 12. So I gave up on that idea and left the package building on the Core i5 server (besides serving files it doesn't do much anyway).
 
I am not that good with scripting so I was not sure if I launched my bhyve instances correctly using daemon(8)
The way that bhyveload works, it dumps back to a command prompt, so no problem.
But with the bhyve commands -it needs to spawn new instances.
I tried suffixing the '&' symbol to send application to background but it did not work right.
Somehow I stumbled into daemon and it worked for me. Is this the proper usage?
 
I'd say yes. I've used daemon(8) before too, to get some in-house built application to start at boot. It's sometimes a little tricky to get the PID correct (do I need -p or -P?) but it should be the correct way to start things.
 
It is nice to peek under the hood to see what these bhyve helpers do thanks to open source and github.
tingo's suggestion is timely as I stumbled into the same path, trying to send stdio to a tmux console, so it would remain alive.
Still struggling to figure out how to run a VM with both stdio and nmdm outputs active, like a standard serial console install allows.
 
I am guessing a dual console (stdio & nmdm) might be available with the UEFI rom loaded.
There seems to be a framebuffer with the UEFI setup and its additional -l options.
I have not loaded the UEFI roms yet although I have them installed.
 
Browsing through the rc.d script that is sysutils/bhyve-rc I noticed it is hard wired to a tap device.
I am using pass-through mode for ethernet cards and it works wonderfully.
Using SuperMicro 4 port card with igb interfaces and a HP card with 4 em interfaces.
I always disliked bridges and taps. So a straight-thru approach is the ultimate way for me.

I might modify that script for my use though. It is great work.
 
How can I shutdown multiple VM's with Bhyve?
Now I do it with multi line commands:
Code:
bhyvectl --destroy --vm=freebsd1
bhyvectl --destroy --vm=freebsd2
bhyvectl --destroy --vm=freebsd3

I tried several iterations:
Code:
bhyvectl --destroy --vm=freebsd1 --vm=freebsd2 --vm=freebsd3
bhyvectl --destroy --vm=freebsd1,freebsd2,freebsd3
How can I tear down multiple instances in one line?
If they're always numbered sequentially like in your example ...
Code:
#!/bin/sh
for x in 1 2 3; do bhyvectl --destroy --vm=freebsd$x; done
... or ...
Code:
#!/bin/sh
x=0; n=3; while [ $x -lt $n ]; do x=$(($x+1)); bhyvectl --destroy --vm=freebsd$x; done
 
You can do something similar with names:
Code:
#!/bin/sh
for vm in onevm anothervm somemorevm; do bhyvectl --destroy --vm=${vm}; done
 
So I would like to fool around with shutting down my VM's with a SIGTERM signal to the bhyve [pid]
FreeBSD 10.1 and later treat the SIGTERM signal sent to bhyve(8) as a virtual power button. If the guest supports ACPI, then sending SIGTERM interrupts the guest to request a clean shutdown[1].
Looking at signal(3) did not help me here as I would like a command line option.

How can I send a SIGTERM to a process [pid] from the command line or in a script?

[1]http://bsdjhb.blogspot.com/2018/10/using-bhyve-for-freebsd-development.html
 
It looks like the command line uses kill(1) with TERM

This shutdown correctly but left remains in /dev/vmm so I need to bhyve --destroy after VM shutdown.
kill -s TERM 3490
kill -s TERM 3492
kill -s TERM 3494
kill -s TERM 3496
 
Nice. I poked thru the /var/log/messages on a VM to confirm graceful shutdown and see this:
Jan 29 23:29:43 freebsd3 syslogd: exiting on signal 15
Signal 15 is what I was looking for.
 
SIGTERM is the default signal of kill(1), so you don't have to provide it.

Code:
     -s signal_name
             A symbolic signal name specifying the signal to be sent instead
             of the default TERM.

Sending a shutdown command to all VMs could be done with pkill(1).
 
You beat me to it. I just verified only kill and pid is needed.
Jan 30 04:57:43 freebsd1 syslogd: exiting on signal 15
 
Last edited:
I am finding ps -ax|grep bhyve much more informative than ls /dev/vmm
I think learning the bhyve basics here is a worthy undertaking. Even if I later use vm-bhyve.
Now I have learned starting up and shutting down VM's with byhve.
It also taught me valuable information about the base system.
 
Back
Top