Solved Shutdown multiple Bhyve VM's

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

How can I shutdown multiple VM's with Bhyve?
Now I do it with multi line commands:
Code:
bhyvectl --destroy --vm=freebsd1
bhyvectl --destroy --vm=freebsd2
bhyvectl --destroy --vm=freebsd3

I tried several iterations:
Code:
bhyvectl --destroy --vm=freebsd1 --vm=freebsd2 --vm=freebsd3
bhyvectl --destroy --vm=freebsd1,freebsd2,freebsd3
How can I tear down multiple instances in one line?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,274
Messages: 38,775

How can I tear down multiple instances in one line?
You can't. The options only accept one argument. So you'll have to shutdown each one individually.

The sysutils/vm-bhyve tool only has a stopall and startall argument, vm stop or vm start doesn't accept multiple hosts either.
 
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

Right now I am using 4 FreeBSD VM's and one Devuan with Grub2.

Setting up my Poudriere server in one now. Ended up using a 120GB image

I opted for the /etc/rc.local file and stuffed it with my 5 VM commands and their loaders too.

Any more I could see the utility of an helper.

I could have gone the rc.d route but just messing around for now.

/etc/rc.local
Code:
#!/bin/sh
#bhyveload -m 4G -S -d /vm/freebsd/freebsd1.img freebsd1;
#bhyveload -m 4G -S -d /vm/freebsd/freebsd2.img freebsd2;
#bhyveload -m 4G -S -d /vm/freebsd/freebsd3.img freebsd3;
#bhyveload -m 8G -S -d /vm/freebsd/freebsd4.img freebsd4;
#grub-bhyve -m /vm/devuan/device.map -r hd0,msdos1 -M 2048M -S devuan1 -d /boot/grub/;
#/usr/sbin/daemon -f bhyve -S -m 4G -c 4 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 5:0,passthru,6/0/0 -s30,xhci,tablet -l com1,/dev/nmdm1A freebsd1;
#/usr/sbin/daemon -f bhyve -S -m 4G -c 4 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd2.img -s 5:0,passthru,6/0/1 -s30,xhci,tablet -l com1,/dev/nmdm2A freebsd2;
#/usr/sbin/daemon -f bhyve -S -m 4G -c 4 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd3.img -s 5:0,passthru,7/0/0 -s30,xhci,tablet -l com1,/dev/nmdm3A freebsd3;
#/usr/sbin/daemon -f bhyve -S -m 8G -c 24 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd4.img -s 5:0,passthru,7/0/1 -s30,xhci,tablet -l com1,/dev/nmdm4A freebsd4;
#/usr/sbin/daemon -f bhyve -S -m 2048M -c 4 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/devuan/devuan1.img -s 5:0,passthru,132/0/0 -s30,xhci,tablet -l com1,/dev/nmdm5A devuan1;
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,274
Messages: 38,775

Setting up my Poudriere server in one now.
I've tried this too. It's dead slow, mainly due to slow I/O. Building 200 or so packages on my 'simple' Core i5 (2 core, 4 threads) server takes around 3-4 hours. Building the same list on the VM (4 cores) took more than 12. So I gave up on that idea and left the package building on the Core i5 server (besides serving files it doesn't do much anyway).
 
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

I am not that good with scripting so I was not sure if I launched my bhyve instances correctly using daemon(8)
The way that bhyveload works, it dumps back to a command prompt, so no problem.
But with the bhyve commands -it needs to spawn new instances.
I tried suffixing the '&' symbol to send application to background but it did not work right.
Somehow I stumbled into daemon and it worked for me. Is this the proper usage?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,274
Messages: 38,775

I'd say yes. I've used daemon(8) before too, to get some in-house built application to start at boot. It's sometimes a little tricky to get the PID correct (do I need -p or -P?) but it should be the correct way to start things.
 
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

It is nice to peek under the hood to see what these bhyve helpers do thanks to open source and github.
tingo's suggestion is timely as I stumbled into the same path, trying to send stdio to a tmux console, so it would remain alive.
Still struggling to figure out how to run a VM with both stdio and nmdm outputs active, like a standard serial console install allows.
 
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

I am guessing a dual console (stdio & nmdm) might be available with the UEFI rom loaded.
There seems to be a framebuffer with the UEFI setup and its additional -l options.
I have not loaded the UEFI roms yet although I have them installed.
 
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

Browsing through the rc.d script that is sysutils/bhyve-rc I noticed it is hard wired to a tap device.
I am using pass-through mode for ethernet cards and it works wonderfully.
Using SuperMicro 4 port card with igb interfaces and a HP card with 4 em interfaces.
I always disliked bridges and taps. So a straight-thru approach is the ultimate way for me.

I might modify that script for my use though. It is great work.
 

Vull

Aspiring Daemon

Reaction score: 436
Messages: 733

How can I shutdown multiple VM's with Bhyve?
Now I do it with multi line commands:
Code:
bhyvectl --destroy --vm=freebsd1
bhyvectl --destroy --vm=freebsd2
bhyvectl --destroy --vm=freebsd3

I tried several iterations:
Code:
bhyvectl --destroy --vm=freebsd1 --vm=freebsd2 --vm=freebsd3
bhyvectl --destroy --vm=freebsd1,freebsd2,freebsd3
How can I tear down multiple instances in one line?
If they're always numbered sequentially like in your example ...
Code:
#!/bin/sh
for x in 1 2 3; do bhyvectl --destroy --vm=freebsd$x; done
... or ...
Code:
#!/bin/sh
x=0; n=3; while [ $x -lt $n ]; do x=$(($x+1)); bhyvectl --destroy --vm=freebsd$x; done
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,274
Messages: 38,775

You can do something similar with names:
Code:
#!/bin/sh
for vm in onevm anothervm somemorevm; do bhyvectl --destroy --vm=${vm}; done
 
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

So I would like to fool around with shutting down my VM's with a SIGTERM signal to the bhyve [pid]
FreeBSD 10.1 and later treat the SIGTERM signal sent to bhyve(8) as a virtual power button. If the guest supports ACPI, then sending SIGTERM interrupts the guest to request a clean shutdown[1].
Looking at signal(3) did not help me here as I would like a command line option.

How can I send a SIGTERM to a process [pid] from the command line or in a script?

[1]http://bsdjhb.blogspot.com/2018/10/using-bhyve-for-freebsd-development.html
 
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

It looks like the command line uses kill(1) with TERM

This shutdown correctly but left remains in /dev/vmm so I need to bhyve --destroy after VM shutdown.
kill -s TERM 3490
kill -s TERM 3492
kill -s TERM 3494
kill -s TERM 3496
 
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

Nice. I poked thru the /var/log/messages on a VM to confirm graceful shutdown and see this:
Jan 29 23:29:43 freebsd3 syslogd: exiting on signal 15
Signal 15 is what I was looking for.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,274
Messages: 38,775

SIGTERM is the default signal of kill(1), so you don't have to provide it.

Code:
     -s signal_name
             A symbolic signal name specifying the signal to be sent instead
             of the default TERM.

Sending a shutdown command to all VMs could be done with pkill(1).
 
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

You beat me to it. I just verified only kill and pid is needed.
Jan 30 04:57:43 freebsd1 syslogd: exiting on signal 15
 
Last edited:
OP
Phishfry

Phishfry

Beastie's Twin

Reaction score: 2,667
Messages: 5,587

I am finding ps -ax|grep bhyve much more informative than ls /dev/vmm
I think learning the bhyve basics here is a worthy undertaking. Even if I later use vm-bhyve.
Now I have learned starting up and shutting down VM's with byhve.
It also taught me valuable information about the base system.
 
Top