bhyve Is it possible to expand volumes inside FreeBSD guest VM without reboot?

Chris I.

Chris I.

Our organization is running FreeBSD 13 and 14 as guests VM's both under bhyve and also AWS. Despite lots of trying, it seems like detecting expanded drives (nvme emulation) requires a reboot. Am I missing something?

I recently expanded the drive of a Ubuntu Linux guest machine, where FreeBSD was the host using bhyve. I simply expanded the host's drive file using truncate. Immediately inside the guest, the larger disk could be seen with lsblk, and then utilized with zpool online.

It'd sure be swell if FreeBSD as guest and host (using bhyve) could support expansion in this same elegant manner. I found an older post on stackexchange that suggested there is a disconnect between the nvmecontrol and gpart layer, but I'm not sure if that would still apply.

I've already experimented a good deal with forced rescan of devices inside the guest, devctl etc., with no luck. It seems like it could be within reach. nvmecontrol devlist shows the new size, but I haven't been able to grow into it without a reboot.
 
I don't think that geom is provisioned for resizing out of the blue at all. I find it interesting that nvmecontrol can see the new size, good find. I'll put it on the TODO to snoop a little.
 
I haven't tried this myself, so YMMV, on the off chance that it's not out of date or wrong, the handbook does talk about this and explicitly notes how to do it without unmounting. I can't even find information on the -e switch, so this may very well be wrong, and definitely try this this on any VM that you haven't got good backups of.
Code: 
# zpool online -e zroot /dev/ada0p2

The one obvious potential issue there is that the disk space might need to already exist,the handbook was not clear about that detail.
 
If I'm understanding it correctly, FreeBSD 15 works like Linux according to this:

www.freebsd.org

FreeBSD 15.0-RELEASE Release Notes

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org

It says: "Add support for dynamically resizing NVMe namespaces. The nvd(4) and nda(4) drivers now notify geom of sizes changes in real time. 86d3ec359a56 (Sponsored by Netflix)"

That "86d3ec359a56" is:

https://cgit.freebsd.org/src/commit/?id=86d3ec359a56
 
AlfredoLlaquet said:
If I'm understanding it correctly, FreeBSD 15 works like Linux according to this:

www.freebsd.org

FreeBSD 15.0-RELEASE Release Notes

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org

It says: "Add support for dynamically resizing NVMe namespaces. The nvd(4) and nda(4) drivers now notify geom of sizes changes in real time. 86d3ec359a56 (Sponsored by Netflix)"

That "86d3ec359a56" is:

https://cgit.freebsd.org/src/commit/?id=86d3ec359a56
Click to expand...

That explains why nvmecontrol can see the new size. But I don't think that GEOM has to pick that up.
 
To bad the guests are not running FreeBSD 15.0.

Just tested in a 15.0-RELEASE bhyve(8) guest (on a 15.0 host). A increase in disk size (truncate(1)), while the vm is running, is picked up immediately:

gpart.png

Chris I. , FreeBSD developers familiar with the issue are on the FreeBSD mailing lists, perhaps ask on freebsd-questions@ or freebsd-current@ if there is a workaround on the 13, 14 branches.
 
You must log in or register to reply here.
Back
Top