bhyve bhyve memory limit

B

batrick

Hello, I've got a VM working well with -m 16G out 64GB available RAM. As soon as I set it to 17GB however, I get:

Code: 
Unable to setup memory (22)

Is there some 25% memory restriction that's undocumented?
 
Last edited by a moderator:
I don't think so. I'm running CentOS 9 that was installed with:

Bash: 
bhyve -c 16 -m 8G -H -s 0:0,hostbridge -s 1:0,lpc -s 2,virtio-net,tap21 -s 3,ahci-cd,/dev/zvol/zroot/iso/CentOS-Stream-8-20230606.0-x86_64-boot.iso -s 4,virtio-blk,/dev/zvol/zroot/bhyve/boot -s 5,virtio-blk,/dev/zvol/zroot/bhyve/root -s 6,virtio-blk,/dev/zvol/zroot/bhyve/home -s 29,fbuf,tcp=127.0.0.1:5900,w=1280,h=720 -l com1,/dev/nmdm-vm.1A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd vm

then run using:

Bash: 
bhyve -c 16 -m 8G -H -s 0:0,hostbridge -s 1:0,lpc -s 2,virtio-net,tap21,mac=XXX -s 4,virtio-blk,/dev/zvol/zroot/bhyve/boot -s 5,virtio-blk,/dev/zvol/zroot/bhyve/root -s 6,virtio-blk,/dev/zvol/zroot/bhyve/home -s 29,fbuf,tcp=127.0.0.1:5900,w=1280,h=720 -l com1,/dev/nmdm-vm.1A -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd vm
 
Both commands have the memory set to 8 GB?

I'm not aware of any memory limit with bhyve(8), but perhaps the memory needs to be continuously (i.e. it has to be one single chunk, it cannot be fragmented). So even if you have enough 'free' memory it may have been too fragmented.
 
I thought that may be part of it but the 25% ratio is an odd coincidence. Yes, when I originally setup the VM I gave it 8GB. Raising it later to 16GB was not an issue. I cannot go above that, e.g. 17GB.
 
When I get home tonight I'll try some things on my bhyve(8) server. Server has 96 GB, so if the 25% is valid then I shouldn't be able to run a VM with 25GB of memory.
 
Oh, oops. Forgot about it. Working from home now, so quick test:
Code: 
root@hosaka:~ # vm list | grep fbsd-test
fbsd-test       stor10k    bhyveload  2    25G     -             Yes [13]  Running (1836)

No issues at all. Host has been running for a while so I'm sure memory wasn't exactly free either.
Code: 
last pid:  2446;  load averages:  1.32,  1.30,  1.43                                                       up 33+12:17:58  10:06:24
61 processes:  1 running, 60 sleeping
CPU:  0.0% user,  0.0% nice,  6.3% system,  0.1% interrupt, 93.5% idle
Mem: 14G Active, 22G Inact, 4785M Laundry, 35G Wired, 2822M Free
ARC: 24G Total, 9123M MFU, 12G MRU, 2468K Anon, 1829M Header, 479M Other
     20G Compressed, 28G Uncompressed, 1.38:1 Ratio
Swap: 16G Total, 4225M Used, 12G Free, 25% Inuse
Code: 
root@hosaka:~ # vm list
NAME            DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO      STATE
case            default    bhyveload  4    4096M   -             Yes [4]   Running (2578)
jenkins         default    bhyveload  4    4096M   -             Yes [6]   Running (3080)
kdc             default    uefi       2    2048M   0.0.0.0:5901  Yes [2]   Running (2026)
lady3jane       default    bhyveload  4    8192M   -             Yes [3]   Running (2331)
debian          stor10k    uefi       4    4096M   -             No        Stopped
errol           stor10k    bhyveload  2    4096M   -             Yes [9]   Running (3855)
fbsd-test       stor10k    bhyveload  2    25G     -             Yes [13]  Running (1836)
foreman         stor10k    uefi       4    8192M   -             No        Stopped
gl-runner-1     stor10k    bhyveload  4    4096M   -             Yes [12]  Running (4905)
gl-runner-2     stor10k    bhyveload  4    4096M   -             No        Stopped
k8s             stor10k    grub       4    8192M   -             Yes [14]  Running (5216)
kibana          stor10k    bhyveload  4    8192M   -             Yes [1]   Running (41183)
phabricator     stor10k    bhyveload  1    4096M   -             No        Stopped
plex            stor10k    bhyveload  4    4096M   -             Yes [7]   Running (3335)
riviera         stor10k    bhyveload  2    4096M   -             Yes [11]  Running (4622)
sdgame01        stor10k    bhyveload  4    4096M   -             No        Stopped
tessierashpool  stor10k    bhyveload  2    4096M   -             Yes [5]   Running (3066)
ubuntu          stor10k    uefi       2    4096M   -             No        Stopped
wintermute      stor10k    bhyveload  4    4096M   -             Yes [10]  Running (81885)
 
I upgrade to 13.2 and that may have solved it for me. No issue increasing the core count or memory to 48GB/64GB.

Thanks for your help.
 
You must log in or register to reply here.
Back
Top