bhyve Configuring cpu, socket, core, thread per each vm?

[kalleboy]

Hello, happy holidays!

Trying to configure CPU values for my bhyve VMs but I'm totally stucked; could someone please kindly quickly explain difference between "cpu=", "socket=", "cpu_cores=" and "cpu_threads="?

I'm truly confused with all those values.

For instance, having a dedicated host machine with a CPU: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz (4 cores - 8 threads),
does that mean, none of my VMs could go more than 4 cores and/or 8 threads? If so, it seems I can only create just few VMs according to my physical host machine's CPU limitation?

Thanks in advance!

 

[amilis]

Hi there!

Yes and No, you are correct.
It’s little complicated, but to make it simple. You have 8 threads, so you “have” 8 vCPU (virtual CPUs to VMs). But! If you have 8 hard working vCPU, then you need some resource to the host as well. So 7 hard working vCPUs. Well.. it’s not that simple, but we hold it in this way.

Say that you have 7 very very lite vCPUs, they more or less idle for the most time. Then you can have some more vCPUs, 10, 12, 20!! It’s deepens on your load.

But, then you can dedicate CPU core(s) for vCPUs. And 1 core = 2 threads = 2 vCPUs. Then you only have 6 vCPU if you dedicate 1 core to a VM.

It’s many “but” here..

Check you host CPU load. If it’s idling, you may have another VM or more vCPUs to a VM.

You can over shot vCPU (you have 8 threads, but use 10-16 vCPUs). But you CAN NOT over shot memory! If you have 32 GB ram in your box, you can only dedicate that. Noting more! But.. again! You need some memory for your host as well. So don’t put all your memory to VMs.

Make some test-VMs and run CPU stress test and check the host CPU. If it’s nere 100% on all 8 threads. Run another VM with some vCPUs and see what’s happen and if it’s gets slower etc. Then you get a hum of where your limits are. Buuut! Don’t do this on a live/production server!

 

[kalleboy]

Hi amilis - Well, I'm even more confused now

For instance, how would to define those "cpu, socket, core, thread" values in config file, if you wish to have a VM with 4 cores and 8 threads?

 

[amilis]

I only use “cpu=X” in my configs (I don’t specify any cores or threads), and I have VMs with 1 vCPU to 24 vCPUs, all detected and running very smooth.

I don’t use Windows (with can be picky) or other OS? that may need to change the settings. If you only put cpu=X for a Windows-server, it may not see all the “cores” (vCPU). In that case you may need cpu_cores=X. As an example. The sockets, cores and threads tells the VM, or OS more about witch CPU configuration you have, if the OS is not detecting it.

If you need specify 4 cores and 8 threads:
cpu_cores=4
cpu_threads=8

But do you need to specify it?

 

[bgavin]

I use an ESXi host with an Intel i7 configured as 4 cores / 8 threads.
My host has some 20+ VMs installed, but each is only powered up when required.
I can "overbook" cpu resources higher than the above configuration, but figure things really slow down from context switching.

 

[PMc]

Lets start from the beginning. You have i7 4c/8t. The host OS will see 8 cores, and distribute the load among them. But in fact you have only 4 physical cores, each with dual pipeline (hyperthreading). The actual compute that you can get is about 4 + 4/3 = 5.3 cores (rule of thumb for average load).

Then, enter bhyve. Option -c gives the number of (virtual) cores that the guest will see. And bhyve will push that many instructions in parallel up to the host. This can be any number, but if it is more than the host can currently accomodate, performance will become horrible.

For most configurations without NUMA this should be enough. You can configure sockets/cores/threads, but there is not much point in doing so, because the host will throw the upcoming instructions on any available core, randomly.

It gets more interesting when you work with dedicated cpus. Your host OS has 8 cores enumerated 0..7, and, say, you decide to use two cores for something specific, 4 others for something else, etc. Then you need to tell bhyve not only how many, but exactly which cores it should use. This is done with the pinning option -p. Let's assume we want to use the upper four cores 4...7 for a bhyve, then we state -c 4 -p 0:4 -p 1:5 -p 2:6 -p 3:7.

But now we no longer have four equal vcpus in the guest, because vcpu 0+1 are just threads on the same physical core, as are 2+3. Running a program on vcpu 0+2 will give us two full cores performance, but running it on 0+1 will give only 1.3 performance. The guest OS may want to know this and make use of that information in some way. So we need a means to tell it. And that is what the sockets/cores/threads is good for.

In this example it would become cpus=4 sockets=1 cores=2 threads=2 (according to manpage, the product of sockets*cores*threads must be equal numcpus)

Obviousely this gets more interesting when we have multiple chips in the machine, and the different bandwidths of QPI/UPI versus the internal ring come into play.