bhyve Bhyve VMs performance degradation after upgrate to 12.2+latest ports


Did someone notice performance degradation of the Bhyve VMs after upgrading to the 12.2?

I followed typical flow to upgrade:
- freebsd-update upgrade -r 12.2-RELEASE
- freebsd-update install
- reboot
- freebsd-update install
- delete all ports installed
- portsnap fetch update
- install ports I need (vm-bhyve, mc, etc.)
- freebsd-update install
- update bootcodes on all disks
- reboot

All VMs and config files remained the same. But the performance of VMs degraded a lot (

Tried 13.0-RC3 - better than 12.2 but still much worse than 12.1.

I assume that issue might be caused by a port version, not by kernel. I tried to roll-back the kernel only (to 12.1) but go with the latest ports tree... And got almost the same low performance.

Now I've reinstalled 12.1 from the original ISO and used "original" old ports collection (which goes with that ISO). VMs running fast again. But having an old kernel and ports is insecure...
On the other hand - having slow-running VMs not an option as well.

What can I do to figure out the reason of such performance degradation to get rid of it?

My system was upgraded to 12.2 with no problems. All my VM's are up and running.
What king of VM's ...Windows or Linux Or FreeBSD? Is it networking, cpu's, I/O,....?
I am agreed with acheron. What is the meaning of "performance degradation? ;)
Define "performance degradation" please.
I have some Windows VMs with MS SQL server installed.
With 12.1 these servers run smoothly - without SQL locks/waits and perform about 15-20k transactions per sec.
But after upgrade to 12.2, a lot of SQL locks/waits appeared and number of transactions fall-down to 2-5k per sec.

In both cases CPU, RAM and disk were not loaded more than 50-70%.

Unfortunately, I was unable to find the root cause of such strange behavior of VMs with MS SQL. So I rolled back to 12.1.
Tried 13-RC3 as well - better than 12.2, but still much worse than 12.1.

I'll appreciate it if you'll suggest to me a good way to do quick system test to figure out what might be a bottleneck with the FreeBSD system.
Or at least to do a comparison between 12.1 and 12.2 to figure out what is changed in terms of performance.

P.S. Maybe something wrong with the new port(s) because I rolled back to the initial 12.1 ports set - i.e. ports set from the 12.1 ISO.
I have no idea about windows tools. In general, the more information you can collect the better. Also important: try to minimize external influence - do not benchmark a virtual machine on a host with many virtual machines - the workload on those virtual machines will probably render any data useless. Try to run the VM you want to benchmark on a dedicated host, where nothing else is interferring during the benchmark, do your benchmarks with the host as 12.1 as well as 13.0 - so you can really compare how much slower it is and mail the facts to the mailing list.

To have a concrete idea: go through your SQL logs and pick the slowest queries. Run them in the VM, compare them, or get a dataset and a script or a tool to do a MS SQL Server benchmark. Dedicate a new empty disk to the VM, do some input/output benchmarks in the VM and compare the value - maybe a tool like iometer or crystaldiskmark helps, or use a search engine for alternatives. It might also be interesting to see what a simple "openssl speed" will deliver, or maybe you run a webserver and test the http connections from another machine using the tool "ab" (apache benchmark) ... or measure throughput with iperf3.

You can also do some tests with those tools on the host, e.g. running a webserver on the host and using another machine to run the tool "ab".

This is for sure some work to do, but that information is also valuable to developers, and you might even be the hero to reveal a serious bug!
I gave up and decided to do a full reinstall from the scratch for both - host and guest systems. And it helped.
Now I have 13.0-RELEASE on the host and everything works well so far.
Unfortunately, I didn't figure out what was the reason for such an accident. But I do hope that it will not reappear in the future.