bhyve kldload vmm.ko crashes system (kernel error)

W

wincc

New to FreeBSD...
I installed "FreeBSD 14 current" on my WRX80-SU8-IPMI with a AMD Ryzen Threadripper PRO 3955WX and was trying "kldload vmm" , the same computer has Arch Linux with QEMU/KVM installed on another disk so I´m sure virtualization is working.
I have run dmesg and found POPCNT processor feature flag on the Features2 line.

What do I need to test to find the "problem" ?
 
Firstly, the forums don't support CURRENT. There's a post about it somewhere, its basic premise is that CURRENT is unsupported and should be considered alpha. The other thing is that if CURRENT discussions are allowed, they often quickly sidetrack into really off-topic discussions, such as what should and shouldn't be included in FreeBSD, discussions that are probably better off in places like reddit. Here's the post. It's in the forum rules and guldeines and good to go over before posting.

forums.freebsd.org

Topics about unsupported FreeBSD versions

The FreeBSD Forums cater primarily to end-users and systems administrators. As such, the Forums focus almost exclusively on FreeBSD versions that are officially supported according to the official FreeBSD website. Since resources are scarce, the FreeBSD Forums strongly suggest that anyone asking...
forums.freebsd.org forums.freebsd.org

Questions with an install of CURRENT should go to the CURRENT mailing list. I suspect you may be new to FreeBSD, and didn't realize this, cleverly deduced by seeing the first line in your post. :)
So, I would suggest installing FreeBSD-13.1-RELEASE. I, and I'm sure many others, have no problem with bhyve on it.

I'm going to add a link that clearly explains CURRENT, RELEASE, et al, written by Freddy Cash. It's old dealing with FreeBSD-5 and 6 but still relevant.

An Explanation of FreeBSD Releases

srobb.net
 
wincc said:
New to FreeBSD...
I installed "FreeBSD 14 current"
Click to expand...
Yep, first error here. And scottro I share your assumption this happened for just not knowing. Although I really wonder what else could be done on freebsd.org to avoid this. It clearly states which versions are production releases :-/

Seems proper release engineering is so uncommon nowadays, people are just too trained to jump on what they think was "the latest and greatest"...

wincc I underline the recommendation, get 13.1-RELEASE (which can be upgraded to 13.2-RELEASE quite soon).

-CURRENT is almost exclusively for developers, in some rare cases, an end user could want it as well, but certainly not a beginner. Adding another detail here: the snapshots published of -CURRENT are built with all sorts of debugging stuff, which makes them perform horribly (cause, they're meant to do development, see ...). You have to compile the whole system yourself in order to get a -CURRENT without that debugging stuff. Just don't use.

BTW, new features that are tested and proved are regularly merged to the "stable" branches from which new -RELEASE versions are created. So once 13.2-RELEASE is out, you will have everything that's considered "ready for prime" in a stable, tested, engineered release ;)
 
Ok, Sorry for that mistake, as Arch Linux user I normally try the latest... I have now also tried FreeBSD-13.2-STABLE-amd64-20230223-bc68c55166ad-254667 which has the same "problem".
I install the recommended FreeBSD-13.1-RELEASE and see if that solves my "problem".

For information I first installed "FreeBSD 14 current" on my 9y old Samsung NP900X3E laptop and there bhyve works as I expected, therefor I wrongly assumed that it would work on my workstation.
 
FreeBSD-13.1-RELEASE has the same problem , I installed only nano mc vm-bhyve and bhyve-firmware and run kldload vmm.ko, new crash and only hard reboot possible.
 
wincc said:
FreeBSD-13.2-STABLE
Click to expand...
This is a development branch AS WELL (the one for pulling the 13.x releases from). It only gets stuff that's been tested on -CURRENT for a while, so, could be a compromise, but definitely another example that isn't recommended for a beginner. 13.2-RELEASE is just around the corner though, and upgrading from 13.1-RELEASE is really simple.
wincc said:
which has the same "problem".
Click to expand...
In case you have a problem with 13.1-RELEASE, make sure to give details please (what do you mean with "crash"? does the system reboot? does it "freeze"? Any kernel messages printed? ...)
 
System freeze, photos of output provided.
 

Attachments

  • Screenshot_2023-02-26_16-11-22.png
    Screenshot_2023-02-26_16-11-22.png
    999.2 KB · Views: 184
  • Screenshot_2023-02-26_16-11-47.png
    Screenshot_2023-02-26_16-11-47.png
    1 MB · Views: 189
  • Screenshot_2023-02-26_16-12-44.png
    Screenshot_2023-02-26_16-12-44.png
    1.2 MB · Views: 196
I'd repeat my statement to read that old Fred Cash article, explaining about CURRENT and STABLE. The names can be deceptive, especially STABLE.

An Explanation of FreeBSD Releases

srobb.net
To quote two paragraphs from that one.
NOTE: -STABLE refers to the programming API/ABI used by the kernel and base applications and libraries. It doesn't refer to the stability of the running code. IOW, developers can write applications and drivers for 6.0 and that same binary and driver will work on 6.9 (with occasional exceptions.) This does not mean that the code quality is poor, as it is not, but it is not the primary meaning of -STABLE.

This is contrary to the Linux world, where "stable" means stable running code and not stable programming APIs/ABIs (which can change on a weekly basis, even in releases.) Then there are the releases which are (in essence) just snapshots of the -STABLE tree at a specific point in time, that get a lot of testing, and are released to the public as a fully-usable OS. At time of writing, the latest releases are: 4.11, 5.4, and 6.0. The cvsup tags for these are:
RELENG_4_11_0_RELEASE, RELENG_5_4_0_RELEASE, and RELENG_6_0_0_RELEASE. These are what are on the installation CDs.
Click to expand...

As I said it was written a long time ago, but the way CURRENT and STABLE are labeled remains the same, though now it's svn, not cvsup.
 
wincc said:
I have run dmesg and found POPCNT processor feature flag on the Features2 line.
Click to expand...
Look for SVM support, not POPCNT (popcount):

vmm(4)
Rich (BB code): 
DESCRIPTION
     vmm.ko provides the kernel portion of the bhyve(4) hypervisor.

     An Intel CPU with VT-x/EPT or AMD CPU with SVM support is required.

Example AMD Ryzen 7 5700U:
Rich (BB code): 
  AMD Features2=0x75c237ff<LAHF,CMP,SVM,
 
AMD Ryzen Threadripper PRO 3955WX has support for SVM.

Anyone who can point me where to find very good instructions howto debug the loading of vmm.ko or otherwise test something I be happy to test that, I am a rookie on FreeBSD but I started my coding with MASM32 back in early 2000.
I have been running Windows ,MacOS and Linux since 1995....

When I post on furums to get support I would like to find the solution for others to use.


Rich (BB code): 
Linux threadripper 6.2.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 26 Feb 2023 03:39:23 +0000 x86_64 GNU/Linux

[root@threadripper Desktop]# dmesg | grep SVM
[   13.732014] SVM: TSC scaling supported
[   13.732019] SVM: kvm: Nested Paging enabled
[   13.732119] SVM: Virtual VMLOAD VMSAVE supported
[   13.732120] SVM: Virtual GIF supported
[   13.732120] SVM: LBR virtualization supported

Screenshot_2023-02-27_07-36-16.png
 
wincc said:
System freeze, photos of output provided.
Click to expand...
I have to admit I'm a bit confused by that as it seems to show two different crashes?

What other modules are loaded? (kldstat)
The second crash screenshot seems somehow related to drm/modeset?
 
wincc said:
Anyone who can point me where to find very good instructions howto debug the loading of vmm.ko
Click to expand...
Have a look at the FreeBSD Developers' Handbook, Chapter 10. Kernel Debugging.

wincc said:
or otherwise test something I be happy to test that, I am a rookie on FreeBSD
Click to expand...
I suggest you open a bug report (PR). FreeBSD developers are rarely here on forums and they prefer to deal that kind of issues on FreeBSD bugzilla.

Following PR shows how nicely users and developer interact: Bug 268246 - crash and panic using pfsync on 13.1-RELEASE

wincc said:
When I post on furums to get support I would like to find the solution for others to use.
Click to expand...
You can update this thread linking to your PR and add comments about progress and outcome.
 
zirias@ said:
I have to admit I'm a bit confused by that as it seems to show two different crashes?

What other modules are loaded? (kldstat)
The second crash screenshot seems somehow related to drm/modeset?
Click to expand...

I have taken screenshots from multiple crashes , in one of my installations I came to a promt where I run kldstat and vmm.ko was show at the end of list.
 
a panic with i915 kmod loaded will trigger a second fault
disable it while you debug the other problems
this happens on 13.1 too
 
covacat said:
a panic with i915 kmod loaded will trigger a second fault
disable it while you debug the other problems
this happens on 13.1 too
Click to expand...
There is no i915kms module loaded in my AMD workstation nor is the amdgpu module loaded, i915kms module is loaded in my intel based Samsung laptop where bhyve run as expected on FreeBSD 14 current.
 
Some more testing done:

# sysctl debug.kdb.panic=1 result in a dump file in /var/crash
# kldload vmm, just freeze the system, no output to /var/crash
 
Ok, now I have recompiled the kernel and "kldload vmm" followed by "dump" command results in a dump file.
Maybe it time for a bug report.
 
wincc said:
Nothing has happened with this bug for 7 weeks so I stick with Arch Linux & zfs filesystem.
Click to expand...
Well then. It certainly looks like people did have a look at that PR. Most likely, nobody could reproduce it so far. I certainly can't, using bhyve now for years on different FreeBSD versions... ?‍♂️
 
You must log in or register to reply here.
Back
Top