Dell R630 - X2Apic enabled fails boot

P

paw

Processor: E5-2650L V3
No Raidcard installed
BIOS: UEFI Mode
Firmware: all up to date

Issue:
When trying to install FreeBSD12.1 with X2APIC enabled in BIOS, the kernel gets in a mess and hangs (see attached)
With X2Apic turned off the OS boots just fine. However with X2APIC turned off I loose Hyper Threading abilities.

HardenedBSD does the same.

I've tried the following turnables:
Code: 
kern.cam.boot_delay="10000"
vfs.root_mount_always_wait=1

but neither work.

Debian boots fine

OmniOS fails with:
No SOF interrupts have been received this usb ehci hostcontroller is unusable

Bug report raised.
 

Attachments

  • USB3.jpg
    USB3.jpg
    212.2 KB · Views: 245
  • USB5.jpg
    USB5.jpg
    157.6 KB · Views: 217
  • Hardened USB.jpg
    Hardened USB.jpg
    157.2 KB · Views: 233
I've tried both via USB stick and with Virtual CD. No Luck.
I have successfully installed the OS without X2Apic, but if I boot to OS with X2APIC and with no USB devices it still throws the same error as above.
 
No luck.

Turn off X2APIC in bios
Boot into OS
Edit /boot/bootloader.conf
Turn X2APIC back on in bios
Pull out power cables
Turn on
Same issue.

-- Edit1
I just tested this on another R630 I have and same error.
This processor is: E5-2660 v3 - No RAID
-- Edit 2
Same issue with FreeBSD 13
It for sure doesn't like NetBSD - resulting in killing the iDrac
-- Edit3
tried using hw.usb.xhci.use_polling=1
Press 3 at loader
set hw.usb.xhci.use_polling=1
boot

from

freebsd-hardware - Re: Dell R630 unable to detect USB, can't boot

Re: Dell R630 unable to detect USB, can't boot. On 02/13/18 14:45, Hans Petter Selasky wrote: > On 02/04/18 23:47, Lee Brown wrote: >> Booting FreeBSD11.1-RELENG from a USB stick, loader works fine...
freebsd.1045724.x6.nabble.com
But still same error.

Workaround for now:
Turn off X2APIC but you will loose HyperThreading capabilities.

This is such a shame. I don't want to use Linux, I love BSD but it's looking like I'm going to have to operate without X2APIC

HardenedBSD - Hangs on usb
OpenBSD - Just fall out crashes
NetBSD - Kills the server iDRAC
FreeBSD - ahcich timeout on slot xx
OmniOS which gives the error: No SOF interrupts have been received this usb ehci hostcontroller is unusable
 
50/50 chance to blame the hardware & BIOS engineers as well as there might be a bug in FreeBSD. Without SMT you loose about 5-15% performance on average -- not optimal, but bearable. Watch out for latest BIOS updates. At least you're not vulnerable to these sophisticated timing attacks w/o SMT. So for services dealing with encryption, disabling SMT is a good idea anyway.
 
Sadly with my focus on bHyve and especially as my product utilizes bHyve , it's quite a big cost in performance for myself.

Having two servers which are not playing nicely on BSD, is a major disappoint. My Dell R610 has had no issues.
Why would these work flawlessy with Linux unless it's foul play with Dell? All BSD variants do the same thing in that they freeze, including OpenSolarias. And no issues with Linux.

Not to blame BSD here, but shrug, just really disappointed.
 
paw said:
Why would these work flawlessy with Linux unless it's foul play with Dell? All BSD variants do the same thing in that they freeze, including OpenSolarias. And no issues with Linux.
Click to expand...
Linux has much more developers & wider access to hardware to test on. Thus there are more tweaks in Linux to work around hardware & firmware bugs. Major hardware vendors test & ship their servers with Linux nowadays, but they do not test on FreeBSD. Nevertheless, it could still be a bug in the BSDs. Consider to file in a bug report (see @top of this forum). EDIT: If you've ever read a technical spec (about some protocol or hardware device like X2APIC), you'll agree that too often they leave room for interpretation. Looks like the BSD developers all interpreted this in a similar way.
 
Thanks mjollnir,
I have opened a bug report. I am even happy to lend the server for testing if that would help the process.

I'll just have to assume that this may be fixed in the future. And that I should just live with the workaround for the time being. No reply from anyone apart from yourself from within FreeBSD to even acknowledged it's an issue, it's unbelievable.

It just concerns me that, if I purchase newer generation hardware I will end up at the same issue. Dell R630 is generation 13 of Dell which was manufactured in 2014, it now being 2020 and with this issue feels bad.
 
Just tried:

hw.usb.ehci.lostintrbug=1
hw.usb.ehci.iaadbug=1
hw.usb.ehci.no_hs=1

and set
hw.usb.ehci.debug to 7

ehci_roothub_exec: type=0xa3 request=0x00 wlen=0x0004 wIndex=0x0002

ahchi1: timeout on slot 8 port 0
ahcich1: is 0000002 cs 000000 ss 00000000 rs 0000800 tfd 50 serr 0000000 cmd 0004cb17
 

Attachments

  • moarerror.jpg
    moarerror.jpg
    205.9 KB · Views: 164
paw said:
I'll just have to assume that this may be fixed in the future. And that I should just live with the workaround for the time being. No reply from anyone apart from yourself from within FreeBSD to even acknowledged it's an issue, it's unbelievable.
Click to expand...
The developers use their time to work & not waste time reading through the forum ;) (some threads are really silly -- not this one). So if none of the users here ran into that same or similar issue, s/he can not help you much except suggesting to file a bug report... The developers can only test on the hardware they have, and that is a small fraction of all that's available. EDIT And they can can only fix bugs (or implement tweaks to work around broken hardware & BIOS/firmware) they know about.
 
paw said:
When trying to install FreeBSD12.1 with X2APIC enabled in BIOS, the kernel gets in a mess and hangs (see attached)
With X2Apic turned off the OS boots just fine. However with X2APIC turned off I loose Hyper Threading abilities.
Click to expand...

Yeah, same problem here with the Dell PowerEdge R730xd servers (which I think use basically the same motherboard as the R630).
X2APIC can't be enabled with FreeBSD on those machines.

It works on the Dell R740xd servers though.
 
Thanks Peter,
I wonder if it's safe to take a chance on Dell R640's if the Dell R740 work.

I've also noticed that ZFS preformance on FreeBSD compared to Linux is a lot slower when using the Dell R630.
FreeBSD gives me a write speed of 200Mb/s where as Linux in the same RAIDZ-2 gives me the full 500MB/s

So to anyone thinking of buying Dell R630's. Don't. they are kind of broken outside of the Linux world.
 
You must log in or register to reply here.
Back
Top