HP Gen9 arrays support

We have purchased DL360 Gen9 server (K8N32A) with H240ar HBA and 2 HDD.
With the installation process of OS (FreeBSD 10.2) some difficulties has arisen, but not where initially expected.
The first problem was the error:
panic: APIC: CPU with APIC ID 0 is not enabled
In the Internet I found that to eliminate this the need to disable in BIOS the CPU option "x2APIC".

It's strange, but on this occasion there are only a few references:
"After the first boot, the system would panic with the message "panic: APIC: CPU with APIC ID 0 is not enabled".
The kernel refused to start until x2APIC has been disabled in the BIOS.
Alternative set hint.apic.0.disabled=1 at the loader prompt before booting the kernel."

I had to go to the data-center instead of installing the OS through ILO. But the biggest problem was in the other.
The OS installation from CD clearly sees the controller H240ar (HBA mode), HDD and built-in network card. Everything is fine installed.
But!o_O After rebooting the server does not recognize that the OS on disk is installed!
It does not depend on the type of installation (from used file system) or available boot settings in the BIOS. (By the way, there was set the mode "Legacy", because I would like to use the "root on ZFS".)

Anybody has experience how to solve this problem?
As far as I can see by the snippets of information in searching, in General on HP Gen9 it is possible to install FreeBSD. But I cannot find how to solve my specific problem.
I was advised to use in the disk controller mode "Smart Array" instead of the HBA.
Indeed, in this mode possible to create a logical drive by creating a RAID1, and I was able to install the OS.:)
I couldn't create in this mode ZFS-mirror, only ZFS-sripe. I hope is that RAID1+ZFS-sripe can still provide protection against failure of one disk.
Most likely, in this combination the speed of reading data will be significantly worse. It is not clear how this will be read simultaneously the controller and ZFS. But this can adequately estimate only experts in ZFS.

By the way, surprise! Despite the fact that H240ar is built-in controller, it has 2GB cache with battery.
Again, it's not obvious what to do with it. Is it useful for this case with ZFS file system or not.
Can anyone give a reliable recommendation?

Can anyone tell what controllers of HP Proliant Gen9 series servers allow to create a ZFS-mirror?
Recently we have purchased DL360 Gen9 server (K8N32A) with integrated disk controller HP H240ar and 2 HDD.
I just installed FreeBSD 10.2 on this server. But I could not realize bootable ZFS-mirror in HBA mode of H240ar, only RAID1+ZFS-sripe.
HBA mode of H240ar absolutely does not provide any system boot (on UFS too) for FreeBSD. (Tested in "Legacy" BIOS mode, because the UEFI does not provide "root on ZFS".)

So, the question: is it possible to do it on any other HP disk controllers?
Can someone give examples and recommendations?
What settings of the controller you used?

Second, I suppose that the speed of reading data may be significantly worse in combination of RAID1+ZFS-sripe compared to ZFS-mirror. Is this true?
It is not clear how this will be read simultaneously by the controller and ZFS.

Also, question about using cache with battery on disk controllers.
It's not obvious what to do with it. Is it useful for this case with ZFS file system or not?
I know that ZFS doesn't need this. But it is already there, so what to do with this?
General summary about installing FreeBSD on servers HP ProLiant Gen9 is next.

Embedded controllers are supported properly, this is no problem (although the software RAIDs of "Bxxi" series is still better not to use). The problem only occurs if you want to use FreeBSD + ZFS.
Disk controllers of HP Gen9 series servers do not support the "clear HBA mode", it specifically modified by the complex logic of these controllers, which can produce some operations that are in conflict with the work of ZFS.
In addition, they do not support OS booting at HBA mode, since they provided booting only from "logical drives", and they are created only in a mode "Smart array" (RAID).
By the way, in the previous series of HP Gen8 servers were controllers, most likely supporting a "clear HBA mode" (B220,H221,H222). But, overall, they are also not completely reliable for use in combination with FreeBSD + ZFS.

If you want to use ZFS, you must to do careful selection of disk controllers from already tested with ZFS file system.
Their list can be found here:

ZFS is strictly incompatible with any variants of the "hardware controllers" and hardware caching. And be aware that most manufacturers do not warn in the documentation that HBA-mode, implemented in their controllers, is not really "clear HBA mode", but modified. It allow intervention of complex logic of the controller in disk operations in HBA-mode.
"ZFS can not fully protect the user's data when using a hardware RAID controller, as it is not able to perform the automatic self-healing unless it controls the redundancy of the disks and data. ZFS prefers direct, exclusive access to the disks, with nothing in between that interferes. If the user insists on using hardware-level RAID, the controller should be configured as JBOD mode (i.e. turn off RAID-functionality) for ZFS to be able to guarantee data integrity.
The problem is that if the underlying RAID controller detects an error it will try to do its own recovery of the data. If it's unable to do so it will declare the volume dead and it won't permit the ZFS file system to access the data in order to recover at that level.
If a drive dies (the error), you replace it and the RAID controller rebuilds the set (the recovery). But if, during the course of the rebuild, it can't read required data from one of the other surviving disks then as far as the controller is concerned the whole volume is toast.
All of this takes place below the ZFS level - the ZFS file system has no ability to control or recover from this process."

"One might be inclined to try using single-drive RAID 0 arrays to try to use a RAID controller like a HBA, but this is not recommended for many of the reasons listed for other hardware RAID types. It is best to use a HBA instead of a RAID controller, for both performance and reliability."

With this in mind, I have to postpone the transition to ZFS for a while.
HP Gen9 doesn't have integrated SmartArray controller. So yours H240ar is not integrated, it's PCI-E card plugged to mainboard by special PCI-E connector (just for 'ar' cards). In earlier Proliant machines - I mean DL360p Gen8, DL360G7, DL360G6 - SmartArray was soldered at mainboard.
In the UEFI setup there's a SmartArray configuration utility, where it will allow you to switch the controller into IT mode. It will then act as a plain SAS HBA without RAID.
  • Thanks
Reactions: smj
In the UEFI setup there's a SmartArray configuration utility, where it will allow you to switch the controller into IT mode. It will then act as a plain SAS HBA without RAID.
That would be welcome news. In the DL360/380 Gen 5 and Gen 6 servers I've been working with, I installed LSI-based add-in controllers and bypassed the built-in controllers that Jarek6 mentioned. Perhaps when I finally outgrow the Gen 6 hardware I'll be able to take advantage of this...
Some additions on this topic.
I was able to install FreeBSD 10.2 with root on ZFS-mirror in HBA mode of H240ar controller (in "Legacy BIOS boot mode" and UEFI mode).
This was possible after upgrading the server BIOS.
But I decided not to risk using this solution, because recently I read about frequent problems with root on ZFS mirror.
In the future I plan to use ZFS with a mirrored root on a couple of USB flashes.
Also the use of separate LSI HBA-controllers preferably for ZFS. Because it's a more understandable and proved thing.
Quick update to this thread.

Putting the P440ar controller in "HBA Mode" worked on the DL380 G9 with 2015 BIOS, via the ciss(4) driver, and continued working after updating BIOS+firmware using the 2021.05 Service Pack for Proliant G9/G10. All this with FreeBSD 13.0-RELEASE and a RAIDZ2 zroot, FWIW.

However with the EFI option ROMs loaded on LSI SAS2008-based cards, I can also pull the P440ar and run off the third-party controllers as expected.