FreeBSD-10.2 on Dell T430 w PERC H730 card

[tfba]

I'm looking to install FreeBSD-10.2 on a Dell T430 with a PERC H730 RAID card. I'm particularly interested in using ZFS. I'm read various posts on the web about the PERC controllers. It looks like I'll need to enable the mrsas driver for this card.

Ideally I want to put the card in to non-RAID pass through and install ZFS as the root with some form of raidz.

If I add the the following to loader.conf:

hw.mfi.mrsas_enable=1
mrsas_load="yes"

Has anyone run FreeBSD-10.2 on similar hardware? Is the mrsas(4) driver stable enough? Are we better off using hardware raid 5/6?

I'm limited to using the given hardware but before I venture down the route of FreeBSD, I just wanted to check if this should be a stable setup?

Thanks

Russell

 

[SirDice]

Can't comment on this particular controller but the mrsas(4) driver has been working just fine for a couple of LSI MegaRAID cards.

 

[tfba]

Thanks for the info.

In case anyone has a Perc H730 with FreeBSD, it's http://serverfault.com/a/735748 that has me concerned as to whether it is a viable solution or if I should consider ZFS on hardware RAID (I really want to use ZFS in some form for the easy fs management and snapshot features).

 

[SirDice]

ZFS on hardware RAID is a bad idea. Use one or the other, not both.

 

[Terry_Kennedy]

In case anyone has a Perc H730 with FreeBSD, it's http://serverfault.com/a/735748 that has me concerned as to whether it is a viable solution or if I should consider ZFS on hardware RAID (I really want to use ZFS in some form for the easy FS management and snapshot features).

The information in the serverfault link isn't entirely accurate in two respects:

1) Setting mfip_load="yes" in /boot/loader.conf will expose a pass(4) device for each physical drive which works perfectly with sysutils/smartmontools.

2) Setting hw.mfi.allow_cam_disk_passthrough=1 will cause a da(4) device to appear for every physical disk, regardless of whether the controller thinks the disk is part of a volume or not. Obviously this should only be done with drives not allocated to any volume on the controller and there should be no references to any mfi(4) drives by their mfid0, etc. names.

 

[tfba]

Hi Terry, thanks for your reply. I successfully set up the server last night, by setting the controller in to non-raid mode and setting the hw.mfi.mrsas_enable=1 at install and in loader.conf. This exposed all drives as da0 - da3.

The hardware notes https://www.freebsd.org/releases/10.2R/hardware.html state that the Perc H730 is supported by the mrsas(4) driver but you're method is to use the mfi(4) driver.

What differences would I see with your setup vs the setup I have? Does it actually matter if I'm not planning on using the RAID functionality of the controller?

 

[Terry_Kennedy]

What differences would I see with your setup vs the setup I have? Does it actually matter if I'm not planning on using the RAID functionality of the controller?

A lot depends on the model (the real LSI model, not the Dell one) of controller. Apparently some controllers work only with mfi(4), some work only with mrsas(4), and some work with either (though possibly with problems on one or the other).

As long as your method lets you collect the SMART data from each drive, it sounds like you're good to go.

 

[tfba]

Thanks for the hints. I've installed sysutils/smartmontools unfortunately, I seem to be hit by the issue mentioned in the server fault link I posted (http://serverfault.com/a/735748) where the device drops out if I try and run

smartctl -a /dev/da0

After it receives some info from the device. This starts to make me nervous about the stability of the mrsas(4) driver. Rebooting makes the device appear again and ZFS successfully resilvers correctly so no dataloss but not encouraging non the less.

I've changed to use mfip and /dev/passN and get a warning

MegaRAID SAT layer is reportedly buggy, use '-d sat' to try anyhow

If I then use smartctl -a -d sat /dev/pass0. It does work and I get plenty of output but I get the following warnings (I've attached the full output).

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: ATA return descriptor not supported by controller firmware
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

I'm new to FreeBSD and was hoping to use ZFS raidz2 rather than hardware raid 6 on the controller but am I opening myself up to incompatibilities between FreeBSD and the Perc H730 and should I revert to raid6 or avoid FreeBSD altogether? I guess my options are:

1) raidz2 with mrsas(4) (but no smart) - Is the mrsas(4) driver stable given then smart drop out issue and is ZFS robust enough to not worry so much about smart?
2) raidz2 with mfip (smart warns about MegaRAID SAT layer as buggy but will generate a report)
3) HW Raid 6 with standard FS (I assume I need mrsas(4) or mfi(4) though still so above issues may still apply)
4) HW Raid 6 with ZFS (I'd prefer this to 3 because of the snapshot and FS creation features etc

I could switch to Ubuntu but in my limited experience I prefer FreeBSD. Our use is only version control hosting (Git/Subversion), Apache, MySQL and file serving for about a dozen users so nothing heavy, so robustness is our main concern.

Sorry for the long post, any advice greatly appreciated.

 

[Terry_Kennedy]

1) raidz2 with mrsas(4) (but no smart) - Is the mrsas(4) driver stable given then smart drop out issue and is ZFS robust enough to not worry so much about smart?

I haven't used mrsas(4) so I can't say. I'd like to have the SMART data available as most of the drive failures I've seen are detected by SMART before the operating system sees any problems. For example, non-zero SMART values for "current pending sector count" or "offline uncorrectable sector count" won't show up as errors in a # zpool scrub unless one of those sectors is in allocated data (as opposed to unused space). Most remaining drive faults are "sudden death" where SMART doesn't help (at best, you'll get a "You should replace this drive soon" warning a few minutes before the disk dies).

2) raidz2 with mfip (smart warns about MegaRAID SAT layer as buggy but will generate a report)

You didn't say you had SATA drives. This seems to be a new-out-of-the-box system from Dell, based on the 55 hours on the drive. I'm surprised to see a 3-year-old SATA drive model supplied by Dell on a high-end controller like the H730, but I just checked the Dell configurator and it really does default to that drive.

Anyway, the "MegaRAID SAT layer is reportedly buggy" error was added to sysutils/smartmontools a long time ago because some revision of some MegaRAID card had problems in its SAS-to-SATA translation code. The idea was to warn users that there might be a problem instead of just auto-scanning that configuration.

I have a number of systems running ZFS on older (PERC 6) mfi(4) controllers using sysutils/smartmontools and I do receive notifications of drive faults.

3) HW Raid 6 with standard FS (I assume I need mrsas(4) or mfi(4) though still so above issues may still apply)
4) HW Raid 6 with ZFS (I'd prefer this to 3 because of the snapshot and FS creation features etc

If you're going to use ZFS, you don't want any hardware RAID involved. You want ZFS to handle parity, etc. - otherwise the controller's RAID is going to "fix" things behind the scenes and ZFS will never know about them. Personally, I don't have a problem with creating single-drive volumes on mfi(4) (and other) controllers and using them for ZFS (based on > 5 years of operation on dozens of systems with a total of around 1.5PB of storage). If real passthru / IT mode is available, I'd prefer to use that, but it isn't always available. Others may disagree. I could go into the "what and why", but it isn't really relevant to the main point here.

 

[tfba]

Thanks Terry. So it looks like I'm getting direct passthrough from the H730 with mrsas and mfip but with mfip I can access the smart information as an extra to zpool scrub (I understand it only checks the used blocks). It is a new system. The hours on it so far have been me filling up the zpool (from /dev/urandom) with 1 thread and then 8 threads and scrubbing the pool to try and force drive errors/system errors.

Both done with the mrsas(4) driver but now I have smart working through mfip, I'll retry with that.

Thanks for your help. Looks like I'll stick with the zfs raidz2 setup but move to mfip for now.