Dell PowerEdge R540, cannot see disks

nerozero

nerozero

Hello,
Got the server for testing for couple of days, trying to install. Now facing issue - no disk visible

here is the camcontrol listdev:
Code: 
# camcontrol devlist
<Kingston DataTraveler 2.0 PMAP>   at scbus14 target 0 lun 0 (pass0,da0)

Here is what DMESG says:
Code: 
...
AVAGO MegaRAID SAS FreeBSD mrsas driver version: 07.709.04.00-fbsd
mfi0: <Invader> port 0x4000-0x40ff mem 0xab100000-0xab10ffff,0xab000000-0xab0fffff irq 34 at device 0.0 numa-domain 0 on pci5
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23
mfi0: FW MaxCmds = 928, limiting to 128
mfi0: MaxCmd = 928, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c03a0
...

The server is brand new, no raid was configured.
Hard disks:
- 2x sas 2TB HDD
- 2x SSD 223 GB

full mfi0 dmesg
Code: 
AVAGO MegaRAID SAS FreeBSD mrsas driver version: 07.709.04.00-fbsd
mfi0: <Invader> port 0x4000-0x40ff mem 0xab100000-0xab10ffff,0xab000000-0xab0fffff irq 34 at device 0.0 numa-domain 0 on pci5
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23
mfi0: FW MaxCmds = 928, limiting to 128
mfi0: MaxCmd = 928, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c03a0
mfi0: 335 (638491014s/0x0020/info) - Shutdown command received from host
mfi0: 336 (boot + 10s/0x0020/info) - Firmware initialization started (PCI ID 005d/1000/1f42/1028)
mfi0: 337 (boot + 10s/0x0020/info) - Firmware version 4.300.00-8352
mfi0: 338 (boot + 11s/0x0008/info) - Battery Present
mfi0: 339 (boot + 11s/0x0020/info) - Package version 25.5.6.0009
mfi0: 340 (boot + 11s/0x0020/info) - Board Revision A01
mfi0: 341 (boot + 15s/0x0008/info) - Battery temperature is normal
mfi0: 342 (boot + 15s/0x0008/info) - Current capacity of the battery is above threshold
mfi0: 343 (boot + 26s/0x0004/info) - Enclosure PD 20(c None/p1) communication restored
mfi0: 344 (boot + 26s/0x0002/info) - Inserted: Encl PD 20
mfi0: 345 (boot + 26s/0x0002/info) - Inserted: PD 20(c None/p1) Info: enclPd=20, scsiType=d, portMap=00, sasAddr=500056b32f681dfd,0000000000000000
mfi0: 346 (boot + 26s/0x0002/info) - Inserted: PD 00(e0x20/s0)
mfi0: 347 (boot + 26s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c500ce8afa51,0000000000000000
mfi0: 348 (boot + 47s/0x0020/info) - Boot Device reset, setting target ID as invalid
mfi0: 349 (boot + 47s/0x0020/info) - Controller operating temperature within normal range, full operation restored
mfi0: 350 (638491080s/0x0020/info) - Time established as 03/25/20 22:38:00; (47 seconds since power on)
mfi0: 351 (boot + 10s/0x0020/info) - Firmware initialization started (PCI ID 005d/1000/1f42/1028)
mfi0: 352 (boot + 10s/0x0020/info) - Firmware version 4.300.00-8352
mfi0: 353 (boot + 11s/0x0008/info) - Battery Present
mfi0: 354 (boot + 11s/0x0020/info) - Package version 25.5.6.0009
mfi0: 355 (boot + 11s/0x0020/info) - Board Revision A01
mfi0: 356 (boot + 15s/0x0008/info) - Battery temperature is normal
mfi0: 357 (boot + 15s/0x0008/info) - Battery started charging
mfi0: 358 (boot + 15s/0x0008/info) - Current capacity of the battery is above threshold
mfi0: 359 (boot + 27s/0x0004/info) - Enclosure PD 20(c None/p1) communication restored
mfi0: 360 (boot + 27s/0x0002/info) - Inserted: Encl PD 20
mfi0: 361 (boot + 27s/0x0002/info) - Inserted: PD 20(c None/p1) Info: enclPd=20, scsiType=d, portMap=00, sasAddr=500056b32f681dfd,0000000000000000
mfi0: 362 (boot + 27s/0x0002/info) - Inserted: PD 00(e0x20/s0)
mfi0: 363 (boot + 27s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=500056b32f681dc0,0000000000000000
mfi0: 364 (boot + 27s/0x0002/info) - Inserted: PD 01(e0x20/s1)
mfi0: 365 (boot + 27s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=500056b32f681dc1,0000000000000000
mfi0: 366 (boot + 27s/0x0002/info) - Inserted: PD 02(e0x20/s2)
mfi0: 367 (boot + 27s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c500a78949f9,0000000000000000
mfi0: 368 (boot + 27s/0x0002/info) - Inserted: PD 03(e0x20/s3)
mfi0: 369 (boot + 27s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c500a789bbf9,0000000000000000
mfi0: 370 (boot + 64s/0x0020/info) - Boot Device reset, setting target ID as invalid
mfi0: 371 (boot + 64s/0x0020/info) - Controller operating temperature within normal range, full operation restored
mfi0: 372 (642693016s/0x0020/info) - Time established as 05/13/20 13:50:16; (64 seconds since power on)
mfi0: 373 (642693017s/0x0020/info) - Patrol Read can't be started, as PDs are either not ONLINE, or are in a VD with an active process, or are in an excluded VD
mfi0: 374 (642693032s/0x0008/info) - Battery charge complete
mfi0: 375 (boot + 10s/0x0020/info) - Firmware initialization started (PCI ID 005d/1000/1f42/1028)
mfi0: 376 (boot + 10s/0x0020/info) - Firmware version 4.300.00-8352
mfi0: 377 (boot + 11s/0x0008/info) - Battery Present
mfi0: 378 (boot + 11s/0x0020/info) - Package version 25.5.6.0009
mfi0: 379 (boot + 11s/0x0020/info) - Board Revision A01
mfi0: 380 (boot + 15s/0x0008/info) - Battery temperature is normal
mfi0: 381 (boot + 15s/0x0008/info) - Current capacity of the battery is above threshold
mfi0: 382 (boot + 25s/0x0004/info) - Enclosure PD 20(c None/p1) communication restored
mfi0: 383 (boot + 25s/0x0002/info) - Inserted: Encl PD 20
mfi0: 384 (boot + 25s/0x0002/info) - Inserted: PD 20(c None/p1) Info: enclPd=20, scsiType=d, portMap=00, sasAddr=500056b32f681dfd,0000000000000000
mfi0: 385 (boot + 25s/0x0002/info) - Inserted: PD 00(e0x20/s0)
mfi0: 386 (boot + 25s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=500056b32f681dc0,0000000000000000
mfi0: 387 (boot + 25s/0x0002/info) - Inserted: PD 01(e0x20/s1)
mfi0: 388 (boot + 25s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=500056b32f681dc1,0000000000000000
mfi0: 389 (boot + 25s/0x0002/info) - Inserted: PD 02(e0x20/s2)
mfi0: 390 (boot + 25s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c500a78949f9,0000000000000000
mfi0: 391 (boot + 25s/0x0002/info) - Inserted: PD 03(e0x20/s3)
mfi0: 392 (boot + 25s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c500a789bbf9,0000000000000000
mfi0: 393 (boot + 10s/0x0020/info) - Firmware initialization started (PCI ID 005d/1000/1f42/1028)
mfi0: 394 (boot + 10s/0x0020/info) - Firmware version 4.300.00-8352
mfi0: 395 (boot + 11s/0x0008/info) - Battery Present
mfi0: 396 (boot + 11s/0x0020/info) - Package version 25.5.6.0009
mfi0: 397 (boot + 11s/0x0020/info) - Board Revision A01
mfi0: 398 (boot + 15s/0x0008/info) - Battery temperature is normal
mfi0: 399 (boot + 15s/0x0008/info) - Battery started charging
mfi0: 400 (boot + 15s/0x0008/info) - Current capacity of the battery is above threshold
mfi0: 401 (boot + 26s/0x0004/info) - Enclosure PD 20(c None/p1) communication restored
mfi0: 402 (boot + 26s/0x0002/info) - Inserted: Encl PD 20
mfi0: 403 (boot + 26s/0x0002/info) - Inserted: PD 20(c None/p1) Info: enclPd=20, scsiType=d, portMap=00, sasAddr=500056b32f681dfd,0000000000000000
mfi0: 404 (boot + 26s/0x0002/info) - Inserted: PD 00(e0x20/s0)
mfi0: 405 (boot + 26s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=500056b32f681dc0,0000000000000000
mfi0: 406 (boot + 26s/0x0002/info) - Inserted: PD 01(e0x20/s1)
mfi0: 407 (boot + 26s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=500056b32f681dc1,0000000000000000
mfi0: 408 (boot + 26s/0x0002/info) - Inserted: PD 02(e0x20/s2)
mfi0: 409 (boot + 26s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c500a78949f9,0000000000000000
mfi0: 410 (boot + 26s/0x0002/info) - Inserted: PD 03(e0x20/s3)
mfi0: 411 (boot + 26s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c500a789bbf9,0000000000000000
mfi0: 412 (boot + 80s/0x0008/info) - Battery charge complete
mfi0: 413 (boot + 178s/0x0020/info) - Boot Device reset, setting target ID as invalid
mfi0: 414 (boot + 178s/0x0020/info) - Controller operating temperature within normal range, full operation restored
mfi0: 415 (642693488s/0x0020/info) - Time established as 05/13/20 13:58:08; (178 seconds since power on)
mfi0: 416 (boot + 10s/0x0020/info) - Firmware initialization started (PCI ID 005d/1000/1f42/1028)
mfi0: 417 (boot + 10s/0x0020/info) - Firmware version 4.300.00-8352
mfi0: 418 (boot + 11s/0x0008/info) - Battery Present
mfi0: 419 (boot + 11s/0x0020/info) - Package version 25.5.6.0009
mfi0: 420 (boot + 11s/0x0020/info) - Board Revision A01
mfi0: 421 (boot + 15s/0x0008/info) - Battery temperature is normal
mfi0: 422 (boot + 15s/0x0008/info) - Current capacity of the battery is above threshold
mfi0: 423 (boot + 25s/0x0004/info) - Enclosure PD 20(c None/p1) communication restored
mfi0: 424 (boot + 25s/0x0002/info) - Inserted: Encl PD 20
mfi0: 425 (boot + 25s/0x0002/info) - Inserted: PD 20(c None/p1) Info: enclPd=20, scsiType=d, portMap=00, sasAddr=500056b32f681dfd,0000000000000000
mfi0: 426 (boot + 25s/0x0002/info) - Inserted: PD 00(e0x20/s0)
mfi0: 427 (boot + 25s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=500056b32f681dc0,0000000000000000
mfi0: 428 (boot + 25s/0x0002/info) - Inserted: PD 01(e0x20/s1)
mfi0: 429 (boot + 25s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=500056b32f681dc1,0000000000000000
mfi0: 430 (boot + 25s/0x0002/info) - Inserted: PD 02(e0x20/s2)
mfi0: 431 (boot + 25s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c500a78949f9,0000000000000000
mfi0: 432 (boot + 25s/0x0002/info) - Inserted: PD 03(e0x20/s3)
mfi0: 433 (boot + 25s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c500a789bbf9,0000000000000000
mfi0: 434 (boot + 95s/0x0020/info) - Boot Device reset, setting target ID as invalid
mfi0: 435 (boot + 95s/0x0020/info) - Controller operating temperature within normal range, full operation restored
mfi0: 436 (642693636s/0x0020/info) - Time established as 05/13/20 14:00:36; (95 seconds since power on)
mfi0: 437 (642694287s/0x0020/info) - Host driver is loaded and operational
 
... after reading some forums and datasheets...

need to use mfiutil to create JBOD drives:

list drives first:
Code: 
# mfiutil show drives
mfi0 Physical Drives:
 0 (  224G) ONLINE <MZ7LH240HAHQ0D3 HE57 serial=...> SATA E1:S0
 1 (  224G) ONLINE <MZ7LH240HAHQ0D3 HE57 serial=...> SATA E1:S1
 2 ( 1863G) ONLINE <SEAGATE ST2000NM0155 DT31 serial=...> SCSI-6 E1:S2
 3 ( 1863G) ONLINE <SEAGATE ST2000NM0155 DT31 serial=...> SCSI-6 E1:S3

use drive IDs to create jbods, for disk 0 ...
# mfiutil create jbod 0
....

now drives are seen under:
/dev/mfid0
/dev/mfid1

so far so good, will reply on progress...
 
You can probably also try the mrsas(4) driver for this card. The only downside is that you can't use mfiutil(8) any more. But sysutils/megacli does work, regardless if it's mfi(4) or mrsas(4).

Code: 
     Using /boot/device.hints (as mentioned below), the user can provide a
     preference for the mrsas driver to detect a MR-Fusion card instead of the
     mfi(4) driver.

           hw.mfi.mrsas_enable="1"
 
It's an entirely new driver that "fits" better in the way FreeBSD handles disks. It's more or less explained in the first couple of paragraphs:
Code: 
     The mrsas name is derived from the	phrase "MegaRAID SAS HBA", which is
     substantially different than the old "MegaRAID" Driver mfi(4) which does
     not connect targets to the	cam(4) layer and thus requires a new driver
     which attaches targets to the cam(4) layer.  Older	MegaRAID controllers
     are supported by mfi(4) and will not work with mrsas, but both the	mfi(4)
     and mrsas drivers can detect and manage the LSI MegaRAID SAS
     2208/2308/3008/3108 series	of controllers.

Older cards only work with mfi(4), some cards are supported by both mfi(4) and mrsas(4) and some newer cards are only supported by mrsas(4). I ran into that last category when we replaced a few servers with new hardware and the controllers weren't detected.
 
With some of the various LSI / Avago / Broadcom cards, you have a choice of different firmware versions. One firmware version will support RAID within the card; that is not recommended to be used with ZFS. There is another firmware version (called the IT firmware) which only supports JBOD mode, meaning every physical disk gets exposed to the OS immediately as a block device. For ZFS and other systems that do their own RAID, that's more convenient.

But: Re-flashing the cards with new firmware is not without risk. I used to have a cardboard box of cards that we had bricked that way. Several times we took that box to LSI offices, and the friendly engineers there un-bricked them for us (we had a particularly good connection with them, we weren't end-users of one or two cards, but were more in a co-development relationship as a very large customer). Also, if you install that firmware, you can't have any RAID for other disks, and that might be bad.
 
I've had a few LSI RAID cards (can't remember the exact types) that had a "global" JBOD setting. Switching that on basically turned the card in a plain, non-RAID, HBA and treated every disk as a single drive. If you have that option in the firmware that's probably a little safer to try instead of flashing it. But that too is an "all or nothing" setting, so you can't mix RAID and JBOD.
 
I don't experience any errors so far. Configured all disk as JBOD, force to 4k sector. ZFS performance is fantastic with ssd cache in addition.
BHyve nested windows server booting in seconds, nested ubuntu server booting like it was running already :)
still testing...

just a little note for those who will face need of windows guest machine. While creating a file system for it - force it to 4k block size.
zfs create -V64G -o volmode=dev -o volblocksize=4K -o compression=off zpool/vm-disk

Thank you so much for your help!
 
You must log in or register to reply here.
Back
Top