ZFS freebsd 13.1, disks ordering is diffe rent from the bay

B

benoitc

I recently moved all disk in the bay and to add 2 SSD at the beginning of the bays.

When I list disks ut return me:
Code: 
camcontrol devlist
<ATA MB004000GWFWB HPG1>           at scbus1 target 0 lun 0 (pass0,da0)
<ATA MB004000GWFWB HPG1>           at scbus1 target 1 lun 0 (pass1,da1)
<ATA MB004000GWFWB HPG1>           at scbus1 target 2 lun 0 (pass2,da2)
<ATA MB004000GWFWB HPG1>           at scbus1 target 3 lun 0 (pass3,da3)
<ATA LK0200GEYMR HPG1>             at scbus1 target 4 lun 0 (pass4,da4)
<ATA LK0200GEYMR HPG1>             at scbus1 target 5 lun 0 (pass5,da5)
<ATA MB004000GWFWB HPG1>           at scbus1 target 6 lun 0 (pass6,da6)
<ATA MB004000GWFWB HPG1>           at scbus1 target 7 lun 0 (pass7,da7)
<ATA MK0480GFDKR HPG0>             at scbus1 target 8 lun 0 (pass8,da8)
<ATA MK0480GFDKR HPG0>             at scbus1 target 9 lun 0 (pass9,da9)

But it should be:

Code: 
camcontrol devlist
<ATA LK0200GEYMR HPG1>             at scbus1 target 0 lun 0 (pass4,da4)
<ATA LK0200GEYMR HPG1>             at scbus1 target 1 lun 0 (pass5,da5)
<ATA MB004000GWFWB HPG1>           at scbus1 target 2 lun 0 (pass0,da0)
<ATA MB004000GWFWB HPG1>           at scbus1 target 3 lun 0 (pass1,da1)
<ATA MB004000GWFWB HPG1>           at scbus1 target 4 lun 0 (pass2,da2)
<ATA MB004000GWFWB HPG1>           at scbus1 target 5 lun 0 (pass3,da3)
<ATA MB004000GWFWB HPG1>           at scbus1 target 6 lun 0 (pass6,da6)
<ATA MB004000GWFWB HPG1>           at scbus1 target 7 lun 0 (pass7,da7)
<ATA MK0480GFDKR HPG0>             at scbus1 target 8 lun 0 (pass8,da8)
<ATA MK0480GFDKR HPG0>             at scbus1 target 9 lun 0 (pass9,da9)

Ie the 2 SSD are in the slots 1 and 2.

Is there a way to fix it? The main goal is to creata zfs 1/0 pool with last the 6 HDD MB004000GWFWB, and use the first 2 SSD LK0200GEYMR as private device (metadata).
 
Use labels to refer to the disks, then the actual device name doesn't matter any more. Disks often move around when you add or remove them.
 
camcontrol only lists drive in the order they appeared - i.e. if they take longer to respond at boot or if they have been added during runtime, they are appended with the next free ID. This is in no way related to any physical location of a disk.

For a physical mapping of disks you have to use sesutil:
Code: 
# sesutil show
ses0: <LSI SAS3x28 0501>; ID: 500304800189ee3f
Desc     Dev     Model                     Ident                Size/Status
Slot00   -       -                         -                    Not Installed
Slot01   -       -                         -                    Not Installed
Slot02   -       -                         -                    Not Installed
Slot03   da0     SanDisk DDYE1920S5xnNMRI  A03AB4A3             2T
Slot04   da1     SanDisk DDYE1920S5xnNMRI  A040672D             2T
Slot05   da2     SanDisk DDYE1920S5xnNMRI  A03AC4E2             2T
Slot06   da3     SanDisk DDYE1920S5xnNMRI  A0417FEC             2T
Slot07   da4     SanDisk DDYE1920S5xnNMRI  A03AC901             2T
Slot08   da5     SanDisk DDYE1920S5xnNMRI  A03AC724             2T
Slot09   -       -                         -                    Not Installed
Slot10   -       -                         -                    Not Installed
Slot11   -       -                         -                    Not Installed

Temperatures: BackPlaneLM75: 30 C, ChipDie: 60 C
Voltages: 5V: 5.09 V, 12V: 12.00 V

Depending on the backplane and controller you are using, and how the backplane is connected to the/which controller, this might not work correctly. Then you have to manually map the drive via its SAS address:
Code: 
# smartctl -x /dev/da0 | grep -m1 'SAS address'
    SAS address = 0x5002538b48c70fe2
# mprutil show devices
B____T    SAS Address      Handle  Parent    Device        Speed Enc  Slot  Wdt
00   03   5001173100759ff9 0009    0001      SAS Target    6.0   0001 03    1
00   02   5001173100750f11 000a    0002      SAS Target    6.0   0001 02    1
00   01   500117310108d179 000b    0004      SAS Target    12    0001 01    1
00   00   5002538b48c70fe2 000c    0003      SAS Target    12    0001 00    1
-> da0 is in slot 03

I've encountered HBAs/backplanes that count from left to right, some from right to left, some from top to bottom and some from bottom to top and any combination of those... So either you verify that first, or (much more convenient) just label the caddy with the appropriate data you want. I usually put the last 6 digits of the serial number on the caddy and also use that for the gpt labels (e.g. "8c70fe0-zfs", "8c70fe0-swap" etc...).
if you put kern.geom.label.disk_ident.enable="0" and kern.geom.label.gptid.enable="0" in you /boot/loader.conf the gpt labels will be used (if present) to represent the drives, e.g. with zpool status, which makes it easier to quickly identify the corresponding drive.
 
Thank! I manage to use gpt id for all disk but the thing I don't understand is why sesutil show doesn't return any disk. Where target is scbus?

Code: 
root@pistil1:~ # camcontrol devlist
<ATA MB004000GWFWB HPG1>           at scbus1 target 0 lun 0 (pass0,da0)
<ATA MB004000GWFWB HPG1>           at scbus1 target 1 lun 0 (pass1,da1)
<ATA LK0200GEYMR HPG1>             at scbus1 target 2 lun 0 (pass2,da2)
<ATA LK0200GEYMR HPG1>             at scbus1 target 3 lun 0 (pass3,da3)
<ATA MB004000GWFWB HPG1>           at scbus1 target 4 lun 0 (pass4,da4)
<ATA MB004000GWFWB HPG1>           at scbus1 target 5 lun 0 (pass5,da5)
<ATA MB004000GWFWB HPG1>           at scbus1 target 6 lun 0 (pass6,da6)
<ATA MB004000GWFWB HPG1>           at scbus1 target 7 lun 0 (pass7,da7)
<ATA MK0480GFDKR HPG0>             at scbus1 target 8 lun 0 (pass8,da8)
<ATA MK0480GFDKR HPG0>             at scbus1 target 9 lun 0 (pass9,da9)
root@pistil1:~ # sesutil show
sesutil: No SES devices found
 
Last edited by a moderator:
It seems you don't have any ses devices in your system. Are those disks attached to a backplane or all directly attached to a controller/mainboard?
 
they are attached to a backplane HPE Smart Array P840ar Controller in LBA mode. Maybe it's not compatible?
 
Most of these SES cage devices are tied in via i2C cable. Is this a factory build with cable?
Does ls /dev|grep i2c show anything?
Sometimes the SAS cable has the sideband signaling embedded. Is this a stock HP machine?
 
Is it smartpqi driver? IIRC, for some weird reason, the built-in SES device (i.e. emulating the "enclosure" for direct-attached disks via the fanout cable) is filtered in the driver (and don't get me started on the code style in there).

EDIT: looks like it was fixed in smartpqi, and that P840ar definitely looks like something older than current HPE HBAs based on Microchip SmartHBA's; sorry for the noise.
 
Phishfry said:
Most of these SES cage devices are tied in via i2C cable. Is this a factory build with cable?
Does ls /dev|grep i2c show anything?
Sometimes the SAS cable has the sideband signaling embedded. Is this a stock HP machine?
Click to expand...
yes it's a full gen9 dl380. ALl HW looks original, though since it's refurbished that may explain... ls /dev|grep i2c return nothing....
 
No it could be hidden as a system service.
Not a useland toy. I figured it is worth looking.
Not all backplanes use i2c signaling.
 
You must log in or register to reply here.
Back
Top