Static Device IDs for SATA Disks on mpt

T

TheSwissKnife

I have searched around to try to work out if there is a way to get my SATA drives use static device references, without any joy. Am hoping someone will point me in the right direction.

I have purchased an LSI 3081E-R PCI SAS/SATA card (using the LSI 1068 chip) - because I have been struggling to find a suitable FreeBSD compliant SATA solution for 8 disks. This card is recognised by the mpt driver without problem but presents to the da driver using dynamically allocated device ids (eg da0-da7). Thus if I lose a drive and reboot all devices will be renamed messing up anything that references via these paths. I have tried FreeBSD 7.3 and 8.1RC2.

I am not sure if the card is AHCI compliant - but since I can find nothing that states it is I assume not), hot-plug does not appears to work either (another annoyance).

I tried the same setup with OpenSolaris and all works fine with devices using the actual card ids which start at 4 going through to 11, and hot-plug works too.

Any suggestions for FreeBSD?
 
I suggest you use labels Either when you create new gpt partition you can add -l label option
or use geom_label

This way, no matter what, each HDD will have exactly same label, that can be used as normal device name.

So for example in fstab you won't write /dev/ad3p2 but rather /dev/disk3p2 or something, that you want
Using this for zfs I name my disks like
r0d0z << (raid 0, disk 0, zfs partition)
 
Use glabel(8) to label the drives. Then use the /dev/label/<name> devices instead of the /dev/da* nodes. That way, it doesn't matter which port you connect the drive to, the label will always be the same.
 
Thanks for the quick replies. I haven't heard of labels...so it seems there is two ways to 'label' a drive - I will have a play with that. Using this method would I still be able to boot another OS and import the zpools that I will add the disks to?

Does anyone know for sure that there is no way to have paths based on the hardware position - as I am more comfortable with that if it is possible. I am also not sure how FreeNAS webgui will allow me to manage via labels, so I may have to give that up or tweak it - that's another story of course.

Any idea about the hot plug? - at present I don't know of any command that will successfully rescan the drive after re-insertion.
 
TheSwissKnife said:
Thanks for the quick replies. I haven't heard of labels...so it seems there is two ways to 'label' a drive - I will have a play with that. Using this method would I still be able to boot another OS and import the zpools that I will add the disks to?
Click to expand...

zfs uses its own, similar labelling schema: The geom(8) system "tastes" each disk (& each slice on each disk (& each partition in each slice)) for geom labels of various sorts, & uses those data to create the proper /dev/ entries. zfs does a similar thing, in that its labels tell it what pool (& cet) to place each unit, regardless of /dev/ entries, there is no need to externally label slices or partitions or disks which you are handing off to zfs.
 
I know that zfs adds an internal label but I still need to do things like apply smartctl commands to disks and physically label disks so that I know which to pull etc. This was ok with the normal ATA controllers I tried where disks went via 'ad' driver but this mpt to da breaks this.
 
Just had a look at this now. I have to re-partition all my disks to GPT presumably losing all the data. Not something I can play with quite yet as I would prefer not to have to restore all the data.
 
No, if you have UFS you can use # tunefs -L labelname /dev/ad*** on an unmounted filesystem (preferrably in single user mode). But that has literally no effect on camcontrol, smartctl and all that, since those work on the physical drives, not the labels or slices.

I kinda long for the old days of jumpering the drives on SCSI: unless you changed the jumper or moved the drive to a different controller, bus0, target0, lun0 was always itself.
 
Precisely....that is what you get with OpenSolaris c#t#d# based devices, static and persistent.

I may end up returning there - but I really wanted to use the FreeNAS all-in-one management GUI.
 
You can do this in FreeBSD as well, via hint. entries in /boot/loader.conf. I originally used this on our storage servers, so that no matter what drive was put into a bay, it would always show up as the same device. However, using labels is just so much nicer.

See the scbus(4) man page for details.
 
Ah, that looks like the ticket (albeit a bit too hidden away) - thanks again. I will look into trying these scbus wiring hints. That should hopefully solve that issue. I would like to try to use labels for their pros but I need first to see how I would then integrate the S.M.A.R.T. management and the disk power management (which incidentally is a right pain at present since I can't see how to get all disks to spin back up in parallel when required after they have spun down - it seems to take an age to get through 8 disks synchronously in sequence).
 
for ufs you can also use /dev/ufsid/3546813456 these are created when you formate ufs, and they don't change untill you format ufs again

or you can label, partitions/labels (depending on scheme) when you format them with newfs by adding -L label
in this case you can use /dev/ufs/label
 
I have spent a while looking at the options and am happy with the hard wiring hints method, but I also like the idea of using labels anyway. I am having problems understanding how the glabel command should be used/work and how it interacts with gpart (especially when labelling disks rather than partitions).

If I take a disk and try to label it (not labelling partitions) via "glabel label" persistent labelling command it does not create a /dev/label/<label> device entry and glabel list will therefore also not list it, but I can use glabel dump <disk> to display the label. If I use "glabel create" temporary labelling command it does create the /dev/label/<label> entry and glabel list will list it as expected. This entry will not survive reboot. The only real purpose for disk labelling requires having device entries that are named according the label, so how exactly is this supposed to be employed? Am I supposed to write a startup script that scans disks for these labels and then runs "gpt create" for each at boot time?

Also if I partition up a disk using gpart (create and add) and then use "glabel create" I end up with "corrupt GPT secondary table" errors and can only fix by using undocumented "gpt recover" command. Does the partition table have to avoid some specific parts of the disk to prevent this?
 
Edit. I just realised I should have written in the last paragraph "glabel label".


Also if I partition up a disk using gpart (create and add) and then use "glabel label" I end up with "corrupt GPT secondary table" errors and can only fix by using undocumented "gpt recover" command. Does the partition table have to avoid some specific parts of the disk to prevent this?
Click to expand...
 
summary and to clear up confusiion

phoenix said:
Use glabel(8) to label the drives. Then use the /dev/label/<name> devices instead of the /dev/da* nodes. That way, it doesn't matter which port you connect the drive to, the label will always be the same.
Click to expand...

I think a moderator should clean up this thread. It is full of confusion and misunderstandings. I realize this thread is old, but I was looking for the command to use in FreeBSD to make a disk label (not a slice label), and ran into this thread, so it is still relevant.

What phoenix said above is clearly better than anything else suggested here, and implies the following:

* This does not need you to repartition or erase your disks (no using gpart).
* You do not need slices/partitions.
* You can give zfs the whole disk (which is a zfs best practice)

You just label the disk itself, and then when zfs is given the disk using the label, the information on the disk does not matter, and the label stays the same. You can then move the disks around at random from bay to bay, and zfs will not care, because it uses the label written to the disk, not some index number.

For example, on my system where disks seem to be numbered randomly, [where da5 is the top right bay, da7 next to the right, da1 is next to the right, etc. ( da5 da7 da1 ... )] I did something like this:
Code: 
glabel create da0 spare0
glabel create da12 spare1
glabel label tank1d1 da5
glabel label tank1d2 da7
glabel label tank1d3 da1
...
glabel label tank2d1 da13

zpool create tank raidz2 label/tank1d1 label/tank1d2 label/tank1d3 ... raidz2 label/tank2d1 ...
zpool add tank spare label/spare0 label/spare1

And the result looks like this:
Code: 
zpool status

        NAME               STATE     READ WRITE CKSUM
        tank               ONLINE       0     0     0
          raidz2           ONLINE       0     0     0
            label/tank1d1  ONLINE       0     0     0
            label/tank1d2  ONLINE       0     0     0
            label/tank1d3  ONLINE       0     0     0
            ...
          raidz2           ONLINE       0     0     0
            label/tank2d1  ONLINE       0     0     0
            ...


And since it is related, here is my script to make the disk lights blink, to identify them, since I couldn't find anything installed by default, like the "format" command to do a read test suggested in Solaris books. So I decided to use dd, and control it with a script:
Code: 
#!/bin/sh
#
echo "Blinks one disk with the given device name."

sleepy() {
    sleep $1
    if [ "$?" != "0" ]; then
        # if sleep ended with ctrl+c, stop completely
        exit 1
    fi
}
#input value, either a disk device ("/dev/da1", "da1", or "label/mydisk") or a number ("1" for /dev/da1)
diskId=$1

if [ "$diskId" = "" ]; then
    echo diskId is required
    echo "USAGE: $0 <diskId>"
    exit 1
fi

disk=${diskId}

if [ '!' -e "$disk" ]; then
    disk=/dev/${diskId}
fi
if [ '!' -e "$disk" ]; then
    disk=/dev/da${diskId}
fi
    
# repeat the whole process forever
while [ 1 = 1 ]; do
    echo disk $disk
    
    while [ 1 = 1 ]; do
        dd if=${disk} of=/dev/null bs=1k count=300 \
                >/dev/null 2>&1 &
        sleep 0.15
        if [ "$?" != "0" ]; then
            break
        fi
    done
done
 
peetaur said:
What phoenix said above is clearly better than anything else suggested here, and implies the following:

* This does not need you to repartition or erase your disks (no using gpart).
Click to expand...

The thing with glabel is that it writes data to the last sector on the disk, so I take that to mean it can overwrite data on your file system. I've never had that 100% clear tho, so if anyone with more experience or knowledge of the source code wants to chip in please do! I've steered clear of glabel for this reason.

* You do not need slices/partitions.
* You can give zfs the whole disk (which is a zfs best practice)
Click to expand...

I think the ZFS best practice "give the whole disk to ZFS" is basically that the IO scheduling assumes that each ZFS physical device is a single disk/spindle. If you create one large patition, that should work equally well as giving the whole disk IMHO. You may want to do this because you prefer to GPT your disks, also I've seen it mentioned that it can be a good idea to create a partition slightly smalled than the physical disks as if you ever need to replace a 2TB disk or whatever with another there is no guarantee they will have an identical number of sectors, and if less on the new disk you won't be able to use it (if you are using enterprise disks then you probably don't have to worry about this).

ta Andy.
 
You must log in or register to reply here.
Back
Top