odd daX device mapping in FreeBSD 9.1 RC1

K

kangaroo

I have a Dell R720 that has 2 LSI 9205-8e cards, each connected to half of an MD1220 - a 24 x 2.5" SAS JBOD.

I'm running the 9.1 RC1 release, and see that the target mapping by CAM seems to be rather weird. Here's the map that I've created:

Code: 
SLOT  DAx   TARGET      SLOT DAx  TARGET
0     da1   34           12  da21 29
1     da2   35           13  da20 28
2     da3   36           14  da15 23
3     da4   37           15  da14 22
4     da5   38           16  da13 21
5     da9   33           17  da12 20 
6     da6   39           18  da18 26
7     da7   40           19  da19 27
8     da8   41           20  da22 30
9     da11  44           21  da23 31
10    da10  43           22  da16 24
11    da9   42           23  da17 25

If I reboot multiple times, I get the same mapping.

Given that the devices are scbus0 and scbus1, I would expect the targets to start at target 0 and go up to 23 like they show up in 9-current, and like they show up in the LSI BIOS, but they don't. scbus0 goes from target 33 to 45, and scbus1 goes from target 20 to
32. In addition, the mapping is all over the place! Now, albeit, I was able to hard-code da0 through da23 in /boot/device.hints using the targets specified by using camcontrol devlist, and sure enough, now, da0 applies to disk 0, all the way up to da23 applying to da23. In addition, I booted the system with every second disk removed, and the remaining disks did get the proper IDs. In fact, I inserted the disks that I had removed live, and they also got the right name... just why the weird targets!? I highly doubt these targets are supposed to be dependent on the FreeBSD release version!

Jason.
 
@kangaroo

When you start using labels to identify the hard drives, you get rid of these headaches:)

If you don´t have any hardware related demands for what drive is named da0 etc. you will benefit greatly from it. Then you can call(label) them to something even more logical, like "shelf1row1disk1" instead.

/Sebulon
 
labels

Hi Sebulon,

I've used labels in the past, and I can appreciate that with a complicated multi-jbod setup, there may be some benefit to using them, but really, they are an additional layer of complexity that I'd rather avoid if I can. Right now, with one jbod, I'd prefer to know that if da0 goes, we're talking the disk in slot 0, and if da23 goes, we're talking the disk in slot 23, and really, there's no valid reason for this strange target mapping. In one server where I was experimenting with using labels (glabel), sometimes I boot the server, and it thinks a disk is "missing". If I offline it and online it, everything is perfectly fine because the label is is there in /dev/label, but if I wasn't using labels, everything would be fine as well! A lot of people use GPT labels instead of glabel, and all over the place you find a scattering of people saying that one is better than the other, but when it comes to GPT labels, they add an ever further complexity to adding a disk -- partition before label. When it comes to an array setup, I'd rather just keep things as simple as possible.

Jason.
 
I can see the difficulty of connecting a label to a drive. Some RAID systems have an option to identify a given disk by turning on its LED. With direct access to a disk, that could be done with some long-running job that reads from the problem disk, like diskinfo -tv /dev/gpt/bob.

Avoid glabel(8) if you can. It creates a separate device that people don't seem inclined to use. GPT labels don't have that problem.
 
Labels really are the solution to this issue.

What happens when you do a needed BIOS update, and the default probe order changes, meaning PCI slot 0 is now PCI slot 6, so your controllers are mapped in reverse order, and all your disks are mapped in semi-reverse order? Happened to us twice already.

What happens when you do a kernel update, and the default probe order changes? Happened to use a couple of times with Debian; it's not impossible with FreeBSD.

What happens when you decide to compile the driver into the kernel, and it changes the probe order? There's a thread on the mailing lists for this very issue.

If you want to guarantee that "disk 0" is always probed as the disk in "bay 0", then you have to label it. Either that, or you have to spend a lot of time wiring down SCSI buses, slots, LUNs via device.hints support in /boot/loader.conf, and even that is dependent on the buses not being renumbered.
 
Argh. I "guess" labels are the way to go, though it's certainly not the most intuitive. On one system, I can't use device.hints anyway because the Chenbro CK12803 SAS expander (which is listed on the compatibility list for the LSI 9211-8i card in this box) doesn't seem to pass on the target information properly. That is, if a disk is in slot 0, it's target 19, and if I remove all the other disks, and put the same disk in slot 23, it's also target 19! On the Dell R720 with the 9205-8e's, I can lock down the targets through device.hints because there is a unique target number for each slot (even though the ordering is wacky, (which I still wish someone could explain)... so that da0 through da23 is properly fixed, but I don't want BIOS changes, kernel changes, driver changes, etc. to mess with that ordering every time or it will be yet another headache!

Assuming that I glabel the disks before they are made part of a zfs filesystem, and I'm the only one using them, can you suggest any reason why glabel wouldn't be the best way to do the labelling? These are not advanced format drives (WD2002FAEX-0), so isn't glabel the simplest method for labelling the disk? If I use gpt, there is additional partitioning as well and I just prefer to use the whole disk for the filesystem. (I wish that as ZFS filesystems were setup, freebsd automatically GPTed the disk behind the scenes, but that's another story.)....

Jason.
 
Ok -- I'm rather new to FreeBSD, but now you have me confused.

As far as I understand it, if I want to use disk labels, there are two ways of doing this under FreeBSD... glabel and gpt label...

If I use glabel, then as I understand it, one block is reserved from the end of the disk for the label, but there is no partitioning... and I do it like this:

Code: 
glabel label enc0disk0 da5
glabel label enc0disk1 da0
zpool create test mirror /dev/label/enc0disk0 /dev/label/enc0disk1

If I want to use gpt, as I understand it, I have to put a GPT partition table on the disks, partition the disks, and label something like this:

Code: 
gpart create -s gpt da0 (with every disk)
gpart add -t freebsd-zfs -l enc0disk0 da5 
gpart add -t freebsd-zfs -l enc0disk1 da0
zpool create test mirror /dev/gpt/enc0disk0 /dev/gpt/enc0disk1

Am I wrong!?
Why is one method preferred over the other one?
 
Okay, I see what you mean. Using gpart(8) is extra work if you normally don't partition disks at all. However, it has some advantages, like being able to align partitions to 4K. And the labels are part of the partition table, not an external structure like with glabel(8).
 
Maybe you've done so already or it is irrelevant, but I've seen mentioned elsewhere one should also physically label the (raid, zfs) disks, so if one needs replacing, one can tell by looking at the physical disk which should be removed... and it maybe has to be done before the initialization or install of the raid and/or operating system.
 
You must log in or register to reply here.
Back
Top