ZFS and disk renumbering

B

b0redom

Hi,
I was up until recently using Opensolaris for my NAS box. Following an aborted attempt to migrate to ESXi with hardware RAID and then Linux, I've decided to go back to ZFS. OpenSolaris is now dead, so I'm now looking at alternatives.

One of the nice things about OpenSolaris was that it didn't renumber the disks, it always remembered that port 3 on the controller mapped to c0t3d0 etc. I know that this is not the case on Linux, that is if ports 0-7 on a controller are populated and the disk on port 5 dies, sda6 -> sda5, sda7->sda6.

Does this happen on FreeBSD too? If a disk disappears, or dies, how can I tell which one it is? Will all the disks be renumbered if this happens? Will my ZFS setup run into issues?

Regards...

b0redom
 
As an additional note, I was planning on using RAW disks rather than paritions. Do I need to label them, or do I just do:

zpool create tank <disk1> <disk2> .........

TIA
 
Static numbering is defined by default:
Code: 
options         ATA_STATIC_ID   # Static device numbering

So ad1 will always be ad1 even if there's no ad0. Unless you turn that option off.
 
Great! One last thing. I presume that they're not probed in order though?

I have a mobo with 6 SATA ports (2 of which are populated). I also have a separate SATA controller with 8x 1.5TB drives attached.

So ad0 will be disk0 (on the mobo)
ad1 will be disk1 (on the mobo)
ad2 will be disk0 (on the controller)
ad3 will be disk1 (on the controller)

What's the best way to identify the disks before I have a massive outage? :)

Cheers...

Tom
 
b0redom said:
Will my ZFS setup run into issues?
Click to expand...
# zpool export <poolname>
# zpool import <poolname>
That series of commands, on any OS with ZFS support, will cause ZFS to probe all the disks in the pool, read the ZFS meta-data off them, and shuffle things around internally so that it knows where everything is. And then continue on its merry way.

You can export, shuffle disks around, import, and it will be like nothing happened. :)

It's one of the beautiful bits of ZFS. :D
 
b0redom said:
As an additional note, I was planning on using RAW disks rather than paritions. Do I need to label them, or do I just do:
Click to expand...

Personally, I like to label things, so that you have nice names to work with (disk01, disk02, disk03, etc) instead of the raw device nodes (which may or may not be static).

How you label them is up to you. If using the entire drive, glabel(8) is simplest:
Code: 
# glabel label disk01 ad0
# glabel label disk02 ad1
# zpool create mypool mirror label/disk01 label/disk02

If you partition the drive using gpt(), then you can use the gpt labels:
Code: 
<partition drives using gpt, give partitions names>
# zpool create mypool mirror gpt/disk01 gpt/disk02

The nice thing about using labels is that you don't have to do the export/import if you shuffle drives around. The kernel handles all that for you, before ZFS is every loaded. :)

The downside to using labels, especially glabel, is that it makes the disks usable in FreeBSD only. GPT labels are more portable, I believe.
 
SirDice said:
Static numbering is defined by default:
Code: 
options         ATA_STATIC_ID   # Static device numbering

So ad1 will always be ad1 even if there's no ad0. Unless you turn that option off.
Click to expand...

Note: that only applies to ATA/IDE devices (/dev/ad* and maybe /dev/ada*).

It does not apply to devices accessed via the CAM/SCSI subsystem. SCSI devices are probed at boot, and are numbered based on what's present at enumeration time. Thus /dev/da* is always incremented starting at 0. If you have drives at boot, one dies, you remove it and reboot, drives will be renumbered.

I ran into that with our 3Ware controller, and is one of the reasons I've started to label everything. :)
 
phoenix said:
How you label them is up to you. If using the entire drive, glabel(8) is simplest:
Code: 
# glabel label disk01 ad0
# glabel label disk02 ad1
# zpool create mypool mirror label/disk01 label/disk02
Click to expand...

OK so assuming I want to do that, I guess the simplest way is to bring it up with only the boot drive, label that, then attach the disk on controller port 1, reboot, and then label that etc etc etc?

Sorry if it seems like I'm being pedantic, but I really want to get this right - most of the data I'm going to be storing can be restored, but it would be a real PITA to do.

Regards....

b0redom
 
You could also write down all their serial numbers and then identify each one in the os with:

Code: 
atacontrol cap ad0

The above command works for adX devices.
You would need to use camcontrol for da/ada devices.

I used this just yesterday to help sort out some messy sata cabling in a server and get 8 drives in the correct order so the left most drive is label/disk01, next is label/disk02, etc.

Another possible idea is to get the serial number with atacontrol, and then label the device with it's serial number. That way if a drive fails, zpool status will clearly show the serial number of the failed drive. When you shutdown you can confirm the correct drive has been removed by looking at the serial number printed on it.
 
phoenix said:
Note: that only applies to ATA/IDE devices (/dev/ad* and maybe /dev/ada*).

It does not apply to devices accessed via the CAM/SCSI subsystem. SCSI devices are probed at boot, and are numbered based on what's present at enumeration time. Thus /dev/da* is always incremented starting at 0. If you have drives at boot, one dies, you remove it and reboot, drives will be renumbered.
Click to expand...
/dev/ada* are not static. Users of ahci(4) should use glabel(8).
 
I had few cases to switch motherboards and some supported AHCI, some didn't. I had no problems to use the same pool, without having to import/export it (which is tricky with a root pool anyway ;))

So no matter if drives were ad? or ada? ZFS was always happy. ZFS doesn't seem to care much about the actual device names, as it stores the necessary metadata on each drive.

I also remember not having to export/import the pool in order to increase capacity by replacing all drives in a vdev, by the way -- as soon as all drives were rebuilt, the pool was with the correct capacity (but the old/spare drives with smaller capacity had to be resilvered to the larger media and not present in the device list).
 
It all depends. :) Sometimes it (export/import) makes a difference, sometimes it doesn't. I've experienced both. Using labels so that the kernel handles the drive access no matter what, provides some extra nice peace of mind. :)
 
You must log in or register to reply here.
Back
Top