ZFS Replacing failed encrypted disk in zpool

L

listentoreason

Nutshell: I need to replace a geli-encrypted drive in my zpool. Looking at my setup notes, the geli devices in the pool are freebsd-zfs partitions, rather than 'raw' devices. Is that the correct approach?

Details: My monitoring script alerted me to a problem in my zpool:
Code: 
  pool: abyss
 state: DEGRADED
status: One or more devices has been removed by the administrator.
   Sufficient replicas exist for the pool to continue functioning in a
   degraded state.
action: Online the device using 'zpool online' or replace the device with
   'zpool replace'.
  scan: scrub repaired 0 in 4h40m with 0 errors on Sun Feb  1 05:40:51 2015
config:

   NAME  STATE  READ WRITE CKSUM
   abyss  DEGRADED  0  0  0
    raidz3-0  DEGRADED  0  0  0
    gpt/GELI-da0.eli  ONLINE  0  0  0
    gpt/GELI-da1.eli  ONLINE  0  0  0
    11837074149101951419  REMOVED  0  0  0  was /dev/gpt/GELI-da2.eli
    gpt/GELI-da3.eli  ONLINE  0  0  0
    gpt/GELI-da4.eli  ONLINE  0  0  0
    gpt/GELI-da5.eli  ONLINE  0  0  0
    gpt/GELI-da6.eli  ONLINE  0  0  0
    gpt/GELI-da7.eli  ONLINE  0  0  0
    gpt/GELI-da8.eli  ONLINE  0  0  0
    gpt/GELI-da9.eli  ONLINE  0  0  0
    gpt/GELI-da11.eli  ONLINE  0  0  0

zpool online does not work, the device is immediately removed. Next time I have the system powered down I'll make sure all the cabling is ok, but for the moment I am assuming the drive is wonky and will replace it with the spare, da10. So far I have done:

Code: 
set DISKNAME=da10
gpart destroy -F "$DISKNAME"
gpart create -s gpt "$DISKNAME"

I thought that the next step would then be (I am using a passphrase only):
Code: 
geli init -s 4096 "/dev/$DISKNAME"

However, looking at my notes I seem to have instead made a freebsd-zfs partition, and then encrypted that:
Code: 
gpart add -t freebsd-zfs -l "GELI-$DISKNAME" -b 2048 -a 4k "$DISKNAME"
geli init -s 4096 "/dev/gpt/GELI-$DISKNAME"

I'm not entirely sure why I have the "additional" partition in there, I presume it was an attempt to assure the blocks are properly 4k aligned (these are 3 TB drives that masquerade with 512 b blocks). Is the gpart add step required, or at least advised? Or have I set things up goofy? If I've been goofy, I presume that the zpool will happily accept the "raw" da10 device (ie, without adding the partition) with it's partitioned brethren? Thanks!

By the way, this is the first hiccup with the pool - using 4 of 20 TB running flawlessly for 16 months without even a single checksum error, and it dutifully warns me when a drive goes belly-up. This is why I corner my colleagues in the break room and try to convince them we need to use ZFS! :D
 
Whilst you can use an entire disk for ZFS without setting up a partition, I believe it is better to use a partition that is slightly smaller than the disk.

The reason is described in the zpool(8) man page:
zpool replace [-f] pool device [new_device]

Replaces old_device with new_device. This is equivalent to attaching
new_device, waiting for it to resilver, and then detaching
old_device.

The size of new_device must be greater than or equal to the minimum
size of all the devices in a mirror or raidz configuration.

new_device is required if the pool is not redundant. If new_device is
not specified, it defaults to old_device. This form of replacement
is useful after an existing disk has failed and has been physically
replaced. In this case, the new disk may have the same /dev path as
the old device, even though it is actually a different disk. ZFS
recognizes this.

-f Forces use of new_device, even if its appears to be in use.
Click to expand...

Thus, if you need to replace the disk in the future, and the replacement disk is just 1 sector smaller than the original, you cannot use it.

This is discussed further here:
http://www.freebsddiary.org/zfs-with-gpart.php
 
Also, as you already mentioned, partitioning can help with 4KB sector alignment, as described in this article:
http://daemon-notes.com/articles/system/advanced-format

Note that the gnop(8) command from the above article is not required if using disk encryption with geli(8), as the geli can enforce the 4KB sector size with a command such as:
# geli init -s 4096 -K /root/da2.key /dev/da2

More details on how to setup geli(8) are available in the handbook section 18.12.2 here (scroll down to geli section):
https://www.freebsd.org/doc/handbook/disks-encrypting.html
 
Sorry for the off-topic, but since FreeBSD 10.1-RELEASE and FreeBSD 9.3-RELEASE gnop() is no longer required either. You can force ashift=12 by issuing the following command:

sysctl vfs.zfs.min_auto_ashift=12

Of course, this can become permanent also if you add it in sysctl.conf
 
gkontos said:
You can force ashift=12 by issuing the following command:
sysctl vfs.zfs.min_auto_ashift=12
Click to expand...

That is extremely useful information, thank you. I found more information regarding it at the ZFS Advanced Topics section of the handbook. Reading that, it implies that the parameter is only checked during pool creation? I am uncertain if adding a replacement drive to a pool would cause the parameter to be consulted.
 
listentoreason said:
That is extremely useful information, thank you. I found more information regarding it at the ZFS Advanced Topics section of the handbook. Reading that, it implies that the parameter is only checked during pool creation? I am uncertain if adding a replacement drive to a pool would cause the parameter to be consulted.
Click to expand...

In general when you replace a drive in a pool, you don't need to use the gnop trick (At least that's what I think) to maintain proper alignment. Therefore, this parameter does not stand.

However, I tend to always have vfs.zfs.min_auto_ashift=12 in my sysctl.conf.
 
You must log in or register to reply here.
Back
Top