zfs resilver restarts

[cjyar]

Similar to this thread and a few others, I'm seeing zfs resilver restart part way through the process, so it never completes.

I'm using 9.0-RELEASE on i386, with 2 zfs pools. The first pool is the system disk, connected via SATA. It has no problems. The second pool is two 2G disks in a USB-to-SATA enclosure, and it's having problems.

The pool was originally one disk (da1), but I added a second one (da2) using [cmd=]zpool attach paydirt da1 da2[/cmd] This starts the resilver, which looks pretty normal:

  pool: paydirt
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Sun Mar 25 11:17:26 2012
    3.18G scanned out of 75.3G at 5.57M/s, 3h41m to go
    3.18G resilvered, 4.22% done
config:

        NAME        STATE     READ WRITE CKSUM
        paydirt     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0  (resilvering)

errors: No known data errors

Eventually, zpool status changes to say that both da1 and da2 are resilvering, then it restarts from the beginning with only da2 resilvering. The restart seems to happen at a different point in the process each time.

 

[phoenix]

Check /var/log/messages around the time of the resilver restarts, and you'll probably find that the USB connection is dropping / being redetected and the disks are going offline/online. It's a common problem with USB enclosures, and is why USB should not be used for anything serious like this.

The best solution is to look into eSATA. An okay solution would be to run normal power/SATA cables through a slot in the back of the case. A decent workaround would be to figure out which USB buses are on which controller and plug each drive into a separate controller.

 

[cjyar]

Thanks for your reply, phoenix. Sorry; I forgot to mention: I checked /var/log/messages and dmesg, and there's nothing in there about any of these devices. Do I need to enable some USB or zfs debugging to see those messages? Or tweak syslog.conf to keep LOG_DEBUG?

The suitability of USB for various applications, and expectations regarding robustness, sound like a separate thread or two.

 

[ekar]

Hi,

I have exactly the same problem.

I have two disks mirror: internal and USB. During the setup of system everything was fine, after i detached USB disk and attached new USB disk i cant get resilvering to finish.

I am suspecious about the fact that new disk i attached i cant get 4k alignment for the partitions (gpart -a 4k) - diskinfo or gpart list shows stripesize=0, but for the internal disk it shows stripezise=4096.

Some people say that after your zfs pool is 4k alignment (ashift=12) you do not need to align new disk partitions for 4k, but not much explaiend about that.

Karolis

 

[Sebulon]

Hi,

I have exactly the same problem.

I have two disks mirror: internal and USB. During the setup of system everything was fine, after i detached USB disk and attached new USB disk i cant get resilvering to finish.

I am suspecious about the fact that new disk i attached i cant get 4k alignment for the partitions (gpart -a 4k) - diskinfo or gpart list shows stripesize=0, but for the internal disk it shows stripezise=4096.

Some people say that after your zfs pool is 4k alignment (ashift=12) you do not need to align new disk partitions for 4k, but not much explaiend about that.

Karolis

It worked out during setup because normal file operations didn´t create the amount of IO that a resilver or scrub does. That´s why you just can´t use USB drives in a ZFS pool, because they drop out during heavy IO. And yes, if you´ve made sure the vdev is set ashift=12, performance works itself out without using an aligned partition, but you should partition so you can label it, and use the label when creating the pool to avoid pulling the wrong drive when replacing. Also it´s needed if you want to have a small boot partition and the rest as a data partition, when booting from the pool.

/Sebulon