Replace 3TB drive with 2TB drive in zpool consisting of almost all 2TB drives

Diod

Diod

I have a zpool with 2 raidz2 vdevs consisting of 6 2TB drives each. One of the drives was starting to fail, so I RMA'd it and got a 3TB drive in return. I know that if I replace the old drive (2TB) with the new one (3TB) that only 2TB will ever be used on the 3TB drive.

What I want to know is if I decide to replace the 3TB drive with a 2TB one later on (while the other drives are still 2TB), is zfs going to give me trouble?

I tried this under VMWare with 2 raidz2 vdevs of 4 x 1TB each and replaced one of the 1TB drives with a 2TB drive. Afterwards I replaced the 2TB drive with a 'fresh' 1TB drive and it seemed to work. However, since I only tested it when the zpool was nearly empty (it contained just some very small files), I was hoping if someone could tell me if it works with a zpool that's 70% full. It probably will, but I was hoping for someone who knows more about zfs than me to comment on this.

By the way, I am running FreeBSD 8.3-RELEASE-p3 with zfs version 5 and zpool version 28.
 
Not that i'm 100% certain but I see no reason why the amount of data on the pool would have any effect on whether you can replace a drive or not. If ZFS doesn't like replacing a 3TB drive with a 2TB, even though the other disks in the vdev are 2TB it would complain even with no data on it.

The size of a vdev is limited by the smallest disk (obviously). You should be able to replace any disk in that vdev, regardless of the amount of pool usage, as long as the new disk is as big or bigger than the smallest disk in the vdev (the one which was used to determine the vdev size originally).

As you say, only 2TB of the 3TB disk is used. If you offline the 3TB disk, I see no reason why you shouldn't be able to replace it with a 2TB disk (unless of course it turns out to be a 2TB disk from a different manufacturer that's a few bytes smaller than the originals)

The output of zdb -v on my machine gives the following for the size for my first vdev.

Code: 
            asize: 1000199946240

You can use that to confirm exactly how much space ZFS is using on the disks in a vdev if they aren't identical sizes. I highly suspect that ZFS will be perfectly happy for me to use any disk in this vdev as part of a replace that's at least that big (931.5GB in bytes - this vdev is made up of 1TB drives)
 
Yes, I know the data on the pool shouldn't have any effect on what I want to do, just wanted to be sure. I wasn't sure because of this (which can be found here):

In previous releases, ZFS was not able to replace an existing disk with another disk or attach a disk if the replacement disk was a slightly different size. In this release, you can replace an existing disk with another disk or attach a new disk that is nominally the same size provided that the pool is not already full.
Click to expand...

Since ZFS seems to behave a different way depending on the amount of data on the pool in the case I quoted, I figured it could also behave differently in other cases.
But then again, as you said, I see no reason why the amount of data on the pool would have an effect on what I want to do so I guess I'll just use the 3TB disk to replace the old one.

Thanks.
 
Create a 2 TB partition on the 3 TB drive, and use that to replace the dead 2 TB drive.

Then, when you get the new 2 TB drive, use that to replace the 2 TB partition.
 
I guess I could do that.
However, I might also decide to replace the other 2TB drives with 3TB drives later on if the need for more space arises, so then I'd have to destroy the partition on this 3TB drive and resilver.

It seems as if replacing the 3TB drive with a 2TB one works in VMware (albeit with 2TB and 1TB drives respectively), so I don't really see why it wouldn't work with my real data set.
Has anyone done this kind of thing before?
 
If you are never going to replace the 3 TB with a 2 TB, then you can just use the whole drive. Later, you can replace the other 2 TB drives with 3 TB drives, and get a lot of extra space in the pool.

If, however, the 3 TB drive is just a temporary thing, and you will be replacing it with a 2 TB once you get it back, then you will want to partition the 3 TB drive, so that everything stays at 2 TB.
 
Well, replacing the 3 TB drive with a 2 TB drive was only hypothetical, in case the 3 TB drive fails before I need to expand the size of my pool.

I've done some more tests in a virtual machine, where the pool is almost full (90+%), but with smaller sized disks (with 128 MB disks representing my 2 TB disks and a 256 MB disk representing the 3 TB disk). It didn't give me any trouble replacing the 256 MB disk (which replaced another 128 MB one) with a 128 MB one. So I guess I'll just use the entire 3 TB disk to replace the failed one.

If I ever run into trouble replacing the 3 TB disk with a 2 TB one, which is unlikely, I'll be sure to report back.

Thanks :)
 
You must log in or register to reply here.
Back
Top