[ZFS] Replace 3TB drive with 2TB drive in zpool consisting of almost all 2TB drives

Diod · #1 July 27th, 2012, 14:22

I have a zpool with 2 raidz2 vdevs consisting of 6 2TB drives each. One of the drives was starting to fail, so I RMA'd it and got a 3TB drive in return. I know that if I replace the old drive (2TB) with the new one (3TB) that only 2TB will ever be used on the 3TB drive.

What I want to know is if I decide to replace the 3TB drive with a 2TB one later on (while the other drives are still 2TB), is zfs going to give me trouble?

I tried this under VMWare with 2 raidz2 vdevs of 4 x 1TB each and replaced one of the 1TB drives with a 2TB drive. Afterwards I replaced the 2TB drive with a 'fresh' 1TB drive and it seemed to work. However, since I only tested it when the zpool was nearly empty (it contained just some very small files), I was hoping if someone could tell me if it works with a zpool that's 70% full. It probably will, but I was hoping for someone who knows more about zfs than me to comment on this.

By the way, I am running FreeBSD 8.3-RELEASE-p3 with zfs version 5 and zpool version 28.

usdmatt · #2 July 27th, 2012, 17:03

Not that i'm 100% certain but I see no reason why the amount of data on the pool would have any effect on whether you can replace a drive or not. If ZFS doesn't like replacing a 3TB drive with a 2TB, even though the other disks in the vdev are 2TB it would complain even with no data on it.

The size of a vdev is limited by the smallest disk (obviously). You should be able to replace any disk in that vdev, regardless of the amount of pool usage, as long as the new disk is as big or bigger than the smallest disk in the vdev (the one which was used to determine the vdev size originally).

As you say, only 2TB of the 3TB disk is used. If you offline the 3TB disk, I see no reason why you shouldn't be able to replace it with a 2TB disk (unless of course it turns out to be a 2TB disk from a different manufacturer that's a few bytes smaller than the originals)

The output of zdb -v on my machine gives the following for the size for my first vdev.

Code:

            asize: 1000199946240

You can use that to confirm exactly how much space ZFS is using on the disks in a vdev if they aren't identical sizes. I highly suspect that ZFS will be perfectly happy for me to use any disk in this vdev as part of a replace that's at least that big (931.5GB in bytes - this vdev is made up of 1TB drives)

Diod · #3 July 27th, 2012, 17:26

Yes, I know the data on the pool shouldn't have any effect on what I want to do, just wanted to be sure. I wasn't sure because of this (which can be found here):

Quote:

In previous releases, ZFS was not able to replace an existing disk with another disk or attach a disk if the replacement disk was a slightly different size. In this release, you can replace an existing disk with another disk or attach a new disk that is nominally the same size provided that the pool is not already full.

Since ZFS seems to behave a different way depending on the amount of data on the pool in the case I quoted, I figured it could also behave differently in other cases.
But then again, as you said, I see no reason why the amount of data on the pool would have an effect on what I want to do so I guess I'll just use the 3TB disk to replace the old one.

Thanks.

**phoenix** · #4 July 28th, 2012, 23:42

Create a 2 TB partition on the 3 TB drive, and use that to replace the dead 2 TB drive.

Then, when you get the new 2 TB drive, use that to replace the 2 TB partition.

Diod · #5 July 29th, 2012, 20:26

I guess I could do that.
However, I might also decide to replace the other 2TB drives with 3TB drives later on if the need for more space arises, so then I'd have to destroy the partition on this 3TB drive and resilver.

It seems as if replacing the 3TB drive with a 2TB one works in VMware (albeit with 2TB and 1TB drives respectively), so I don't really see why it wouldn't work with my real data set.
Has anyone done this kind of thing before?

**phoenix** · #6 July 30th, 2012, 21:05

If you are never going to replace the 3 TB with a 2 TB, then you can just use the whole drive. Later, you can replace the other 2 TB drives with 3 TB drives, and get a lot of extra space in the pool.

If, however, the 3 TB drive is just a temporary thing, and you will be replacing it with a 2 TB once you get it back, then you will want to partition the 3 TB drive, so that everything stays at 2 TB.

Diod · #7 July 31st, 2012, 00:23

Well, replacing the 3 TB drive with a 2 TB drive was only hypothetical, in case the 3 TB drive fails before I need to expand the size of my pool.

I've done some more tests in a virtual machine, where the pool is almost full (90+%), but with smaller sized disks (with 128 MB disks representing my 2 TB disks and a 256 MB disk representing the 3 TB disk). It didn't give me any trouble replacing the 256 MB disk (which replaced another 128 MB one) with a 128 MB one. So I guess I'll just use the entire 3 TB disk to replace the failed one.

If I ever run into trouble replacing the 3 TB disk with a 2 TB one, which is unlikely, I'll be sure to report back.

Thanks

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Solved] ZFS cannot replace failed drive	rbauer_snow	Storage	6	March 24th, 2012 20:46
Zpool - Incorrecly added a drive to my raidz configuration	ZpoolNoob	General	7	August 16th, 2011 18:37
Chances of failure to drives on a 4 drive ZFS raidz1	overmind	General	33	November 13th, 2010 11:15
[Solved] Do I need multiple hard drives (or unformatted space on a single hard drive) for ZFS?	Agi93	General	21	January 18th, 2010 20:33
zfs , adding a usb hard drive to a zpool	wonslung	General	4	December 2nd, 2009 04:03

The Following User Says Thank You to usdmatt For This Useful Post:
Diod (July 27th, 2012)

The Following User Says Thank You to phoenix For This Useful Post:
Diod (July 31st, 2012)