Hi all,
Since I keep reading things here and there, I decided to share my experience growing a ZFS pool on a backup NAS.
I have 16 drives used for storage.
2 x 4TB
2 x 3TB
14 x 1TB
For this exercise, I already have 7TB of data on an ZFS pool of the 2 x 4TB as a stripe (was used on a different system as external drives before I found my backup system for a proper NAS).
In order to avoid, transferring from my main NAS over the network and leverage the fact the data is already on the usb pool, I decided to create a pool and grow it first to test, then to transfer data with ZFS.
Once it is done, I will add the 2 extra 4 TB as a mirror.
I started by creating a small pool of 2 mirrors vdev made of 2 drives each.
Then I created my pool made of 4 drives:
Which results in:
Follow the same steps for preparing other drives as before and add another mirror
Giving me:
and
then adding mirror after mirror...
until I have enough:
Then snapshot my previous stripe
NOTE: My usb pool was full so I had to delete some data to allow for the migrate snapshot to be created
Let it run until full completion (currently cruising at 256MB/s for the first 20 minutes on three old SATA-II supermicro controllers) ETA: after a good night of sleep
UPDATE1:
Estimate was a bit shorter than reality:
The only question for now is why is the destination zpool (zstuff here) not visible with df during the zfs send/receive?
I can access it:
I was replicating a pool and not a dataset. Thanks to Oko and kpa for the insights on the reason for what I observed. That makes sense and everything is nicely accessible now it is complete.
I then destroy my ZFS pool called usb.
As my two 4TB harddrives are called ada0 and ada1, I prepared them for being added as a new mirror vdev using the same process as above and ended with the following:
Migration is done - Thanks to all people working on the ZFS implementation on FreeBSD to make this possible.
Since I keep reading things here and there, I decided to share my experience growing a ZFS pool on a backup NAS.
I have 16 drives used for storage.
2 x 4TB
2 x 3TB
14 x 1TB
For this exercise, I already have 7TB of data on an ZFS pool of the 2 x 4TB as a stripe (was used on a different system as external drives before I found my backup system for a proper NAS).
Code:
pool: usb
state: ONLINE
scan: resilvered 0 in 0h0m with 0 errors on Thu Oct 9 21:11:28 2014
config:
NAME STATE READ WRITE CKSUM
usb ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada0 ONLINE 0 0 0
Code:
usb 7.1T 7.1T 0B 100% /usbIn order to avoid, transferring from my main NAS over the network and leverage the fact the data is already on the usb pool, I decided to create a pool and grow it first to test, then to transfer data with ZFS.
Once it is done, I will add the 2 extra 4 TB as a mirror.
I started by creating a small pool of 2 mirrors vdev made of 2 drives each.
gpart destroy adaX <<< where X is your drive
gpart create -s gpt adaX
gpart add -a 1m -t freebsd-zfs -L 1TB-<SerialNumber_Of_the_drive> adaX
gnop create -S 4k gpt/1TB-<SerialNumber_Of_the_drive>Then I created my pool made of 4 drives:
zpool create zstuff mirror gpt/1TB-9QJ1LBA8 gpt/1TB-9XG4PTLH mirror gpt/1TB-9XG2WLM2 gpt/1TB-WD-WCAV5J25313Which results in:
Code:
pool: zstuff
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zstuff ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/1TB-9QJ1LBA8.nop ONLINE 0 0 0
gpt/1TB-9XG4PTLH ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gpt/1TB-9XG2WLM2 ONLINE 0 0 0
gpt/1TB-WD-WCAV5J253135 ONLINE 0 0 0Follow the same steps for preparing other drives as before and add another mirror
zpool add zstuff mirror gpt/1TB-WD-WMAV50349152 gpt/1TB-9XG4PY8CGiving me:
Code:
pool: zstuff
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zstuff ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/1TB-9QJ1LBA8.nop ONLINE 0 0 0
gpt/1TB-9XG4PTLH ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gpt/1TB-9XG2WLM2 ONLINE 0 0 0
gpt/1TB-WD-WCAV5J253135 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gpt/1TB-WD-WMAV50349152 ONLINE 0 0 0
gpt/1TB-9XG4PY8C ONLINE 0 0 0and
Code:
zstuff 2.6T 58K 2.6T 0% /zstuffthen adding mirror after mirror...
Code:
zstuff 3.5T 58K 3.5T 0% /zstuffuntil I have enough:
Code:
zstuff 7.9T 58K 7.9T 0% /zstuffThen snapshot my previous stripe
NOTE: My usb pool was full so I had to delete some data to allow for the migrate snapshot to be created
Code:
root@oz:/usb/Downloads # cd /
root@oz:/ # zfs snapshot -r usb@migrate
root@oz:/ # zfs send -Rv usb@migrate | zfs receive -vFd zstuff
send from @ to usb@migrate estimated size is 6.97T
total estimated size is 6.97T
TIME SENT SNAPSHOT
receiving full stream of usb@migrate into zstuff@migrate
21:32:16 57.2K usb@migrate
21:32:17 117M usb@migrate
21:32:18 399M usb@migrate
21:32:19 692M usb@migrate
21:32:20 929M usb@migrate
21:32:21 1.20G usb@migrate
21:32:22 1.47G usb@migrate
21:32:23 1.75G usb@migrate
21:32:24 2.02G usb@migrate
21:32:25 2.19G usb@migrate
21:32:26 2.46G usb@migrateLet it run until full completion (currently cruising at 256MB/s for the first 20 minutes on three old SATA-II supermicro controllers) ETA: after a good night of sleep
UPDATE1:
Estimate was a bit shorter than reality:
Code:
received 6.97TB stream in 37183 seconds (197MB/sec)The only question for now is why is the destination zpool (zstuff here) not visible with df during the zfs send/receive?
I can access it:
Code:
Filesystem Size Used Avail Capacity Mounted on
zroot/ROOT/default 46G 1.3G 44G 3% /
devfs 1.0K 1.0K 0B 100% /dev
zroot/tmp 44G 96K 44G 0% /tmp
usb 7.0T 7.0T 63G 99% /usb
zroot/usr/home 44G 136K 44G 0% /usr/home
zroot/usr/ports 45G 623M 44G 1% /usr/ports
zroot/usr/src 44G 96K 44G 0% /usr/src
zroot/var/crash 44G 96K 44G 0% /var/crash
zroot/var/log 44G 168K 44G 0% /var/log
zroot/var/mail 44G 96K 44G 0% /var/mail
zroot/var/tmp 44G 96K 44G 0% /var/tmp
$ cd /zstuff
$ pwd
/zstuff
root@oz:~ # df -h zstuff
df: zstuff: No such file or directory
root@oz:~ # df -h usb
Filesystem Size Used Avail Capacity Mounted on
usb 7.0T 7.0T 63G 99% /usbI was replicating a pool and not a dataset. Thanks to Oko and kpa for the insights on the reason for what I observed. That makes sense and everything is nicely accessible now it is complete.
I then destroy my ZFS pool called usb.
zpool destroy usbAs my two 4TB harddrives are called ada0 and ada1, I prepared them for being added as a new mirror vdev using the same process as above and ended with the following:
Code:
pool: zstuff
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zstuff ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/1TB-9QJ1LBA8.nop ONLINE 0 0 0
gpt/1TB-9XG4PTLH ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gpt/1TB-9XG2WLM2 ONLINE 0 0 0
gpt/1TB-WD-WCAV5J253135 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gpt/1TB-WD-WMAV50349152 ONLINE 0 0 0
gpt/1TB-9XG4PY8C ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
gpt/1TB-9XG32ABP ONLINE 0 0 0
gpt/1TB-9XG4PSMW ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
gpt/1TB-9XG2ZY30 ONLINE 0 0 0
gpt/1TB-9XG4PZFJ ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
gpt/1TB-915ITAKCT ONLINE 0 0 0
gpt/1TB-9QJ20JWX ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
gpt/3TB-WD-WCC1T0406094 ONLINE 0 0 0
gpt/3TB-WD-WMC1T2787521 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
gpt/4TB-W300R1HE ONLINE 0 0 0
gpt/4TB-W300R00H ONLINE 0 0 0 df -h | grep zstuff
zstuff 11T 7.0T 4.5T 61% /zstuffMigration is done - Thanks to all people working on the ZFS implementation on FreeBSD to make this possible.
Last edited by a moderator:
