ZFS Remove a disc from a RAIDZ1 pool and recreate a new pool

patpro

patpro

Hello,

I have a server with 4 SAS HDDs (no empty slot remaining), configured as a RAIDZ1
sh: 
$ zpool list -v sas
NAME            SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
sas            21.8T  4.36T  17.4T        -         -    38%    20%  1.00x    ONLINE  -
  raidz1-0     21.8T  4.36T  17.4T        -         -    38%  20.0%      -    ONLINE
    da0p3      5.45T      -      -        -         -      -      -      -    ONLINE
    da1p3      5.45T      -      -        -         -      -      -      -    ONLINE
    da2p3      5.45T      -      -        -         -      -      -      -    ONLINE
    da3p3      5.45T      -      -        -         -      -      -      -    ONLINE
logs               -      -      -        -         -      -      -      -  -
  gpt/nvmelog    16G  5.11M  15.5G        -         -     0%  0.03%      -    ONLINE

It's been running for years without problem but my workload has changed and I would like to replace it with SSD storage (need IOps). I am not interested in rebuilding the same pool with SSDs, it would be way too expensive.

The server is in a datacenter far away and the best thing I can do is to prepare a drive and ship it to my hosting company, so they replace one drive at a time. I have IPMI access.

I think I can follow this scenario:
  1. - prepare 1 SSD in another machine (FreeBSD install+config+data rsync)
  2. - ship 2 SSDs (may be Kingston DC600M SSD 2.5” Enterprise SATA SSD 7.6TB)
  3. - zpool remove 1 HDD from pool
  4. - physical removal of HDD (hosting tech support)
  5. - insertion of first SSD (hosting tech support)
  6. - service shutdown (mail server, web, etc)
  7. - last data rsync
  8. - reboot on SSD
  9. if it’s all OK:
    1. - destruction of sas zpool
    2. - physical removal of HDD (hosting tech support)
    3. - insertion of second SSD (hosting tech support)
    4. - mirror creation between 1st and 2nd SSD (all data & system)
    5. - mirror creation between 2 remaining HDD (empty)

Does this scenario looks OK to you?
Did I miss something?
Also, I need a good 3.5 to 2.5 adapter to secure the SSD in the hot plug HDD tray, pointers appreciated.

Thanks a lot
 
I always get nervous when degrading a RAID set and making significant changes at the same time.

The potential is for finger trouble at the worst possible time.

Hope your backups are sound, and recovery verified.

Last time I re-laid a pool like yours I purchased external SATA to USB converters, and re-built the RAID set on externally attached drives which could be plugged internally after the event. Maybe not an option for a managed facility.

Have you considered using PCIe NVMe adapters?
 
Agreed, it makes me nervous too, but back in the days I used to upgrade FreeBSD by make/make install/etc, remotely over ssh and without IPMI ;)
Good preparation is the key (and competent support inside the DC in case it derails).

NVMe is not an option for me:
- for the price of 2x8TB NVMe I could get 3x7.6TB SSD
- I dont have room for a PCI adapter anyway

(and by the way, my HGST Ultrastar 7K6 6TB 7200RPM SAS can do what, 600 100 to 140 iops? The Kingston SSD mentioned above tops at 94,000 read and 34,000 write, I don’t need NVMe)

So back to square one -> is there any loophole/error in my scenario?
 
Last edited:
If I understand you correctly, what you intend to do is send the SSDs to the data center, degrade the Z1 to be able to place the SSD and create a new pool and then synchronize the data from the Z1 to the SSD pool, is that correct?

That could be dangerous, as colleagues commented. Would you be able to have a machine at your current site and create the pool locally and synchronize the data remotely?

At my old job I had a server with ZFS and I incrementally sent my snapshots to the small server at home through a pipe with ssh, I don't know if you could do that. On the other hand, why would you use rsync with ZFS?

If it doesn't seem like much more work, you can do the following:

- Install the Kingston DC600M SSDs on a local machine, create the pool.
- Initialize an SSH server to receive data with the machine in the datacenter, you can also use applications such as mbuffer but you would need a VPN.
- Create a snapshot of the ZVOL or data sheets that you want to import from the datacenter machine, then initialize a send through an ssh pipe and a recv on the remote machine, example:

Code: 
datacenter# zfs snap mypool/myzvol@snap0
datacenter# zfs send mypool/myzvol@snap0 | ssh homeIP zfs recv ssdpool/myzvol   #recv in your local machine

- Once you have the data exported, I would mark the SSD pool as exportable and send the disks would import the new pool, and then create the mirrors etc...

zpool export ssdpool

......

Keep in mind that you would have to create some NAT rule or include the receiving machine in a DMZ to make this possible, choose which would be the best solution for you, we do not know your network topology.

This could be another idea.
 
freejlr said:
Would you be able to have a machine at your current site and create the pool locally and synchronize the data remotely?
Click to expand...

that’s actually step 1 of my scenario :)
I prepare one SSD, and ship it with another blank SSD (step 2)

freejlr said:
- Once you have the data exported, I would mark the SSD pool as exportable and send the disks would import the new pool, and then create the mirrors etc...

zpool export ssdpool
Click to expand...

ohhh I would probably have forgot this one, thanks.
I’ll have to import my SAS pool after booting on the SSD, too.

About data synchronization: I plan to restore most files from a backup, then complete with a long distance rsync.
A second rsync will do the job before destroying the SAS  pool (step 7)
Some particular data will have to be transferred the «ZFS way», for example some bhyve VM storage that exist only as «volume». Probably step 7 too.
 
I’m currently testing parts and variants of this scenario.
It can be really important to use an altroot when importing a pool ;)

After step 5, I should do
Code: 
zpool import -o altroot=/mnt my-other-pool
to prevent a huge mess.

Also, as I plan to ship a FreeBSD 14 install on SSD to insert in a FreeBSD 13 system, I’ll have to reboot on the SSD first, otherwise I wont be able to import/mount/sync data at step 7:

Bash: 
# zpool import -f -o altroot=/mnt ssd
Apr 25 11:24:33 zraid1-test ZFS[684]: failed to load zpool ssd
This pool uses the following feature(s) not supported by this system:
        com.klarasystems:vdev_zaps_v2
cannot import 'ssd': unsupported version or feature

So new plan could be:

  1. - prepare 1 SSD in another machine (FreeBSD install+config+data rsync)
  2. - ship 2 SSDs (may be Kingston DC600M SSD 2.5” Enterprise SATA SSD 7.6TB)
  3. - zpool remove 1 HDD from pool
  4. - physical removal of HDD (hosting tech support)
  5. - insertion of first SSD (hosting tech support)
  6. - reboot on SSD and zpool import -f -o altroot=/mnt sas
  7. - last data rsync
  8. if it’s all OK:
    1. - destruction of sas zpool
    2. - physical removal of HDD (hosting tech support)
    3. - insertion of second SSD (hosting tech support)
    4. - mirror creation between 1st and 2nd SSD (all data & system)
    5. - mirror creation between 2 remaining HDD (empty)
Alternate plan is to keep first plan but upgrade current server from 13-RELEASE to 14-RELEASE before step 2. I’m too lazy for that.
 
gpw928 said:
Last time I re-laid a pool like yours I purchased external SATA to USB converters, and re-built the RAID set on externally attached drives which could be plugged internally after the event. Maybe not an option for a managed facility.
Click to expand...
gpw928 makes some valid points, it's a pretty risky scenario, and I would maybe try something like this with a combination of the answer of freejlr.
I would probably go about it like this:
1. Build the complete system at your place, create a new zpool and replicate all data using zfs send / zfs receive (or rsync if not all of it is on a zpool)
2. Test the system, it should be ready for 1-1 in-place replacement so you would only have to plug it in and be ready to go
3. Export the new zpool you have and ship the SSDs with express shipment
4. At the datacenter: Have some means of plugging your new pool in beside the current production machine (or even plug them in to the machine you already have if possible), be it another machine or external adapters like gpw928 mentioned
5. service shutdown, data sync, migration etc.
 
You must log in or register to reply here.
Back
Top