ZFS Adding another mirror to a stripe.

I

i.am.the.problem

Hi,
I currently have the following zpool

Code: 
# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  10.9T  10.3T   646G        -         -    43%    94%  82.65x    ONLINE  -

# zpool status

  pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0p3   ONLINE       0     0     0
            da1p3   ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            da2p3   ONLINE       0     0     0
            da3p3   ONLINE       0     0     0
        cache
          nvd0      ONLINE       0     0     0

The disks are partitioned as follows:

Code: 
# gpart show

=>         40  11721045088  da0  GPT  (5.5T)
           40       532480    1  efi  (260M)
       532520         1024    2  freebsd-boot  (512K)
       533544          984       - free -  (492K)
       534528  11720509440    3  freebsd-zfs  (5.5T)
  11721043968         1160       - free -  (580K)

=>         40  11721045088  da1  GPT  (5.5T)
           40       532480    1  efi  (260M)
       532520         1024    2  freebsd-boot  (512K)
       533544          984       - free -  (492K)
       534528  11720509440    3  freebsd-zfs  (5.5T)
  11721043968         1160       - free -  (580K)

=>         40  11721045088  da2  GPT  (5.5T)
           40       532480    1  efi  (260M)
       532520         1024    2  freebsd-boot  (512K)
       533544          984       - free -  (492K)
       534528  11720509440    3  freebsd-zfs  (5.5T)
  11721043968         1160       - free -  (580K)

=>         40  11721045088  da3  GPT  (5.5T)
           40       532480    1  efi  (260M)
       532520         1024    2  freebsd-boot  (512K)
       533544          984       - free -  (492K)
       534528  11720509440    3  freebsd-zfs  (5.5T)
  11721043968         1160       - free -  (580K)

I have another 2 matching disks which I want to add as another mirror to give me another ~6TB of space.

I want to confirm that the following steps are correct to partition and set up the new drives:

Code: 
gpart add -a 4k -s 532480 -t efi /dev/da4
gpart add -a 4k -s 532480 -t efi /dev/da5

gpart add -a 4k -s 1024 -t freebsd-boot  /dev/da4
gpart add -a 4k -s 1024 -t freebsd-boot  /dev/da5

gpart add -a 4k -t freebsd-zfs /dev/da4
gpart add -a 4k -t freebsd-zfs /dev/da5

newfs_msdos -F 32 -c 1 /dev/da4p1
newfs_msdos -F 32 -c 1 /dev/da5p1

mount -t msdosfs /dev/da0p1 /mnt/temp
mkdir -p /mnt/temp/EFI/BOOT
cp /boot/loader.efi /mnt/temp/EFI/BOOT/BOOTx64.efi
umount /mnt/temp

mount -t msdosfs /dev/da5p1 /mnt/temp
mkdir -p /mnt/temp/EFI/BOOT
cp /boot/loader.efi /mnt/temp/EFI/BOOT/BOOTx64.efi
umount /mnt/temp

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 /dev/da4
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 /dev/da5

zpool add zroot mirror /dev/da4p3 /dev/da5p3

Code: 
# freebsd-version  
13.4-RELEASE-p3

I'll be upgrading after this...
 
That's nice.
And your question was?
😁

I guess you want feedback on what you're doing.
You can do so, of course, but may I suggest you may consider if it was also better if you:
- use gpt-labels instead of binding partitions into pools
- instead of adding the third mirror pool to do a raidz2 over all your disks
Edit:
You may have your reasons (I can't see from just the partition and pool tables), but instead of having a boot, and Efi partition on each drive, plus (as it seems) having the system also being part of your the large ~15T pool, I personally rather would do one small pool (mirror, if even) ~128G...256G for the system, and the rest of those six 5T drives, having one partition each, and doing one large raidz2 pool, and keep all data, including /home on that one separate from the system.
But, as I said, that's just my suggestion how I would do it; and may you have your reason to do it your way.
 
Maturin said:
That's nice.
And your question was?
😁

I guess you want feedback on what you're doing.
You can do so, of course, but may I suggest you may consider if it was also better if you:
- use gpt-labels instead of partitions into pools
- instead of adding the third mirror pool to do a raidz2 over all your disks
Click to expand...
Hi. Thanks for the response.

Yeah I wanted to sanity check that I won't be doing anything stupid with the steps as detailed.

I wanted to add a new mirror as I think it will be the fastest/least effort way to add more space.

Presumably to convert to a raidz2 I could do something like:
  1. Create a RAID0 pool using the two new disks giving me ~12TB of space.
  2. Copy the data from the existing pool onto the new RAID0 pool.
  3. Destroy the original pool.
  4. Create a RAIDZ2 using the original 4 disks.
  5. Copy the data back from the RAID0 pool to the RAIDZ2 pool.
  6. Destroy the RAID0 pool.
  7. Add the two new disks to the RAIDZ2 pool making it ~24TB in size.
 
i.am.the.problem said:
the RAIDZ2 pool making it ~24TB in size
Click to expand...
Exactly.
Personally (again, I don't know your actual situation in detail) I probably would do it otherwise.
Of course, you can do it like you suggest. To me that's kind of an operation at the open heart; feels tricky, and dangerous to me (Don't drink beer, while you do it! 😁) - I first would securely backup all data (all the resilvering will also cost you time, too), and give the whole shebang a fresh, clean start:
- one drive for the system (maybe your nvd, you're using at the moment for 'cache', if it's is big enough; maybe a small mirror, as I said above.)
- create one partition on all 6 5Ts with lables, and create one large raidz2 pool.

But this just as an idea (of course you somehow need to bu ~11T to something first...)

Edit: (first read completely, than answer [I mean me] :cool:)
Yeah, this may work, if the 4 drives raidz2 pool already provides the ~11T (I didn't checked that; but you better, before start the stunt.)

Anyway I strongly recommend to use labels.
 
Maturin said:
Anyway I strongly recommend to use labels.
Click to expand...
Absolutely! That's the best way to prevent silly mistakes, which are the #1 through #10 cause of data loss.

About your configuration: You current are using 4 disks, which are giving you 2 disk's worth of capacity (a mirror pair of stripes). Soon you'll have 6 disks giving you 3 disk's worth of capacity. The good news about mirrors and stripes is that they are usually very fast to do IO. The bad news about mirrors is that they can only tolerate failure of 1 disk (or in general one disk error); the moment you get a second disk failure, your data is at risk. Given that you are also striping, in some cases two disk failures are OK, but you can't bank on that.

Let's contrast that with a parity based system: With RAID-Z on 6 disks, you get 5 disk's worth of capacity, and can tolerate a single disk failure. With an array that large and modern rdisks, I would go for RAID-Z2, which gives you 4 disk's worth of capacity, and can tolerate full failure of any two disks. Matter-of-fact with a 4-disk system, RAID-Z2 would have given you 2 disk's worth of capacity, and the guarantee of handling full failure of any two disks: same capacity as striped mirrors, but better fault tolerance.

The performance of such a system will be less, but for most home users and small servers, IO performance is irrelevant.

In all this, I ignored your boot/root partition, as it is small.
 
The server was originally set up with a stripe of mirrors for the fast IO performance.


I agree though that what I'm suggesting is all very risky. Its not an idea situation at all.


I'm not sure that the server, DELL R540, will boot from the nvme drive.

While all the data is backed up to another offsite server I don't have anything big enough to backup the data to locally and it would take days to copy it all back from the offsite server.
 
i.am.the.problem said:
The server was originally set up with a stripe of mirrors for the fast IO performance.
Click to expand...
If you need the IO performance of mirrors and stripes, then all the RAID-Z... suggestions would need to be evaluated before using them.

i.am.the.problem said:
While all the data is backed up to another offsite server I don't have anything big enough to backup the data to locally and it would take days to copy it all back from the offsite server.
Click to expand...
Sadly, there is no way to migrate from mirror / stripe to RAID-Z... without recreating the pool. And until very recently, it was not possible to change the size of a RAID-Z... pool once created, but that capability was recently added (I have never looked in detail, research it before you commit to it). So in theory, the following strategy would work: (0) Do a good backup (that may already be impractical). (1) Unmirror your current pool, which liberates two disks. You still are storing 2 disk's worth of capacity, but you have NO redundancy at the moment. (2) Using the two new disks you are adding, set up a 4-disk RAID-Z2 pool, which has 2 disk's worth of capacity. (3) Move all data from the old to the new pool. (4) Destroy the old pool, which liberates two more disks. (5) Add those two disks to the new RAID-Z2 pool. Given the complexity of this task, it is high risk and lots of work.
 
If this is kind of a company's server that simply needs to run, then the here suggested "better ways" by ralphbsz and me may no short term solution at the moment, 'cause the ~11T needs to be shoveled at least twice safe -plus the whole system needs to be installed, configured... - nothing I see could be done within a single night (except you have really fast drives :cool:) - but it really should be considered anyway.

I sense/presume/guess this server was once started as a simple two drive mirror. And when its capacity was reached, quickly and simply another mirror was added. Okay.
Now the capacity needs to be increased again.
This can be done quickly by the process described within the very first post by simply adding another mirror, again.
But in my eyes this will lead to several issues in the future, and need to be considered to be changed - asap.

As ralphbsz already pointed out in detail you're not only wasting a lot of capacity, you obviously need, but further more lower the safety you originally wanted to gain by doing raid in the long term; 'cause after all you have a stripe - of two disk mirrors, but a stripe.
A stripe means all capacity is summed up into one pool, but no redundancy. The loss of one single volume means the whole stripe is killed. All redundancy you have at the moment is within each mirror, only, but no overall redundancy over the whole amount of drives. This means, no matter how many mirrors you may add to your stripe in the future, losing only one volume (mirror) means everything's gone. All redundancy - saftey - you have at the moment, and still will have in the future when continue this way, is the redundancy of a single two drive mirror, only.
All you can afford is to lose one single drive, no matter how large your pool grows. If the second drive you lose, before the first is replaced, is from the same mirror: Game Over

The ways we suggested here - to switch to a six drives raidz2 (maybe raidz3) pool, which not only gives you the redundancy of two (or three) drives, doesn't matter which one fails, but also the possibility to simply add more (even single) drives to it later while keeping redundancy, gaining more capacity, plus changing your setup from hardware designators to labels, which brings additional benefits of more flexibilitiy and security - is nothing that can be done overnight - a weekend maybe, if you really versed in what you're doing (and don't do mistakes because of stress) - but are recommended.
 
Thank you for your input and advice.

I've raised the issues with my manager. I'm suggesting that we get three more drives:

2 x 1TB to use in a mirror for a new install of the OS.
1 x 16TB to copy the data onto while the reinstall happens and the RAIDZ2 with the 6x6TB drives is brought online.

The server can tolerate some downtime. Especially todo it right.

Is the following what you mean when you say to use labels?

Code: 
gpart add -a 1M -t freebsd-zfs -l disk1 /dev/da0
gpart add -a 1M -t freebsd-zfs -l disk2 /dev/da1
zpool create zpool mirror gpt/disk1 gpt/disk2
 
That's exactly kind of what I thought was the best solution.
This way you'll get a clean, fresh, server, you can better rely on in the future; while you have 6 disks you want to add to the pool, also may consider, and decide for yourself of course, if you want to do raidz2, or raidz3; second you'd gain more redundancy for the price you lose a bit capacity... need to see for yourself.

Also you can do a clean installation of 14.2, and spare the update.

i.am.the.problem said:
gpart add -a 1M -t freebsd-zfs -l disk1 /dev/da0 gpart add -a 1M -t freebsd-zfs -l disk2 /dev/da1 zpool create zpool mirror gpt/disk1 gpt/disk2
Click to expand...
That seems to be right - if you see this as an example for a two drive mirror.

Edit:
Don't do the 6 drives pool as a mirror! You'll only gain the capacity of a single volume (5T) (for the price all other five may fail - most redundancy, less capacity; I don't think that's what you want.)
Do a raid!
I recommend raidz2, or raidz3, but which you need in detail you need to decide for yourself, of course.
You'll find formulars [or even calculators] on the internet, that give you the amount of pool's capacity you get by a certain number of volumes you add, their capacity, and the chosen raid-type.
 
Thought so. But want to be sure. You don't seem the kind of guy who thoughtlessly simply copy-pastes the first thing he grabs from the internet. But we are in an open, public forum; who knows who else reads this; I don't want this fish coming back at me :cool:
May I also suggest to write a small, simple docu on what you've done to create the new pool, above all how to replace, and add disks - your successor may be thankful.
 
You must log in or register to reply here.
Back
Top