Solved Need help on RAID1 modification

Switch2BSD

Member

Reaction score: 12
Messages: 81

Hello,
I have a machine with 4 HDD drives on pre-installed FreeBSD 12.2 with RAID1 for all 4 disks. Now I would like to change RAID configuration into 2 couples of RAID1, ada0 + ada1, and ada2+ada3, to have more disk space. Please advise me, a newbie on FreeBSD, the ways to do that.
Thanks!
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

Please tell the output of
  • uname -aU
  • zpool list -v
  • gmirror list
  • graid list
  • gvinum list
  • If cannot be seen from the output of the above, how large are the disks?
Please be aware that using 2-way mirrors is commonly considered dangerous in a professional environment; for use @home and/or for non-vital data, it's ok if the disks are small enough, i.e. < 9TB.
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Because this is a pre-installed OS, but possibly it uses geom as I see this in boot loader:
# nano /boot/loader.conf
Code:
geom_mirror_load="YES"                                                                                                  
kern.geom.label.disk_ident.enable="0"                                                                                   
kern.geom.label.gptid.enable="0"                                                                                        
zfs_load="YES"
and system use zroot zfs
# df -h
Code:
zroot/ROOT/default     64G    2.7G     62G     4%    /                                                                  
devfs                 1.0K    1.0K      0B   100%    /dev                                                               
zroot/home            205G    1.2G    204G     1%    /home                                                              
zroot                 1.7T     88K    1.7T     0%    /zroot
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

FreeBSD 12.2-RELEASE-p1 GENERIC amd64 1202000

Code:
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT                                
zroot       1.81T  20.3G  1.79T        -         -     0%     1%  1.00x  ONLINE  -                                        
mirror    1.81T  20.3G  1.79T        -         -     0%  1.09%                                                            
ada0p4      -      -      -        -         -      -      -                                                            
ada1p4      -      -      -        -         -      -      -                                                            
ada2p4      -      -      -        -         -      -      -                                                            
ada3p4      -      -      -        -         -      -      -
and

Code:
Geom name: swap
State: COMPLETE
Components: 4
Balance: load
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 1648000086
Type: AUTOMATIC
Providers:
1. Name: mirror/swap
   Mediasize: 4194303488 (3.9G)
   Sectorsize: 512
   Mode: r1w1e0
Consumers:
1. Name: ada0p3
   Mediasize: 4194304000 (3.9G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2097152
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: NONE
   GenID: 0
   SyncID: 1
   ID: 604727882
2. Name: ada1p3
   Mediasize: 4194304000 (3.9G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2097152
   Mode: r1w1e1
   State: ACTIVE
   Priority: 1
   Flags: NONE
   GenID: 0
   SyncID: 1
   ID: 158265821
3. Name: ada2p3
   Mediasize: 4194304000 (3.9G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2097152
   Mode: r1w1e1
   State: ACTIVE
   Priority: 2
   Flags: NONE
   GenID: 0
   SyncID: 1
   ID: 3831199638
4. Name: ada3p3
   Mediasize: 4194304000 (3.9G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2097152
   Mode: r1w1e1
   State: ACTIVE
   Priority: 3
   Flags: NONE
   GenID: 0
   SyncID: 1
   ID: 2310490028
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

graid list: returns null
and
gvinum list as is:
Code:
0 drives:

0 volumes:

0 plexes:

0 subdisks:
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,719
Messages: 39,309

This is new right? I mean there's no data on them yet? I would suggest actually using ZFS for this. You're already using ZFS for the root filesystem, might as well use ZFS for the data too.

Where's zroot stored on though? Please post the output from gpart show and zpool status. And please use [code] not [quote] to post the output.
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Yes, it is new with a very few data (not important). I see that I am using ZFS.
I would like to detach ada2 and ada3 from current raid and create new raid for ada2 and ada3 (raid1 of 4 into 2 raid1 of 2).
Then I should run
gpart destroy ada2/3
gmirror remove "geom_name" ada2/ada3
But I see only "swap" geom_name, but not others.
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

glabel list:
Code:
Geom name: ada0p1
Providers:
1. Name: gpt/efiboot0
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1600
   length: 819200
   index: 0
Consumers:
1. Name: ada0p1
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0

Geom name: ada0p1
Providers:
1. Name: msdosfs/EFI
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1600
   length: 819200
   index: 0
Consumers:
1. Name: ada0p1
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0

Geom name: ada0p2
Providers:
1. Name: gpt/gptboot0
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 839680
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1024
   length: 524288
   index: 0
Consumers:
1. Name: ada0p2
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 839680
   Mode: r0w0e0

Geom name: ada1p1
Providers:
1. Name: gpt/efiboot1
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1600
   length: 819200
   index: 0
Consumers:
1. Name: ada1p1
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0

Geom name: ada1p2
Providers:
1. Name: gpt/gptboot1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 839680
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1024
   length: 524288
   index: 0
Consumers:
1. Name: ada1p2
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 839680
   Mode: r0w0e0

Geom name: ada2p1
Providers:
1. Name: gpt/efiboot2
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1600
   length: 819200
   index: 0
Consumers:
1. Name: ada2p1
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0

Geom name: ada2p2
Providers:
1. Name: gpt/gptboot2
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 839680
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1024
   length: 524288
   index: 0
Consumers:
1. Name: ada2p2
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 839680
   Mode: r0w0e0

Geom name: ada3p1
Providers:
1. Name: gpt/efiboot3
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1600
   length: 819200
   index: 0
Consumers:
1. Name: ada3p1
   Mediasize: 819200 (800K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0

Geom name: ada3p2
Providers:
1. Name: gpt/gptboot3
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 839680
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1024
   length: 524288
   index: 0
Consumers:
1. Name: ada3p2
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 839680
   Mode: r0w0e0
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

gpart show
Code:
=>        40  3907029088  ada0  GPT  (1.8T)
          40        1600     1  efi  (800K)
        1640        1024     2  freebsd-boot  (512K)
        2664        1432        - free -  (716K)
        4096     8192000     3  freebsd-swap  (3.9G)
     8196096  3898832896     4  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)

=>        40  3907029088  ada1  GPT  (1.8T)
          40        1600     1  efi  (800K)
        1640        1024     2  freebsd-boot  (512K)
        2664        1432        - free -  (716K)
        4096     8192000     3  freebsd-swap  (3.9G)
     8196096  3898832896     4  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)

=>        40  3907029088  ada2  GPT  (1.8T)
          40        1600     1  efi  (800K)
        1640        1024     2  freebsd-boot  (512K)
        2664        1432        - free -  (716K)
        4096     8192000     3  freebsd-swap  (3.9G)
     8196096  3898832896     4  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)

=>        40  3907029088  ada3  GPT  (1.8T)
          40        1600     1  efi  (800K)
        1640        1024     2  freebsd-boot  (512K)
        2664        1432        - free -  (716K)
        4096     8192000     3  freebsd-swap  (3.9G)
     8196096  3898832896     4  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)

zpool status
Code:
  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p4  ONLINE       0     0     0
        ada1p4  ONLINE       0     0     0
        ada2p4  ONLINE       0     0     0
        ada3p4  ONLINE       0     0     0

errors: No known data errors
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

OT: depending on the desired use case of your system, you may want to reserve more than 4GB (mirrored) swap. The traditional rule of thumb is swap = 2 * RAM. How much RAM does your box have? ( bc -le "scale=2;`sysctl -n hw.physmem`/(1024*1024*1024)" -e quit)
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Thanks Mjölnir,​

I have 32 GB RAM, but swap of 4GB is okay, because I see the complication to resize partitions too. I think I could use swap file if I need, instead of swap partition (but I have not tried that).
Code:
# sysctl -n hw.physmem
34208407552
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

If you go ZFS you can comfortably add a ZFS volume to swapspace; the swap on the dedicated partition is still needed as dump device. Beware that a minidump uses approx. 1/3 of kernel memory, and you can leave out ZFS metadata (vfs.zfs.zio.exclude_metadata="1" in loader.conf(5)). So the default is usually fine, but depending on your use case (e.g. developer workstation) you may want to have a dump partition of a size >= 1/3 * RAM.
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

The first thing I do with FreeBSD is to follow this link: https://github.com/churchers/vm-bhyve to create a testing virtual machine, then well, it works like a charm. Love much the this virtualization.
I used to know mdadm, but
Code:
pkg search mdadm
returns empty. It is clear that ZFS needs more time to read documents/handbooks.
Code:
bc -le (scale=2;`sysctl -n hw.physmem`/(1024*1024*1024)" -e quit
31.85
 

Zirias

Son of Beastie

Reaction score: 1,701
Messages: 2,870

OT: depending on the desired use case of your system, you may want to reserve more than 4GB (mirrored) swap. The traditional rule of thumb is swap = 2 * RAM. How much RAM does your box have? ( bc -le "scale=2;`sysctl -n hw.physmem`/(1024*1024*1024)" -e quit)
This recommendation doesn't apply to today's RAM sizes. It's a good rule of thumb for roughly up to 4GB of RAM.

With much more RAM, you will want to cap off swap size at maybe 8 to 16 GB. This is because swapping in and out is expensive (slow), and you only want swap to be used as a temporary storage for memory pages that are indeed not currently needed. You don't want your system to reach a state where memory that is frequently accessed gets swapped. If you ever hit such a situation, having the OOM killer as a last resort is the better alternative.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,719
Messages: 39,309

Ok, this isn't as bad as I though it would be. I was fearing this was a ZFS filesystem on a gmirror(8), which is the worst possible way of installing this. You can remove that geom_mirror loading, you don't need it.

This is actually a 4-way mirror:
Code:
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p4  ONLINE       0     0     0
        ada1p4  ONLINE       0     0     0
        ada2p4  ONLINE       0     0     0
        ada3p4  ONLINE       0     0     0
So you can just remove the two disks from it. Then add them as a mirrored pair to the existing zroot pool.
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

Now the main argument against hardware-assisted RAID (via graid(8)) is that you can't use that on another machine that uses another chipset with different metadata format. There have even been cases in the past where the metadata format changed even within a chipset revision... That's one good reason why the ZFS wizzards designed ZFS RAID to be software-only. You have to decide whether you want to use hardware-assisted RAID or software-only, and decide whether you want to use gmirror(8) or ZFS-mirror. For the mirrored swap/dump partitions, you'll want to use gmirror(8) anyway, but IMHO there are good reasons to use ZFS RAID for all the data. You have to take that decision, we can not know your use case & preferences.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,719
Messages: 39,309

But I don't know why I just see one partition (swap) is for RAID1
There's no other way to mirror swap. So the installer uses gmirror(8) for this. gmirror(8) can be used as "software" RAID, or with so-called "fake" RAID controllers. Fake RAID controllers implement RAID partially in hardware and partially in software.

Other partitions are on RAID1 or not?
Yes, ZFS's variant. RAID1 is a mirror.
Code:
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p4  ONLINE       0     0     0
        ada1p4  ONLINE       0     0     0
        ada2p4  ONLINE       0     0     0
        ada3p4  ONLINE       0     0     0
You might be more familiar with 2-way mirrors, but it's actually possible to have a 3-way or 4-way mirror. Each 'side' is mirrored to every other disk, regardless of many 'copies'. You just need 2 as a minimum.
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Should I detach 2 disks as following?
Code:
# gpart destroy -F ada2 && gpart destroy -F ada3
# gmirror remove mirror-0 ada2 && gmirror remove mirror-0 ada3
# gmirror clear ada2 && gmirror clear ada3

Then I create new mirror for them:
Code:
# gmirror label -v mirror-data /dev/ada2 /dev/ada3

Right?
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

IMHO now your target configuration is to have RAID-10 for both swap (via gmirror(8)) and data (via ZFS). You'll be taking out two partitions of two disks out of the resp. swap & ZFS RAID-1, and put them into a new mirror each. Then add these 2nd mirrors to swap, and the mirrored data partitions to the ZFS pool; both do automagically stripe these additional mirrors with the existing mirror.
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

The first thing I do with FreeBSD is to follow this link: https://github.com/churchers/vm-bhyve to create a testing virtual machine, then well, it works like a charm. Love much the this virtualization.
I used to know mdadm, but
Code:
pkg search mdadm
returns empty.
mdadm is L*x stuff. apropos mdm:
Code:
ppmdmkfont(0, 1) - Create Ppmdfont "standard".
nmdm(4) - nullmodem terminal driver
mdmfs, mount_mfs(8) - configure and mount an in-memory file system using the md 4 driver or the tmpfs 5 filesystem
& whereis mdmfs: mdmfs: /sbin/mdmfs /usr/share/man/man8/mdmfs.8.gz /usr/src/sbin/mdmfs
It is clear that ZFS needs more time to read documents/handbooks.
Right you are. It will very likely also tell you how to succeed with the remaining tasks. Else keep asking here. Since you don't seem to have any valuable data on that box yet, you can do no harm.
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Nice thanks to @Mjölnir SirDice Zirias !
After your questions about my machine configs, and after reading https://www.freebsd.org/cgi/man.cgi?zpool(8)
I have made (and thought it is) successfully:
Code:
# zpool detach zroot ada3p4
# zpool detach zroot ada2p4
# zpool create ztoor mirror ada2p4 ada3p4
# zfs create ztoor/data
# zfs set mountpoint=/data ztoor/data
Now:
Code:
# zpool list
returns
Code:
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot  1.81T  20.3G  1.79T        -         -     0%     1%  1.00x  ONLINE  -
ztoor  1.81T   158K  1.81T        -         -     0%     0%  1.00x  ONLINE  -

is it ZFS RAID-1, right?

Before, I don't know that my machine using RAID1 by 2 tools
But now, I understand that it was using 2 tools, "swap" on top of mirror by gmirror, and "zroot" on top of mirror by zfs raid1.
I am going to read for configuring of RAID10 for swap.
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

When I understand that there are 2 tools for my RAID1 configs, it's clear to look for documents/handbook.
Code:
# gmirror list
# gmirror status
# gmirror deactivate swap ada3p3
# gmirror deactivate swap ada2p3
# gmirror label -v -n -b load swap2 ada2p3 ada3p3
# nano /etc/fstab
# swapon -a

swapon: adding /dev/mirror/swap2 as swap device

Until now, I have swap parts: 4 GB + 4 GB = 8 GB, still raid1 (gmirror).
Should I let them work or mirror them into one swap? please advice me and tell me benefits of let them there vs mirror them (swaps).
 
Top