Solved Need help on RAID1 modification

OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Another question after all of above things is why my /data is empty although these 2 drives were mirrored before.
I have checked for /data have nothing inside. Which step(s) got wrong?
 

im

Active Member

Reaction score: 46
Messages: 109


I am going to read for configuring of RAID10 for swap.
Just use two mirrored swap partition instead, it is correct.
You may build RAID10 as gstripe of two gmirrors, but it is no matter for swap.
Chech the swapinfo output.

# gmirror deactivate swap ada3p3 # gmirror deactivate swap ada2p3
I usually use gmirror remove name prov instead of deactivate
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Yes, thanks. But perhaps a better write latency needed once swap going used.
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

The kernel automagically stripes all swap devices. From swapon(8):
Code:
     [...] added automatically when the system goes multi-user.  Swap devices use a
     fixed interleave; the maximum number of devices is unlimited.  There is
     no priority mechanism.
I.e. in effect, two mirrored swap devices of equal size act as if they're a RAID-10.

Now for your data on ZFS: IMHO I would want to have one ZFS pool (RAID-10), not two RAID-1. I.e. you do not create a new zpool, but instead
So you can just remove the two disks from it. Then add them as a mirrored pair to the existing zroot pool.
So after you destroyed that superfluous ztoor pool, just do zpool add zroot mirror ada2p4 ada3p4 to turn your zroot ZFS pool into a RAID-10 (Example #5 in RTFM zpool(8)). Final note:
There's no other way to mirror swap. So the installer uses gmirror(8) for this. gmirror(8) can be used as "software" RAID,
Strictly speaking, this is not true, since you can also use
or with so-called "fake" RAID controllers. Fake RAID controllers implement RAID partially in hardware and partially in software.
a graid(1) hardware-assisted RAID-1 mirror (and other RAID levels), with the constraints mentioned above.
 

VladiBG

Daemon

Reaction score: 582
Messages: 1,260

If you are going to use UEFI boot you should increase your EFI partition size otherwise when you update to 13.x in the future you will be unable to install the new loader.efi as it's size is larger. So consider to increase your EFI to 200MB.
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

You might be more familiar with 2-way mirrors, but it's actually possible to have a 3-way or 4-way mirror. Each 'side' is mirrored to every other disk, regardless of many 'copies'. You just need 2 as a minimum.
Fastidious sidenote: the term "every other" in english means "every 2nd", i.e. 1-3-5-..., 2-4-6-... etc.pp. E.g. RTFM crontab(5) & search for "0-23/2". IMHO what you meant was "all other".
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

If you are going to use UEFI boot you should increase your EFI partition size otherwise when you update to 13.x in the future you will be unable to install the new loader.efi as it's size is larger. So consider to increase your EFI to 200MB.
Thank for prompting me.
This is a remote machine that I have not IPMI/KVM access, it was installed by a pre-configured script with FreeBSD 11, and I didnot know the way it boots (legacy or efi). I would be able to boot it into rescue mode and then modify partition size. How could it be difficulty level?
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Now for your data on ZFS: IMHO I would want to have one ZFS pool (RAID-10), not two RAID-1. I.e. you do not create a new zpool, but instead

So after you destroyed that superfluous ztoor pool, just do zpool add zroot mirror ada2p4 ada3p4 to turn your zroot ZFS pool into a RAID-10 (Example #5 in RTFM zpool(8)).
Nice thanks for guides.
I do zpool add zroot mirror ada2p4 ada3p4, zroot has now double size, it means that I have RAID10 automagically or not?
Do I need to create a new zpool, or need to strip 2 mirrors inside zroot?

Fastidious sidenote: the term "every other" in english means "every 2nd", i.e. 1-3-5-..., 2-4-6-... etc.pp. E.g. RTFM crontab(5) & search for "0-23/2". IMHO what you meant was "all other".
Sorry for my english. I think there is no differences if mirroring {0,1} and {2,3} vs mirroring {0,2} and {1,3}, right?
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Currently, zpool status shows:
Code:
  pool: zroot
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p4  ONLINE       0     0     0
        ada1p4  ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        ada2p4  ONLINE       0     0     0
        ada3p4  ONLINE       0     0     0

errors: No known data errors

I am reading/searching documents/guides to choose beetween gstripe or zpool raidz to continue..
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

You're done. SUCCESS!!! ...except for that EFI partition size, which would mean you have to reinstall to fix that... :(
 

Zirias

Son of Beastie

Reaction score: 1,701
Messages: 2,870

You're done. SUCCESS!!! ...except for that EFI partition size, which would mean you have to reinstall to fix that... :(
No need to reinstall. If you're only booting in UEFI mode, the freebsd-boot partition is unnecessary. There's also some free space after that. Right now, on 13, 1MB is enough to hold the loader.efi.

So, what I'd do there (for every disk) is roughly this:
Code:
gpart delete -i 1 ada0
gpart delete -i 2 ada0
gpart add -t efi -i 1 -b 40 ada0
newfs_msdos /dev/ada0p1
mount -t msdosfs /dev/ada0p1 /boot/efi
mkdir -p /boot/efi/efi/boot
cp /boot/loader.efi /boot/efi/efi/boot/BOOTx64.efi
umount /boot/efi
 

VladiBG

Daemon

Reaction score: 582
Messages: 1,260

You can use sysctl machdep.bootmethod and see if you are booting using BIOS or UEFI.
Don't delete any partition for now.

Zirias EFI partition need to be FAT32 for non removable disks and FAT16 for removable. For non removable disk the cluster size for FAT32 should be 1 (512bytes) otherwise you can't format as it's below 260MB
That's why during format of EFI must be done with newfs_msdos -F 32 -c 1 /dev/da0p1
13.3 File System Format
EFI encompasses the use of FAT32 for a system partition, and FAT12 or FAT16 for removable media
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

You're done. SUCCESS!!! ...except for that EFI partition size, which would mean you have to reinstall to fix that... :(

Thank you, great papa!
I also intend to re-install OS next weekends, but the pre-configured template script (auto deploy of FreeBSD 11) will be the same, and the EFI will be the same for it size.
Any suggestion for other methods to install (no KVM/IPMI, no netboot.xyz, but having a rescue disk)?
 

Zirias

Son of Beastie

Reaction score: 1,701
Messages: 2,870

Any suggestion for other methods to install (no KVM/IPMI, no netboot.xyz, but having a rescue disk)?
Is the issue that this machine doesn't have a VGA console? If so, does it have a serial port?
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

You can use sysctl machdep.bootmethod and see if you are booting using BIOS or UEFI.
Code:
# sysctl machdep.bootmethod                                                       
machdep.bootmethod: UEFI

Thank you,
I will have a try. A great chance to try. (I am a medical doctor, so I have not enough time to enjoy all, but FreeBSD - the more and more attractive OS to me, especially bhyve).
About re-installation, I have a chance to install v11 on other machine via netboot.xyz, its mechanism perhaps would apply for installing from rescue cd, right? I see from the panel they have an option to boot into rescue mode. No VGA console or serial port (pciconf)
 

Zirias

Son of Beastie

Reaction score: 1,701
Messages: 2,870

[FONT=monospace]Zirias[/FONT] EFI partition need to be FAT32 for non removable disks and FAT16 for removable.
I haven't seen any UEFI implementation yet that would enforce this (and it would be pretty unexpected as UEFI must be able to handle all these FAT types anyways and they all use the same partition GUID). I'm pretty sure this is informative only. There's no way to format an 800k or 1M partition as FAT32 anyways.
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

Thank you, great papa!
De nada.
I also intend to re-install OS next weekends, but the pre-configured template script (auto deploy of FreeBSD 11) will be the same, and the EFI will be the same for it size.
Any suggestion for other methods to install (no KVM/IPMI, no netboot.xyz, but having a rescue disk)?
Serial console. See boot.config(5). Very oldschool, though.
 

Zirias

Son of Beastie

Reaction score: 1,701
Messages: 2,870

BTW, if an UEFI BIOS would refuse a FAT12 ESP, just because it's found on a fixed disk, a lot of FreeBSD 11 and 12 machines would be unable to boot with UEFI:
Code:
# file /var/jail/.release/12.2/boot/boot1.efifat 
/var/jail/.release/12.2/boot/boot1.efifat: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "BSD4.4  ", root entries 512, sectors 1600 (volumes <=32 MB), sectors/FAT 5, sectors/track 63, heads 1, serial number 0xbd4111ee, label: "EFISYS     ", FAT (12 bit), followed by FAT
 

Zirias

Son of Beastie

Reaction score: 1,701
Messages: 2,870

You can read the entire discussion here: https://reviews.freebsd.org/D6935
So, someone found a BIOS that expects FAT32 and nothing else. I wonder whether this BIOS would accept FAT12 if found on removable media, but still, this is very uncommon. If you need FAT32 indeed, you need a much larger partition. It definitely doesn't apply for the OP who seems to have the old boot1.efifat (FAT12) which boots fine on his system.
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

No need to reinstall. If you're only booting in UEFI mode, the freebsd-boot partition is unnecessary. There's also some free space after that. Right now, on 13, 1MB is enough to hold the loader.efi.

So, what I'd do there (for every disk) is roughly this:
Code:
gpart delete -i 1 ada0
gpart delete -i 2 ada0
gpart add -t efi -i 1 -b 40 ada0
newfs_msdos /dev/ada0p1
mount -t msdosfs /dev/ada0p1 /boot/efi
mkdir -p /boot/efi/efi/boot
cp /boot/loader.efi /boot/efi/efi/boot/BOOTx64.efi
umount /boot/efi
At first, I will try this method. But efi will arrive to 2 MB, not 200MB. If succeeded, I will resize swap partition to have it arrived to 200MB.
However,
# mount -t msdosfs /dev/ada0p1 /boot/efi
mount: /boot/efi: No such file or directory

Code:
# ls /boot
total 4377
drwxr-xr-x  10 root  wheel    65B Mar 23 11:00 .
drwxr-xr-x  23 root  wheel    30B Mar 27 20:05 ..
-r--r--r--   1 root  wheel   3.5K Dec 29 14:48 beastie.4th
-r--r--r--   1 root  wheel   8.0K Dec 29 14:48 boot
-r--r--r--   1 root  wheel   512B Jun 13  2020 boot0
-r--r--r--   1 root  wheel   512B Jun 13  2020 boot0sio
-r--r--r--   1 root  wheel   512B Jun 13  2020 boot1
-r-xr-xr-x   1 root  wheel    92K Dec 29 14:48 boot1.efi
-r--r--r--   1 root  wheel   800K Dec 29 14:48 boot1.efifat
-r--r--r--   1 root  wheel   7.5K Dec 29 14:48 boot2
-r--r--r--   1 root  wheel   2.1K Dec 29 14:48 brand-fbsd.4th
-r--r--r--   1 root  wheel   2.7K Dec 29 14:48 brand.4th
-r--r--r--   1 root  wheel   1.2K Dec 29 14:48 cdboot
-r--r--r--   1 root  wheel   6.1K Dec 29 14:48 check-password.4th
-r--r--r--   1 root  wheel   2.1K Dec 29 14:48 color.4th
drwxr-xr-x   2 root  wheel     3B Dec 29 14:48 defaults
-r--r--r--   1 root  wheel   4.0K Dec 29 14:48 delay.4th
-r--r--r--   1 root  wheel   829B Dec 29 14:48 device.hints
drwxr-xr-x   5 root  wheel     5B Mar 23 11:00 dtb
-r--r--r--   1 root  wheel   1.6K Dec 29 14:48 efi.4th
-rw-------   1 root  wheel   4.0K Mar 27 20:05 entropy
drwxr-xr-x   2 root  wheel     2B Jun 13  2020 firmware
-r--r--r--   1 root  wheel   3.6K Dec 29 14:48 frames.4th
-r--r--r--   1 root  wheel    55K Dec 29 14:48 gptboot
-r-xr-xr-x   1 root  wheel    88K Dec 29 14:48 gptboot.efi
-r--r--r--   1 root  wheel   800K Dec 29 14:48 gptboot.efifat
-r--r--r--   1 root  wheel    88K Dec 29 14:48 gptzfsboot
-r--r--r--   1 root  wheel    12K Dec 29 14:48 isoboot
drwxr-xr-x   2 root  wheel   903B Mar 23 11:00 kernel
drwxr-xr-x   2 root  wheel   899B Mar 23 11:00 kernel.old
-r-xr-xr-x   3 root  wheel   364K Dec 29 14:48 loader
-r--r--r--   1 root  wheel   7.3K Dec 29 14:48 loader.4th
-rw-r--r--   1 root  wheel   168B Mar 27 11:57 loader.conf
-r-xr-xr-x   2 root  wheel   478K Dec 29 14:48 loader.efi
-r--r--r--   1 root  wheel   468B Dec 29 14:48 loader.rc
-r-xr-xr-x   1 root  wheel   320K Dec 29 14:48 loader_4th
-r-xr-xr-x   1 root  wheel   417K Dec 29 14:48 loader_4th.efi
-r-xr-xr-x   3 root  wheel   364K Dec 29 14:48 loader_lua
-r-xr-xr-x   2 root  wheel   478K Dec 29 14:48 loader_lua.efi
-r-xr-xr-x   1 root  wheel   272K Dec 29 14:48 loader_simp
-r-xr-xr-x   1 root  wheel   358K Dec 29 14:48 loader_simp.efi
-r--r--r--   1 root  wheel   3.0K Dec 29 14:48 logo-beastie.4th
-r--r--r--   1 root  wheel   2.6K Dec 29 14:48 logo-beastiebw.4th
-r--r--r--   1 root  wheel   2.2K Dec 29 14:48 logo-fbsdbw.4th
-r--r--r--   1 root  wheel   2.6K Dec 29 14:48 logo-orb.4th
-r--r--r--   1 root  wheel   2.3K Dec 29 14:48 logo-orbbw.4th
drwxr-xr-x   2 root  wheel    17B Dec 29 14:48 lua
-r--r--r--   1 root  wheel   512B Jun 13  2020 mbr
-r--r--r--   1 root  wheel   9.0K Dec 29 14:48 menu-commands.4th
-r--r--r--   1 root  wheel    35K Dec 29 14:48 menu.4th
-r--r--r--   1 root  wheel   6.2K Dec 29 14:48 menu.rc
-r--r--r--   1 root  wheel    18K Dec 29 14:48 menusets.4th
drwxr-xr-x   2 root  wheel     2B Jun 13  2020 modules
-r--r--r--   1 root  wheel   512B Jun 13  2020 pmbr
-r--r--r--   1 root  wheel   366K Dec 29 14:48 pxeboot
-r--r--r--   1 root  wheel   2.7K Dec 29 14:48 screen.4th
-r--r--r--   1 root  wheel   2.6K Dec 29 14:48 shortcuts.4th
-r--r--r--   1 root  wheel    35K Dec 29 14:48 support.4th
-r--r--r--   2 root  wheel   415K Dec 29 14:48 userboot.so
-r--r--r--   1 root  wheel   348K Dec 29 14:48 userboot_4th.so
-r--r--r--   2 root  wheel   415K Dec 29 14:48 userboot_lua.so
-r--r--r--   1 root  wheel   3.0K Dec 29 14:48 version.4th
drwxr-xr-x   2 root  wheel     3B Mar 27 20:05 zfs
-r--r--r--   1 root  wheel   257K Dec 29 14:48 zfsboot
-r-xr-xr-x   3 root  wheel   364K Dec 29 14:48 zfsloader
 

Zirias

Son of Beastie

Reaction score: 1,701
Messages: 2,870

At first, I will try this method. But efi will arrive to 2 MB, not 200MB. If succeeded, I will resize swap partition to have it arrived to 200MB.
There's no need, as long as your BIOS happily accepts the small and FAT12-formatted partition. And it obviously does: it boots correctly even from the 800k sized one. All you need in the ESP for booting FreeBSD is loader.efi, and that's still well below 1MB in FreeBSD 13.
However,
Code:
 # mount -t msdosfs /dev/ada0p1 /boot/efi 
mount: /boot/efi: No such file or directory
I don't know when exactly /boot/efi was introduced, but it's just an empty directory serving as the mountpoint for the ESP, so just create it: mkdir /boot/efi.
 
OP
S

Switch2BSD

Member

Reaction score: 12
Messages: 81

Now, gpart show shows:
Code:
=>        40  3907029088  ada0  GPT  (1.8T)
          40        4056     1  efi  (2.0M)
        4096     8192000     3  freebsd-swap  (3.9G)
     8196096  3898832896     4  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)

=>        40  3907029088  ada1  GPT  (1.8T)
          40        4056     1  efi  (2.0M)
        4096     8192000     3  freebsd-swap  (3.9G)
     8196096  3898832896     4  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)

=>        40  3907029088  ada2  GPT  (1.8T)
          40        4056     1  efi  (2.0M)
        4096     8192000     3  freebsd-swap  (3.9G)
     8196096  3898832896     4  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)

=>        40  3907029088  ada3  GPT  (1.8T)
          40        4056     1  efi  (2.0M)
        4096     8192000     3  freebsd-swap  (3.9G)
     8196096  3898832896     4  freebsd-zfs  (1.8T)
  3907028992         136        - free -  (68K)
 

Zirias

Son of Beastie

Reaction score: 1,701
Messages: 2,870

Looks fine. You don't have a picky UEFI BIOS insisting on FAT32, otherwise the old 800k partition wouldn't have been bootable. And loader.efi in FreeBSD 13 is still < 900k (and was a lot smaller on 12), so 2MB should be enough for a few major releases. If this is ever an issue, you can shrink swap to make more room for the ESP.
 
Top