Gmirror before Installation

M

Mayhem30

Are there any instruction anywhere on how to Gmirror your disks before installation?

I found this how-to guide for UEFI : https://forums.freebsd.org/threads/install-uefi-freebsd-on-gmirror.71904/

Is anyone able to convert the following for non-UEFI?

Code: 
kldload geom_mirror
dd if=/dev/zero of=/dev/ada0 bs=1M  <<< I wipe my disks before use as RAID <<<
dd if=/dev/zero of=/dev/ada1 bs=1M  <<< You better know what your doing here <<<
gmirror label -v gm0 /dev/ada0 /dev/ada1
gpart create -s gpt /dev/mirror/gm0
gpart add -t efi -a4k -s 200M /dev/mirror/gm0
gpart add -t freebsd-ufs /dev/mirror/gm0
newfs_msdos -F 32 -c 1 /dev/mirror/gm0p1
mount -t msdosfs /dev/mirror/gm0p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt
newfs -U /dev/mirror/gm0p2
mount /dev/mirror/gm0p2 /mnt
echo "/dev/mirror/gm0p2 / ufs rw 1 1" >> /tmp/bsdinstall_etc/fstab
echo 'geom_mirror_load="YES"' >> /tmp/bsdinstall_boot/loader.conf
exit
 
There's one in the Handbook:
www.freebsd.org

Chapter 21. GEOM: Modular Disk Transformation Framework

In FreeBSD, the GEOM framework permits access and control to classes, such as Master Boot Records and BSD labels, through the use of providers, or the disk devices in /dev.
www.freebsd.org www.freebsd.org

I also found this useful:
www.ateamsystems.com

HOW TO: Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme - A-Team Systems

HOW TO: Installing FreeBSD with gmirror (software RAID 1) and the GPT partitioning scheme
www.ateamsystems.com www.ateamsystems.com
 
One of the problems with following someone elses instructions are they may not be the best setting for your drives.
Take this as shown in the handbook for gmirror.
-a 4K
Click to expand...
This is fine for actual 4K drives but you need to research your physical drive sectors.
Blindly putting 4K is not a good solution as it will reduce your disk throughput if you have 512 sector drives..
 
Phishfry said:
One of the problems with following someone elses instructions are they may not be the best setting for your drives.
Take this as shown in the handbook for gmirror.
This is fine for actual 4K drives but you need to research your physical drive sectors.
Blindly putting 4K is not a good solution as it will reduce your disk throughput if you have 512 sector drives..
Click to expand...

One of the guides suggested to use "1m" for SSD's - is that the right setting?

Here is what I have so far (untested) :

Does this look right?

Code: 
gmirror load

gmirror label -v gm0 /dev/ada0 /dev/ada1
gpart create -s MBR mirror/gm0
gpart add -t freebsd -a 1m mirror/gm0

gpart create -s BSD mirror/gm0s1
gpart add -t freebsd-ufs  -a 1m       mirror/gm0s1
gpart add -t freebsd-swap -a 1m -s 4g mirror/gm0s1

gpart bootcode -b /boot/mbr mirror/gm0
gpart set -a active -i 1 mirror/gm0
gpart bootcode -b /boot/boot mirror/gm0s1

newfs -t -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt
 
Mayhem30 said:
Code: 
gpart add -t freebsd-ufs  -a 1m       mirror/gm0s1
gpart add -t freebsd-swap -a 1m -s 4g mirror/gm0s1
Click to expand...
I think "gpart add" with no size (-s) parameter will allocate all the space until the end of the slice. You'll wind up with a single "a" partition that takes up the entire mirror, and the second command to allocate swap will fail. You'll have to use the -i parameter to specify the "index" of the partition you are adding. These indexes show up as letters in the partition names e.g., /dev/mirror/gm0s1a will be the partition at index 1. I dunno why gpart(8) doesn't use letters to match the partition device names. Makes it confusing.

I think what you're trying to do is in English "make one 4GB swap partition, and use the rest of the disk for a root partition". If so, the gpart commands you want are:
Code: 
gpart add -t freebsd-swap -a 1m -s 4g -i 2 mirror/gm0s1
gpart add -t freebsd-ufs  -a 1m -i 1       mirror/gm0s1

N.B., You cannot use index 3, aka the "c" partition. That is reserved and always spans the entire slice.
 
Ok, thank you.

My last question is the "1m" part .. is that th correct setting for Samsung 860 Pro 256GB SSD's?
 
I believe you want 4K for those drives.
SSD's tend to lie about their geometry.
Basically you have 3 common sector sizes.
512
4K
8K
Even with a mirror you would not add two together (ie 1024)
Each mirrored disk is independant.
 
Thank you. My final code now looks like this :

Code: 
gmirror load

gmirror label -v gm0 /dev/ada0 /dev/ada1
gpart create -s MBR mirror/gm0
gpart add -t freebsd -a 4k mirror/gm0

gpart create -s BSD mirror/gm0s1
gpart add -t freebsd-swap -a 4k -s 4g -i 2 mirror/gm0s1
gpart add -t freebsd-ufs -a 4k -i 1 mirror/gm0s1

gpart bootcode -b /boot/mbr mirror/gm0
gpart set -a active -i 1 mirror/gm0
gpart bootcode -b /boot/boot mirror/gm0s1

newfs -t -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt
 
That looks very plausible.
Last time I wanted to do this on a high value live physical system, I built a disposable VM to test it first.

I see that you are placing only one primary partition on the disk, that you have turned on TRIM for the file systems, and that soft updates journaling is off.

Here are the notes I kept to clear the metadata, if you need to repeat or test the process:
Code: 
# Get the sector count for /dev/ada0 and divide it by "1024x1024x1024".
# This is the number of GiB on the drive.  Call it NNN.  Subtract 5 => NN5.
# Nuke the start and end of the drive (partition table and gmirror metadata).
dd if=/dev/zero of=/dev/ada0 bs=1g count=2
dd if=/dev/zero of=/dev/ada0 bs=1g seek=NN5 count=999999
# You must clean ada0 of gmirror metadata *before* you load geom_mirror
 
I read that it's a good idea to create a ~4GB partition and leave it empty so the drive can use those sectors for wear-leveling.

Is that still a good practice for new SSD's?
 
SSDs generally have integrated (hidden) over-provisioning. Good ones generally have more than crappy ones.

Creating an empty partition (and TRIMming it) increases the over-provisioning of the SSD.

There's a pretty good overview of why you need over-provisioning here.

You should not need to manually over-provision unless your duty cycle is unusually heavy, or TRIM is not working, or both. In that case you should be monitoring the SSD with sysutils/smartmontools with a view to understand what's happening.

Some years ago I saw a post on this forum from somebody who was manually setting aside 50% of the SSD capacity for additional over-provisioning. However I don't believe TRIM was supported back then, and the duty cycle was manic.
 
As an observation, I have just ordered all the parts I need to retire my last remaining gmirror root.

I'm replacing two spinning disks with a pair of SSDs, on which I shall install a Root-on-ZFS mirror.

I built the system before you could boot from ZFS, but the functional advantages you get from a ZFS root (including snapshots and boot environments) are just too good to ignore any longer.
 
gpw928 said:
Here are the notes I kept to clear the metadata, if you need to repeat or test the process:
Code: 
# Get the sector count for /dev/ada0 and divide it by "1024x1024x1024".
# This is the number of GiB on the drive.  Call it NNN.  Subtract 5 => NN5.
# Nuke the start and end of the drive (partition table and gmirror metadata).
dd if=/dev/zero of=/dev/ada0 bs=1g count=2
dd if=/dev/zero of=/dev/ada0 bs=1g seek=NN5 count=999999
# You must clean ada0 of gmirror metadata *before* you load geom_mirror
Click to expand...

Could you not just do a "gpart delete/destroy" and "gmirror clear" instead?
 
I would have to test it, and don't have the facilities right now. If you are aware of metadata being at the end of the disk, then fore-warned if fore-armed.
 
I've tried several times and everything works … until the reboot.

Then it get stuck in endless reboots and I am unable to see what the issue is - as I am using a virtual KVM to connect to the server and it's lagged.

How can I access dmesg on ada0 booting from the FreeBSD usb key installer?
 
Do you necessarily want to mirror the whole disk? You could partition the disks separately but identically, for example: ada0p1/ada1p1: 512k freebsd-boot, ada0p2/ada1p2 - the rest.
You copy the boot code to both ada0p1 and ada1p1.
Then you can create a mirror on top of ada0p2 and ada1p2 and put your root on it. You can create another mirror for the swap but I never mirror swap - unless you need a hot-pluggable or a more crash resistant system, it's better to add the swap partitions separately - you get a larger swap.
The only thing that's worth mirroring is your data, the rest just complicates the boot process.

I've been using this configuration for years and rarely had problems booting. Mostly when I added or reformatted disks and forgot to put the bootcode on them.
 
Mayhem30 said:
How can I access dmesg on ada0 booting from the FreeBSD usb key installer?
Click to expand...
Do you have physical access to the system? If so, use a real keyboard and monitor.

If not, please describe exactly how you get to the console.

Have you done this prior to rebooting (I think you should boot from one disk without it)?
Code: 
echo 'geom_mirror_load="YES"' >>/boot/loader.conf
Before you exit the installation shell, collect some status information.

By way of example, this is what I have on my gmirror. I have three primary partitions to accommodate two operating systems (booting from the third partition), shared swap, and logging:
Code: 
[sherman.136] $ gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  ada0 (ACTIVE)
                      ada1 (ACTIVE)
Code: 
[sherman.137] $ gpart show
=>       63  488397104  mirror/gm0  MBR  (233G)
         63          1              - free -  (512B)
         64  220200960           1  freebsd  (105G)
  220201024   33554432           2  freebsd  (16G)
  253755456  220200960           3  freebsd  [active]  (105G)
  473956416   14440751              - free -  (6.9G)

=>        0  220200960  mirror/gm0s1  BSD  (105G)
          0   10485760             1  freebsd-ufs  (5.0G)
   10485760   10485760             2  freebsd-ufs  (5.0G)
   20971520  199229440             4  freebsd-ufs  (95G)

=>       0  33554432  mirror/gm0s2  BSD  (16G)
         0  25165824             1  freebsd-swap  (12G)
  25165824   2097152             2  freebsd-swap  (1.0G)
  27262976   2097152             4  freebsd-swap  (1.0G)
  29360128   4194304             5  freebsd-swap  (2.0G)

=>        0  220200960  mirror/gm0s3  BSD  (105G)
          0   10485760             1  freebsd-ufs  (5.0G)
   10485760   10485760             2  freebsd-ufs  (5.0G)
   20971520  199229440             4  freebsd-ufs  (95G)
Code: 
[sherman.135] $ gmirror list
Geom name: gm0
State: COMPLETE
Components: 2
Balance: load
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3163051507
Type: AUTOMATIC
Providers:
1. Name: mirror/gm0
   Mediasize: 250059349504 (233G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r4w4e11
Consumers:
1. Name: ada0
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: DIRTY
   GenID: 0
   SyncID: 1
   ID: 707674990
2. Name: ada1
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: DIRTY
   GenID: 0
   SyncID: 1
   ID: 76102181
 
Yes, I have the entry in loader.conf before rebooting (after the install). I do not have physical access to the server.

I am able to uses scroll lock and see some information:

Code: 
GEOM_MIRROR: Device mirror/gm0 launched (2/2).
Root mount waiting for : usbbus0
Root mount waiting for : usbbus0
...
...
Root mount waiting for : usbbus0
Root mount waiting for : usbbus0
...
...
Root mount waiting for : usbbus0
Root mount waiting for : usbbus0
...
mountroot> ugen0.3:  <Peppercon AG Multidevice> at usbbus0 (disconnected)

There is no mention of "Trying to mount root from ufs:/dev/mirror/gm0s1a".
 
No, but it appears my /etc/fstab was empty .. but I am still having an issue.

After booting up the installer and going to shell, I was able to mount ada0s1a and add the missing entries.

After rebooting, everything loaded up fine. The entries were missing again from fstab (gmirror sync got rid of them?).

So I added them again, and rebooted the system and now getting "the following sectors could not be read" errors and the boot fails.
 
By mounting a physical disk, you have compromised the mirror.

From the emergency shell, you have to "kldload geom_mirror" and mount /dev/mirror/gm0s1a.

I'm looking at the manual here, and have tested nothing, so beware...

I would try to deactivate the second disk from the emergency shell, edit as required, boot form the mirror, and reactivate the second disk:
Code: 
# boot from usb
kldload geom_mirror
gmirror status
gmirror deactivate gm0 ada1
gmirror status
# mount /dev/mirror/gm0s1a and edit as required
reboot # from sata
gmirror activate gm0 ada1
gmirror status
 
Ok, before doing so I would like to reinstall the OS because I tried to fsck and that made things even worse.

The commands I ran were :

Code: 
gmirrow destory gm0

gpart destroy -F ada0s1
gpart destroy -F ada1s1

gpart destroy -F ada0
gpart destroy -F ada1

Now gpart show shows - (unable to copy/paste from console window)

gpart.png


Any idea how to clear the leftover fluff on diskid/DISK-**** ?
 
You must log in or register to reply here.
Back
Top