gmirror existing root partition

F

freebuser

Hi All,

I have a server running with a single root disk with ssdboot and ssdrootfs gpt partitions.

Is it possible to create a gmirror adding another similarly partitioned disk without losing any existing data?
 
You can use gmirror and edit /etc/fstab with the correspongind /dev/mirror/label.
I am not sure if you need to be on live cd or not.
 
The FreeBSD Handbook specifically covers the case for UFS with "creates a mirror on a single new drive, copies the old drive’s data to it, then inserts the old drive into the mirror. While this procedure is slightly more complicated, it only requires one new drive".

ZFS generally lets you add a mirror too.

The output of gpart show would help with specifics.
 
gpw928 said:
The output of gpart show would help with specifics, but the FreeBSD Handbook specifically covers your case of "creates a mirror on a single new drive, copies the old drive’s data to it, then inserts the old drive into the mirror. While this procedure is slightly more complicated, it only requires one new drive".
Click to expand...

Yes, but the Handbook uses the full disk. I am not sure if this is possible with gpt partitions.

Phishfry said:
The gmirror will rebuild the new disk layout. No need to partition or format it.

Whatever you do, make a backup first before attempting this.

EFI or Legacy install??.
Click to expand...

My current root disk is only 120G SSD, I have another 120G SSD with all the jails in it which is zfs.

My new disk is 466G SSD which I am hoping to divide and mirror the UFS root disk and ZFS jails partitions.

Likely UEFI, can't remember.
 
You don't have to keep what you have got. You can create (one half of) a new mirror with a new disk, copy your existing data to that, and then redeploy the existing disk as second half of the new mirror. Once again, show us what you have with gpart show.
 
gpw928 said:
You don't have to keep what you have got. You can create (one half of) a new mirror with a new disk, copy your existing data to that, and then redeploy the existing disk as second half of the new mirror. Once again, show us what you have with gpart show.
Click to expand...

=> 40 27344764848 ada0 GPT (13T) 40 27344764848 1 backup-disk-01 (13T) => 40 19532873648 ada1 GPT (9.1T) 40 2008 - free - (1.0M) 2048 19327352832 1 10DISK2 (9.0T) 19327354880 205518808 - free - (98G) => 40 19532873648 ada2 GPT (9.1T) 40 19327352832 1 10DISK1 (9.0T) 19327352872 205520816 - free - (98G) => 40 250069600 ada3 GPT (119G) 40 1024 1 ssdboot (512K) 1064 984 - free - (492K) 2048 247463936 2 ssdrootfs (118G) 247465984 2603656 - free - (1.2G) => 40 234441568 da0 GPT (112G) 40 2008 - free - (1.0M) 2048 230686720 1 gpt-myjails (110G) 230688768 3752840 - free - (1.8G) => 40 976773088 ada4 GPT (466G) 40 2008 - free - (1.0M) 2048 230686720 1 gpt-myjails2 (110G) 230688768 746084360 - free - (356G) => 40 976773088 diskid/DISK-S4FNNF0M955732F GPT (466G) 40 2008 - free - (1.0M) 2048 230686720 1 gpt-myjails2 (110G) 230688768 746084360 - free - (356G)
 
Like noted above the handbook gives a pretty contorted way to do this with NOP.

Truthfully I don't think I would convert. Backup vital files and start fresh.

So a possible route would use the two 120GB disks for gmirror and new 466GB disk for ZFS.
Use your existing installation to create the zfs pool and copy everything into the pool.
Then do a fresh install with EFI gmirror on the 120GB pair.
forums.freebsd.org

Install UEFI FreeBSD on gmirror

I wanted to post instruction for installing FreeBSD on a GEOM_MIRROR -aka- gmirror(8). This is an advanced topic so I assume you are capable of determining that your two chosen disks are empty. UFS RAID1 on FreeBSD is enabled with the geom_mirror module. I am using ada0 and ada1 as examples...
forums.freebsd.org forums.freebsd.org
After, Import your pool, do OS customization and re-install packages and copy over your configs from zfs pool.
 
My instincts are broadly with Phishfry.

I might think about a second 466GB SSD to mirror the new 466GB disk. These things don't cost a lot, and I can't count the number of times redundant RAID saved me form major grief.

Is da0 a USB device, or is it on an internal SAS/SATA controller? What make/model are da0 and ada3?
 
Phishfry said:
What I don't like is your 120GB drives are not identical.
I am really weird about that. Matching drives or nil.
Click to expand...

Yep, that's the reason I am trying to mirror the GPT partitions not the drives.
I have very limited downtime so need to take it easy on the changes.
Hoping to transfer the OS to new boot and root partitions first and see how it goes.

gpw928 said:
My instincts are broadly with Phishfry.

I might think about a second 466GB SSD to mirror the new 466GB disk. These things don't cost a lot, and I can't count the number of times redundant RAID saved me form major grief.

Is da0 a USB device, or is it on an internal SAS/SATA controller? What make/model are da0 and ada3?
Click to expand...

I understand, but trying work with what I got :-(

Code: 
ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
ada3: <Samsung SSD 850 PRO 128GB EXM02B6Q> ACS-2 ATA SATA 3.x device
ada3: Serial Number S1SMNSAG111711H
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes)
ada3: Command Queueing enabled
ada3: 122104MB (250069680 512 byte sectors)
ada3: quirks=0x3<4K,NCQ_TRIM_BROKEN>
ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
ada4: <Samsung SSD 860 EVO 500GB RVT03B6Q> ACS-4 ATA SATA 3.x device
ada4: Serial Number S4FNNF0M955732F
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes)
ada4: Command Queueing enabled
ada4: 476940MB (976773168 512 byte sectors)
da0 at mps0 bus 0 scbus0 target 1 lun 0
da0: <ATA Samsung SSD 850 1B6Q> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number S21SNXAG801236X
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 114473MB (234441648 512 byte sectors)
da0: quirks=0x8<4K>
 
So it looks like you have on-board SATA 2 (ada) and PCIe plug-in SATA/SAS (da) controllers. Is that correct?

Your disk controllers have very different performance characteristics. Your SATA 3 SSDs attached to the on board ada controllers are running in SATA 2 mode, which has a significant performance cost.

I do get it that you need to work with what you have.

If you mirror ada3 against da0, writes will run at the slowest speed of all the components (controller and SDD). Reads will likely fare better, with the fastest response used. Given that these are SSDs, the worst that they can do is a whole lot better than any spinning disk, so I would not be too discouraged by the technology mis-match. But it might be better if the controllers were not so wildly different.

How many free SATA ports do you have, and what kind are they?

I would not mirror the 120GB SSDs to parts of the Samsung 860 EVO. It's just too messy. If you can't get a mirror for the 860 EVO, just make sure that your backup scheme is fit for purpose.

I would mirror the 120GB SSDs to each other for the root/swap.

Do you have a BIOS option to boot from da0?
 
gpw928 said:
So it looks like you have on-board SATA 2 (ada) and PCIe plug-in SATA/SAS (da) controllers. Is that correct?

Your disk controllers have very different performance characteristics. Your SATA 3 SSDs attached to the on board ada controllers are running in SATA 2 mode, which has a significant performance cost.

I do get it that you need to work with what you have.

If you mirror ada3 against da0, writes will run at the slowest speed of all the components (controller and SDD). Reads will likely fare better, with the fastest response used. Given that these are SSDs, the worst that they can do is a whole lot better than any spinning disk, so I would not be too discouraged by the technology mis-match. But it might be better if the controllers were not so wildly different.

How many free SATA ports do you have, and what kind are they?

I would not mirror the 120GB SSDs to parts of the Samsung 860 EVO. It's just too messy. If you can't get a mirror for the 860 EVO, just make sure that your backup scheme is fit for purpose.

I would mirror the 120GB SSDs to each other for the root/swap.

Do you have a BIOS option to boot from da0?
Click to expand...

Yes it is a Dell 310 PERC controller in IT mode. I have two SATA ports (single cable) one connected to LTO-5 tape drive and another to the da0.

No problems with the performance as it is not really a busy server but it is part of our small business setup, exposed to the clients side so don't want to shutdown that part.

I have five onboard SATA controllers with 2x10TB and 1x14TB for data and backups.
4th attached to the rootfs 128GB SSD and
5th is CD/DVD port which is currently attached to the new 466GB SSD.

SATA 1 : 14TB Backup disk (GPT+ZFS)
SATA 2 & 3 : 2 x 10TB Data disks (GPT+ZFS)
SATA 4 : Rootfs 128GB SDD (GPT+UFS)

SATA 5 : 466GB SSD (120GB partitioned to zfs mirror with jails and remaining hoping to gmirror with rootfs GPT partitions).

SAS/SATA 1 : LTO-5 Tape
SAS/SATA 2 : 120GB running jails

No More SATA ports left.


So far I have
1. created a ZFS mirror with part of the 466GB SDD and 120GB jails disk.
2. created the gpt partitions and gmirror from the remainder of the 466GB disk to match the sizes of current Rootfs (boot and rootfs).
gmirror is running with single disk for now with no data in it.

Question is (performance hit aside), will I be able to
1. move my current rootfs to the new gmirror (SATA5)
2. create appropriate partitions in SATA4 (current rootfs)
3. Attach SATA4 partitions to current gmirror in SATA 5.

In a nutshell, I am trying to use the existing 466GB disk to create redundancy for my jails and rootfs.

What future issues will I have to deal with if one of the disk fails?

At least I am guessing it can't be worse than my current setup without redundancy.
 
OK, if that's the way you want to go...

Improvements would require down-time and further risk.

However, it's messy. It's also arduous.

You have ada0 through ada4 internally, and da0 plus da1 (LTO) on the PERC.

Your plan is to mirror ada3 and ada4 for boot. And also to mirror the remainder of ada4 with da0, for jails.

Your current partitioning of ada4 is unworkable for a root mirror. Boot needs to be on partition 1 -- so you can boot from ada4 if ada3 fails. Boot partitions are usually manually duplicated. To proceed, you would need to trash ada4, and start again.

ada3 is ~120 GB, and it dictates limits to partition sizes for the root disk. The partitions you create on ada3 and ada4 for the root should be exactly the same size (same block count). I would also use the same partition numbers. i.e. make ada3 and ada4 *identical* for the first ~120GB. You need a minimum of three partitions, boot, swap, and root. Warren Bock has a good guide on partitioning with gpart.

You don't need to use all 120 GB. Think about the size of swap and the root. 50GB shoud be plenty for a UFS root, but use your own experience and judgement. So a rough guess is that you need:
  1. partition 1: boot: 1GB
  2. partition 2: 16 or 32 GB: swap
  3. partition 3: 50 GB: UFS root
  4. partition 4: unused space
You start with ada4. Preserve anything you need to keep, as its contents will be trashed. Partition as above with gpart(8). Make sure you install the bootcode onto partition 1. Then use gmirror label <swap|root> to place the swap and root partitions into one half of a mirror. Make an empty ufs file system on the ada4 root mirror (/dev/mirror/root).

Mount the ada4 root mirror, (/dev/mirror/root), temporarily (e.g. on /mnt). Copy the existing root from ada3 with dump -0Lau -f - / | (cd /mnt && restore -ruf -). Modify /mnt/etc/fstab on the ada4 root mirror to reflect the mirrors used for swap and root (both swap and / will be mounted on /dev/mirror/<name>).

Boot from ada4. Repeat the identical partitioning on ada3 (trashing it). Then insert the root and swap partitions on ada3 into the mirror configuration using gmirror insert (at which point the mirrors will sync).

If you can get this far, then mirroring the jails on da0 to the spare space on ada4 will be a doddle.

That's the broad plan. For risk minimisation, you need to make the detailed written plan yourself...
 
gpw928 said:
OK, if that's the way you want to go...

Improvements would require down-time and further risk.

However, it's messy. It's also arduous.

You have ada0 through ada4 internally, and da0 plus da1 (LTO) on the PERC.

Your plan is to mirror ada3 and ada4 for boot. And also to mirror the remainder of ada4 with da0, for jails.

Your current partitioning of ada4 is unworkable for a root mirror. Boot needs to be on partition 1 -- so you can boot from ada4 if ada3 fails. Boot partitions are usually manually duplicated. To proceed, you would need to trash ada4, and start again.

ada3 is ~120 GB, and it dictates limits to partition sizes for the root disk. The partitions you create on ada3 and ada4 for the root should be exactly the same size (same block count). I would also use the same partition numbers. i.e. make ada3 and ada4 *identical* for the first ~120GB. You need a minimum of three partitions, boot, swap, and root. Warren Bock has a good guide on partitioning with gpart.

You don't need to use all 120 GB. Think about the size of swap and the root. 50GB shoud be plenty for a UFS root, but use your own experience and judgement. So a rough guess is that you need:
  1. partition 1: boot: 1GB
  2. partition 2: 16 or 32 GB: swap
  3. partition 3: 50 GB: UFS root
  4. partition 4: unused space
You start with ada4. Preserve anything you need to keep, as its contents will be trashed. Partition as above with gpart(8). Make sure you install the bootcode onto partition 1. Then use gmirror label <swap|root> to place the swap and root partitions into one half of a mirror. Make an empty ufs file system on the ada4 root mirror (/dev/mirror/root).

Mount the ada4 root mirror, (/dev/mirror/root), temporarily (e.g. on /mnt). Copy the existing root from ada3 (dump | restore). Modify /etc/fstab on the ada4 root mirror to reflect the mirrors used for swap and root (both swap and / will be mounted on /dev/mirror/<name>).

Boot from ada4. Repeat the identical partitioning on ada3 (trashing it). Then insert the root and swap partitions on ada3 into the mirror configuration using gmirror insert (at which point the mirrors will sync).

If you can get this far, then mirroring the jails on da0 to the spare space on ada4 will be a doddle.

That's the broad plan. For risk minimistion, you need to make the detailed written plan yourself...
Click to expand...

Thanks for the your time for the detailed write up.

So far able to boot with ada4 single disk mirror setup.
I will run this for a while and if all good I will add the ada3 to the mirror.
 
Well done. Check that the partitions in use are actually the ones you think by busying the disk and looking at iostat -x.
freebuser said:
I have another 120G SSD with all the jails in it which is zfs.
Click to expand...
I missed this point. Perhaps you are aware, but, just in case you are not, you can mirror da0p1 using ZFS alone. See the Handbook example "Upgrade the single disk (stripe) vdev ada0p3 to a mirror by attaching ada1p3".

[I have edited my post above to improve clarity regarding the use of dump and restore.]
 
gpw928 said:
Well done. Check that the partitions in use are actually the ones you think by busying the disk and looking at iostat -x.

I missed this point. Perhaps you are aware, but, just in case you are not, you can mirror da0p1 using ZFS alone. See the Handbook example "Upgrade the single disk (stripe) vdev ada0p3 to a mirror by attaching ada1p3".

[I have edited my post above to improve clarity regarding the use of dump and restore.]
Click to expand...
Yes I am aware of it and that's what I did.
My old root disk (ada3) is not even mounted (confirmed by dump too when I used the -L for a backup dump) and / is mounted from mirror/rootfs, so I am certain the correct disk is used for / now.

I never had to recover a zpool in the past, but just in case if the os crashed is it just a matter of installing the os again and zpool export, zpool import to recover all my pools?
 
freebuser said:
I never had to recover a zpool in the past, but just in case if the os crashed is it just a matter of installing the os again and zpool export, zpool import to recover all my pools?
Click to expand...
Generally recovery is automatic after a crash. If the O/S loses memory of the ZFS pools for some reason (like a root rebuild), you may need to use zpool-import(8).
 
You must log in or register to reply here.
Back
Top