9.0_RELEASE upgrade with RAID1

von_Gaden said:
So here is my verbose boot mode output:
Code:
Jan 23 22:09:38 leo kernel: GEOM_MIRROR: Device mirror/gm0 launched (2/2).
Jan 23 22:09:38 leo kernel: GEOM_PART: partition 1 has end offset beyond last LBA: 1250263727 > 1250263726
Jan 23 22:09:38 leo kernel: GEOM_PART: integrity check failed (mirror/gm0, MBR)

The one way how you may fix this problem:
1. Detach one component from your mirror.
2. Kill partition table and filesystems on it.
3. Create new mirror, then create partition table.
4. dump+restore the data from old mirror to the new one.
5. Reboot system from the new mirror, kill old mirror and attach second disk to new mirror.
 
I'd disagree since this is my exact current setup and I believe it causes my problem. My gmirror now is:
Code:
      Name    Status  Components
mirror/gm0  COMPLETE  ada0 (ACTIVE)
                      ada1 (ACTIVE)
/dev/mirror/gm0s1a on / (ufs, local)
/dev/mirror/gm0s1d on /usr (ufs, local, soft-updates)
...
 
Your mirror works fine, but your partition table is still marked as corrupt and you can not change it.
Also, if you remove kern.geom.part.check_integrity line from your loader.conf, then your system will become unbootable.
 
I mean my mirror is on whole drive and partitions were created on top of it as it was described in the Handbook. I see now there is a different procedure but some time ago it was just like that: FreeBSD installed on first drive, second drive added to newly created mirror, mirror is partitioned, contents dumped from the first drive, reboot from mirror and then wipe out first drive and add it to the mirror. So the partitioned drive is mirror, not the providers as you mention above.
And yes, this is inappropriate now and without kern.geom.part.check_integrity it won't boot.
 
I managed to remove my previous MBR scheme following the steps below. For details I would like to refer to this great tutorial provided by Warren. Great job Warren!

1. I attached an external USB disk to my dedicated FreeBSD server. On the external disk I created a new partition scheme according the GPT layout using gpart and newfs.

2. Kill most processes (httpd, mysld..) to avoid later locking file problems during boot from the external disk.

3. Use dump and restore to copy all data to the external disk.

4. mount the root directory on the external disk to /mnt and edit the /mnt/etc/fstab file and /mnt/boot/loader.conf.

5. Re-boot your system from the external USB disk.

6. Prepare the (orginal) harddisks for the gmirror following this great tutorial provided by Warren.

I am more relaxed since my output of gpart status looks much better now:

Code:
Name    Status  Components
ada0p1      OK  ada0
ada0p2      OK  ada0
ada0p3      OK  ada0
ada0p4      OK  ada0
ada0p5      OK  ada0
ada0p6      OK  ada0
ada1p1      OK  ada1
ada1p2      OK  ada1
ada1p3      OK  ada1
ada1p4      OK  ada1
ada1p5      OK  ada1
ada1p6      OK  ada1

Another positive thing is that my knowledge of FreeBSD, especially about booting and the filesystem, has grown a lot! Thanks to all extensive FreeBSD documentation and contributions. My believe in the FreeBSD only grew!;)
 
von_Gaden said:
I mean my mirror is on whole drive and partitions were created on top of it as it was described in the Handbook. I see now there is a different procedure but some time ago it was just like that: FreeBSD installed on first drive, second drive added to newly created mirror, mirror is partitioned, contents dumped from the first drive, reboot from mirror and then wipe out first drive and add it to the mirror. So the partitioned drive is mirror, not the providers as you mention above.
And yes, this is inappropriate now and without kern.geom.part.check_integrity it won't boot.

If you use MBR partitioning and leave a little unpartitioned space at the end of the drive, the Handbook procedure should work (untested!). That would leave a block free for gmirror metadata, and the boot process wouldn't complain about a partition being larger than it should be.

That will not work for GPT partitioning, which puts the backup partition table at the end of the disk (not logical device) and can't be put inside a mirror. If you use GPT, create identical partitions on the two disks first. The disks don't have to be the same size, but the partitions do. Then mirror the partitions. As mentioned elsewhere, there's a potential for disk head thrashing on resyncing the mirrors, so a single / filesystem like default bsdinstall setup might be an advantage. Non-filesystem partitions like freebsd-boot or freebsd-swap could even go without mirroring.

Oh: thanks for the kind words, Eric!
 
As I promised I updated one of my newer servers with mirrored GPT partitions. Partition and mirror schemes are very similar to the amazing and useful gmirror article posted by wblock earlier in this thread. Two of the mirrors are also journaled.

As wblock@ guessed everything was fine and trouble-free! There are no integrity check fails or any other disturbing messages.

I think it will be safer to put
Code:
kern.geom.part.check_integrity=0
in /boot/loader.conf when updating to FreeBSD 9. Then examine the log and remove this setting if everything is OK. This is of essential importance when you are far away from the server.

I'm afraid that the only cure for "integrity check failed partitions" like my first MBR over mirrored devices may be only dump, re-partition and restore. This is really annoying and time-consuming on some thousand gigabytes of data. Luckily more than a terabyte on MBR and gmirror will be very rare case.
 
tovo said:
Hi Lido,
I've experienced exactly the same problem after an upgrade. I followed this old thread and it solved my problem.
To summarize, when you are on the boot menu, you need to choose to boot in single user and after run an FSCK on your filesystem.
Hope it helps

This doesn't seem to work. If you follow Eric's instructions on how to boot (except use boot -s to get to single user mode), you can run fsck, but it doesn't fix the issue the next time you boot.
 
Summarizing: The Handbook procedure used a quick hack to turn a non-mirrored drive into a mirrored drive without having to repartition; the last sector of the partition was reused for gmirror metadata without changing the partition size. It's a hack because that sector could be overwritten by the filesystem on it (although it's unlikely). Done properly, the mirror would have been created first, then the partition would be one sector smaller instead of "sharing" that sector. The Handbook procedure let the user skip the inconvenience of repartitioning.

That method worked under FreeBSD 8 because the boot loader wasn't very particular about partition sizes.

In FreeBSD 9, the boot loader checks partition sizes, and stops when it finds the overlap. The workaround, turning off the partition integrity check, is shown in the detailed release notes: http://www.freebsd.org/releases/9.0R/relnotes-detailed.html#AEN1277

The real cure is to back up, then recreate the mirror and partitions from scratch. Do not follow the "RAID 1 - Mirroring" Handbook procedure if you are using FreeBSD 9, but create a mirror, then MBR partitions on it, or create GPT partitions and mirror those. Finally, restore.

There may be a way to shrink a partition by one sector and correct a UFS filesystem on it, but AFAIK that has not been documented.
 
WBlock, in your tutorial, you leave a few partitions un-mirrored. In the Linux side, from what I remember the swap needed to be mirrored or the system would have trouble if the disk with the swap failed. Have you tested that with gmirror? I'm planning on using RAID1 mainly to keep the system up in the even a drive dies. Thanks.
 
I just encounter this kind of problem upgrading from 8.3-R to 9.0-R. The issue is 9.0-R can't seem to find partition on gm0.

My step to solve and boot into the 9.0-R :
- Using 9.0-R CD as live-cd to edit the ad0 fstab
- from mirror/gm0 to /dev/ad0s1a
- Reboot [ press 2 on options list, load into loader ] and type
>boot kernel.old
- Remove the gmirror ( ad4 and ad6 is from kernel.old )
gmirror remove gm0 ad6
gmirror remove gm0 ad4
- Reboot and you can boot into your new 9.0-R

I try to build back the raid but need to test first from my other machine.

Hope can help others with this kind of issue :)
 
Partition conflicts could prevent the mirror from being recognized. The gmirror(8) module must also be loaded or mirrors can't work. Usually done in /boot/loader.conf:
Code:
geom_mirror_load="YES"

Incidentally, the Handbook mirroring section has been updated to show the correct way to create a mirror with either one or two new drives.
 
Back
Top