FreeBSD 8.2 to 9.0 RELEASE + gmirror + custom kernel

xy16644 · May 5, 2012

Hello All

I want to upgrade my FreeBSD 8.2 RELEASE server to 9.0 RELEASE. I do run a custom kernel and I am running a mirrored disk setup with gmirror (from what I understand this can cause issues after the upgrade).

I have made a full backup of the server as follows:
[cmd=]dump -0aLf /tmp/backup.tar /dev/mirror/RootMirror4s1a[/cmd]

Do I need to do anything special before performing the upgrade due to my configuration?

I was going to run the following commands to do my upgrade:

Code:

freebsd-update upgrade -r 9.0-RELEASE

cd /usr/src

make KERNCONF=ALPHA kernel

freebsd-update install

shutdown -r now

freebsd-update install

shutdown -r now

I'm sure I' m missing something here. Do I need to do anything else to make sure my system will still boot up since I use gmirror? Or anything else in general I have missed?

Many thanks

wblock@ · May 5, 2012

xy16644 said:
I want to upgrade my FreeBSD 8.2 RELEASE server to 9.0 RELEASE. I do run a custom kernel and I am running a mirrored disk setup with gmirror (from what I understand this can cause issues after the upgrade).

If you followed the Handbook example to create a mirror in-place on an existing drive, expect problems. There are workarounds. The right way is a full backup, creating the mirror from scratch, then a restore to the mirror. Mirrors created that way will work fine.

Code:
I have made a full backup of the server as follows:

Code:

dump -0aLf /tmp/backup.tar /dev/mirror/RootMirror4s1a

1. Giving dump(8) files a ".tar" extension will cause regret later.
2. Probably that is not a full backup. s1a is usually just the / filesystem. Remember that dump(8) does not cross filesystems. See Backup Options For FreeBSD for examples.

Do I need to do anything special before performing the upgrade due to my configuration?

Do a full backup (or several) and restore it to another drive. Test that drive, make sure all the files are there and correct. Then do the upgrade.

xy16644 · May 5, 2012

wblock@ said:
If you followed the Handbook example to create a mirror in-place on an existing drive, expect problems. There are workarounds. The right way is a full backup, creating the mirror from scratch, then a restore to the mirror. Mirrors created that way will work fine.

1. Giving dump(8) files a ".tar" extension will cause regret later.
2. Probably that is not a full backup. s1a is usually just the / filesystem. Remember that dump(8) does not cross filesystems. See Backup Options For FreeBSD for examples.

Do a full backup (or several) and restore it to another drive. Test that drive, make sure all the files are there and correct. Then do the upgrade.

Thats got me a bit scared!

I can't remember where I read about how to create the disk mirror using gmirror but I did document the steps I did:

Code:

Ensure:
echo geom_mirror_load="YES" > /boot/loader.conf
Create /etc/fstab-mirror as follows:
/dev/mirror/RootMirror4s1b              none            swap    sw              0       0
/dev/mirror/RootMirror4s1a              /               ufs     rw              1       1
/dev/acd0               /cdrom          cd9660  ro,noauto       0       0

Boot into single user mode and then run the following commands:

mount -a

cp /etc/fstab /etc/fstab-nomirror

cp /etc/fstab-mirror /etc/fstab

sysctl kern.geom.debugflags=16

gmirror label -v RootMirror4 /dev/ad4

Reboot in multi user mode and run:

gmirror insert RootMirror4 /dev/ad6

(sorry if this comes out a bit messy)

Are you saying that the way I am using dump currently is not correct for having a full backup of the system? Currently I am using about 16GB on the server and when the dump command I posted previously is complete the tar file is about 11.5GB in size.

So I assume I need to get the dump backup command correct first to do a FULL backup of the system and then I need to restore it to a test (maybe VM) server and do a restore? Then do the upgrade?

wblock@ · May 5, 2012

For a single filesystem layout like that, that dump is adequate (but testing is still required). Verifying it in a VM is not a bad idea. Better yet, a VM gives the opportunity to rehearse and document the steps so doing it for real is quicker and easier.

The steps don't say whether ad4 already had data on it. gmirror(8) needs one block for metadata at the end of the drive or partition. If the drive or partition is already full with a filesystem, that procedure turns off the safety with the sysctl. Then the mirror metadata can be written to the last block, which (probably) does not contain filesystem data.

When the newer geom code sees that problem, one geom (the mirror) inside the boundaries of another (the original partition), it complains and does not boot.

So again, the right way to do this:

1. Back up all data. Verify it. Back it up again to other media, and verify that, too.
2. Create the mirror from scratch. Do not use kern.geom.debugflags, because it will not be needed. This will wipe out all data on the mirror drives!
# gmirror label mymirror0 ad4 ad6
3. Set up MBR slices and FreeBSD partitions on the mirror. Remember to make it bootable (Disk Setup On FreeBSD) and load the geom_mirrormodule. (Another option is to mirror partitions. See gmirror With Disk Partitions, but be aware of the downsides.)
4. Restore backup to the mirror.

xy16644 · May 5, 2012

I am busy installing VirtualBox on my laptop now to experiment with.

A bit of background (which may answer some of your questions). When I initially built the FreeBSD server it was on a single 80GB drive. It was only later on that I added a second drive and then mirrored it. So I think the answer to your question is yes in that ad4 did have data on it before mirroring.

I have never done a full system restore before using my dump backup file...is this fairly straight forward?

This is quite overwhelming. Let me just break it down into steps to make sure I fully understand it all:

1) Install FreeBSD 8.2 into VM with one disk (I assume)

2) Restore system from dump tar file

3) Setup disk mirroring (not sure which comes first...step 2 or 3)

NB: In theory, at this stage, my VM should match my live server

4) Break the mirror and recreate from scratch

5) Upgrade to FreeBSD 9

I'm not 100% sure of all the commands etc at this stage. I just want to understand the steps involved first to make sure I get the logic behind it all.

wblock@ · May 5, 2012

1. Boot VM with mfsBSD.
2. Create mirror. For a test system, a one-disk mirror would be adequate. Use two disks when you do it for real.
3. fdisk/bsdlabel mirror.
4. Restore backup onto mirror.
5. Test VM.
6. Upgrade VM to FreeBSD 9.
7. Test VM.

xy16644 · May 6, 2012

wblock@ said:
1. Boot VM with mfsBSD.
2. Create mirror. For a test system, a one-disk mirror would be adequate. Use two disks when you do it for real.
3. fdisk/bsdlabel mirror.
4. Restore backup onto mirror.
5. Test VM.
6. Upgrade VM to FreeBSD 9.
7. Test VM.

I didn't get very far with this last night. My VM in VirtualBox kept crashing. So it got me thinking, what are my options to get my system from 8.2 to 9.0:

1) Remove one of the disks from the gmirror (RAID1). Do the upgrade (on the remaining disk) and hope for the best. If it fails troubleshoot. If all else fails, remove the disk with the failed upgrade on it and put the second disk back in with 8.2 on it.

2) Build a new server with FreeBSD 9.0 RELEASE and do a full restore from my dump file (is this possible considering I'll be going from FreeBSD 8.2 to 9.0?) I don't have the hardware for this option unfortunately right now.

3) Clone/image the current system and the do the upgrade. If it fails revert back to a (working tested) copy

I'm not sure if my thinking is correct regarding this.

Would my upgrade to 9.0 be easier if I didn't have the system mirrored using gmirror in RAID1? ie: would the upgrade work 100% if I was running the server on a single disk?

Thanks for all the help. This is the toughest upgrade I have done yet.

wblock@ · May 7, 2012

The only difference in setup is the mirror, otherwise it's no harder and no less reliable.

Make certain you have the latest VirtualBox, and that it's using PIIX4 for the hard drive controller.

If you install FreeBSD 9 and then restore over it, you'll have FreeBSD 8 again. Unless you only restore user information, not operating system and applications.

xy16644 · May 7, 2012

Ok, so what about breaking the mirror? If the 8.2 system has no gmirror RAID 1 on it (ie: single bootable disk), would that simplify things and enable me to perform the upgrade to 9.0?

wblock@ · May 8, 2012

Why break the existing mirror? Set up two virtual drives in VirtualBox, or use a scratch drive if you want to do it on real hardware.

The only difference with the mirror rather than a single disk is that you have to create it first and use the mirror device names instead of raw disk names.

xy16644 · May 8, 2012

wblock@ said:
Why break the existing mirror? Set up two virtual drives in VirtualBox, or use a scratch drive if you want to do it on real hardware.

The only difference with the mirror rather than a single disk is that you have to create it first and use the mirror device names instead of raw disk names.

I have managed to find a physical machine that is indentical to the one I currently have FreeBSD 8.2 installed on. It even has two hard drives of the same capacity.

So how does this sound:

Install FreeBSD 8.2 on identical machine with two hard drives
Copy dump backup file onto newly built machine
Run full restore on newly built machine (at this stage the newly built machine should have the same config as my current live server)
Update FreeBSD 8.2 to 9 on test machine - test/check that all is working

xy16644 · May 12, 2012

I have installed FreeBSD 8.2 RELEASE on a machine that is indentical to what my production machine is. It has two drives that are the same capacity as my production server.

What I did after installing FreeBSD 8.2 was to mount the second drive as /mnt/seconddrive. I then ran the following while in /mnt/seconddrive:
[cmd=]restore -rf /tmp/restore.tar[/cmd]

After it finished running I can see ALL my data on this second drive as it is on my live server.

I then powered down the server and removed the drive I had installed FreeBSD 8.2 on and rebooted.

I now have the following error:

Code:

FreeBSD/x86 boot
Default: 0:ad(0,a)/boot/kernel/kernel

I have read a bit about this but I still can't get the newly restored drive to boot. I have tried editing the /etc/fstab file and tried the bsdlabel command but I have not had any luck with it. I also tried making the drive bootable from sysinstall but that didn't work either.

Any ideas please? I am well stuck! :e

Once I can get this drive to boot I can perform the upgrade to FreeBSD 9.0 RELEASE as a test.

wblock@ · May 16, 2012

Most likely you restored the old /etc/fstab over the one that was installed. That's a problem with brute-force restores. It can be avoided by using labels.

Let me point out one more time that giving dump files a ".tar" extension will cause you pain, sooner or later.

xy16644 · May 19, 2012

wblock@ said:
Most likely you restored the old /etc/fstab over the one that was installed. That's a problem with brute-force restores. It can be avoided by using labels.

Let me point out one more time that giving dump files a ".tar" extension will cause you pain, sooner or later.

Hi Wblock, thanks for the reply :e

I have taken your advice and stopped using .tar extensions for my backups. I now use .dump. Thank you for pointing that out to me.

I also had a read through that link you provided that explains labels. These seem very useful. Couple of questions though:

1) How do I go about using these on a mirrored (RAID1) system using gmirror?

Currently my /etc/fstab looks like this:

Code:

/dev/mirror/RootMirror4s1b              none            swap    sw              0       0
/dev/mirror/RootMirror4s1a              /               ufs     rw              1       1

and if I run a:
[cmd=]gmirror status[/cmd]
I get this:

Code:

mirror/RootMirror4  COMPLETE  ad4
                              ad6

2) If I go ahead and make these changes with labels, does this mean I will be able to successfully upgrade from 8.2 to 9.0 and not experience the boot problems that most people do with gmirror?

Thanks again.

wblock@ · May 19, 2012

1) How do I go about using these on a mirrored (RAID1) system using gmirror?

Filesystem or glabel(8) labels show up whenever a disk is attached. They can be used in /etc/fstab instead of absolute device names. So the fstab shown above would be

Code:

/dev/label/mswap              none            swap    sw              0       0
/dev/ufs/mrootfs              /               ufs     rw              1       1

That's if glabel(8) was used to assign the name mswap to the swap partition, and tunefs(8) was used to assign mrootfs to the UFS filesystem. These would work regardless of whether there were other labels or other mirrors.

2) If I go ahead and make these changes with labels, does this mean I will be able to successfully upgrade from 8.2 to 9.0 and not experience the boot problems that most people do with gmirror?

No, unfortunately that is a different problem, where the mirror and filesystem areas overlap. If the mirror was created as shown in the Handbook, where a drive was added to an existing one with the kern.geom.debugflags workaround, the areas overlap and FreeBSD 9 will refuse to boot. The right way to fix those is to back up everything, then boot with mfsBSD or the FreeBSD 9.0 install CD, go to a shell, and create the mirror from scratch. The workaround is not necessary and the mirror will be correct. Restore all the data to it, then use normally.

xy16644 · May 19, 2012

wblock@ said:
No, unfortunately that is a different problem, where the mirror and filesystem areas overlap. If the mirror was created as shown in the Handbook, where a drive was added to an existing one with the kern.geom.debugflags workaround, the areas overlap and FreeBSD 9 will refuse to boot. The right way to fix those is to back up everything, then boot with mfsBSD or the FreeBSD 9.0 install CD, go to a shell, and create the mirror from scratch. The workaround is not necessary and the mirror will be correct. Restore all the data to it, then use normally.

This is what has me nervous and unsure of how to proceed next. I created a fresh dump of my current live server and then did a restore on indentical hardware. It didn't boot after I completed the restore. Now I understand that is due to the disks having different names but I had to "hack" the /etc/fstab file to boot off ONE disk. I didn't have a mirrored pair after the restore. And to boot each time (after a reboot I had to enter:

Code:

0:ad(6,d)/boot/loader
ufs:ad6s1d

(took me ages to figure this out!)

Sorry for all the questions but if I had to break this down very simply step by step how would it look? This is the most tricky upgrade I have done and am not familiar with using mfsBSD.

1) Backup current live system with dump (using a .dump file extension!)

2) Boot with FreeBSD 9 DVD on live system into the shell

3) This is where I get lost. Must I format the drives and then recreate the mirror from scratch here? And also use labels to make things easier in the future?

wblock@ · May 19, 2012

xy16644 said:
This is what has me nervous and unsure of how to proceed next. I created a fresh dump of my current live server and then did a restore on indentical hardware. It didn't boot after I completed the restore. Now I understand that is due to the disks having different names but I had to "hack" the /etc/fstab file to boot off ONE disk. I didn't have a mirrored pair after the restore.

Right. The system sees a mirror as a single drive. Unless you create a mirror and restore onto it, you'll be back to one drive.

Code:
And to boot each time (after a reboot I had to enter:

Code:

0:ad(6,d)/boot/loader ufs:ad6s1d

Labels will help with that.

Sorry for all the questions but if I had to break this down very simply step by step how would it look? This is the most tricky upgrade I have done and am not familiar with using mfsBSD.

mfsBSD is just a convenient FreeBSD system that runs from memory. It has all the standard FreeBSD commands.

1) Backup current live system with dump (using a .dump file extension!)

Yes. Use dump -C16 -b64 -0uanL, and make sure you back up each filesystem. There might be only one, or there might be /, /var, /tmp, and /usr. Maybe others, like a separate /home. Each has to be dumped separately, dump(8) does not cross filesystems.

Boot the system with mfsBSD or the FreeBSD 9 CD and create the mirror. Then create the filesystems on it, and restore to those filesystems. It's a lot safer to use new disks, because then the old disk can be disconnected and that data on it isn't endangered.

xy16644 · May 20, 2012

wblock@ said:
Yes. Use dump -C16 -b64 -0uanL, and make sure you back up each filesystem. There might be only one, or there might be /, /var, /tmp, and /usr. Maybe others, like a separate /home. Each has to be dumped separately, dump(8) does not cross filesystems.

I have created a new dump of the system using:
[CMD=""]dump -C16 -b64 -0uanLf /tmp/TestRestore.dump /dev/mirror/RootMIrror4s1a[/CMD]

At this time, there is only one filesystem and that is for root (/).

wblock@ said:
Boot the system with mfsBSD or the FreeBSD 9 CD and create the mirror. Then create the filesystems on it, and restore to those filesystems. It's a lot safer to use new disks, because then the old disk can be disconnected and that data on it isn't endangered.

This is the bit I need some help on. Let's assume I do this on my test machine that has identical hardware. Let's also assume that both disks in the test machine are empty/formatted.

Once I have booted off the FreeBSD 9 DVD, do I do an install? Do I do an fdisk? Do I go into the shell and create the mirror? Surely if I do a full restore of my system once the mirror is created it'll roll back to 8.2? Shouldn't I boot off the 8.2 DVD rather? Am I doing a basic install of 8.2, mirroring it and then restoring over it?

I understand that I need to create a new mirror and then do a restore using my dump file but I am battling to grasp the exact steps needed to achieve this. I am happy to try anything on the test machine :e

wblock@ · May 20, 2012

As the Perl people would say, TMTOWTDI.

I'd get the 8.2 system onto a properly-created mirror and then use a source upgrade to get to 9-stable. The 9.0 install CD would be used just to provide live CD to create the mirror and restore to it.

Boot with the 9.0 install CD.
Select Shell.
Use glabel(8) to label the two disks. (Same as filesystem labels, these provide a static label that always refers to a single disk.)
Create the mirror with the two disks, using the labels instead of device names:
# gmirror label -v mymirror0 /dev/label/mirrordisk0 /dev/label/mirrordisk1
Use gpart(8) to create an MBR, make it bootable, then create bsdlabel "partitions" on the mirror, just one in this case. (Note: gpart(8) does everything fdisk(8) and bsdlabel(8) did, plus a lot more.)
glabel(8) the swap area to give it a name.
newfs(8) the filesystem, giving it a filesystem label at the same time.
mount(8) the empty filesystem on /mnt.
cd to /mnt and restore(8) the backup.
While it's still mounted, edit /mnt/etc/fstab to use the filesystem and swap label.
Reboot and check that it comes up. Then verify all the data is there.

After that, update source to FreeBSD 9.0-RELEASE or 9.0-STABLE, buildworld, kernel, installworld, mergemaster -Ui, reboot. Because the mirror was created right, FreeBSD 9 won't have a problem with it.

xy16644 · May 20, 2012

Many thanks for this! I shall be giving this a try and reporting back as to how it goes. I will list the commands in the order I try them. This may be useful for other forum members.

xy16644 · May 23, 2012

When I have restored my system, can I use the FreeBSD 9 DVD to perform the upgrade instead of using the following method:
[CMD=]freebsd-update upgrade -r 9.0-RELEASE[/cmd]

Also, you mentioned upgrading to FreeBSD 9 STABLE. I am running 8.2 RELEASE with a custom kernel. Will this cause any issues?

wblock@ · May 23, 2012

xy16644 said:
When I have restored my system, can I use the FreeBSD 9 DVD to perform the upgrade instead of using the following method:
[CMD="freebsd-update upgrade -r 9.0-RELEASE"]#[/CMD]

I have never tried it, but have no reason to think it wouldn't work.

Also, you mentioned upgrading to FreeBSD 9 STABLE. I am running 8.2 RELEASE with a custom kernel. Will this cause any issues?

Not really, although I think it means that freebsd-update(8) can't be used. And it may be necessary to adjust the kernel config file for differences in FreeBSD 9.
Simplifying FreeBSD Kernel Config Files
Upgrading FreeBSD To -STABLE

xy16644 · May 23, 2012

The only reason I want to use the DVD to perform the upgrade is because once I restore the system, it'll have the same IP as my live server. I don't want to change the IP on the freshly restored server so as to keep things "like for like" which means it can't be connected to my network.

Do I just mount the DVD and run sysinstall to perform the upgrade?

I think I am going to stick with the FreeBSD 9.0 RELEASE version for now.

wblock@ · May 23, 2012

IP address will be an issue regardless of whether it's 8.2, 9.0-RELEASE, or 9-STABLE. Unless you mean that the machine would not need to be connected to the net at all if you upgrade from the DVD. Save that until after the mirror conversion is complete.

Then comment the ifconfig line in /etc/rc.conf, add another to temporarily use DHCP, and do the 8.x to 9.x upgrade. That is probably desirable to be able to test network services on the updated machine anyway.

xy16644 · May 23, 2012

I realise that the version I upgrade to has nothing to do with the IP address issue. What I meant to say was, I need to keep the restored machine disconnected from the LAN which means that I won't have any internet access. I thought the workaround could be to just use the DVD to perform the upgrade. I don't mind changing the IP address AFTER the upgrade to 9.0.

So the steps (I think) will be (while the NIC is not connected to the netowork):

Rebuild mirror
Restore system from dump file
Boot off DVD (?) and select upgrade option to upgrade to 9.0
After upgrade, login and change IP in rc.conf
Connect to LAN
Test network services, ports, etc