1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

FreeBSD 8.2 to 9.0 RELEASE + gmirror + custom kernel

Discussion in 'Installing and Upgrading FreeBSD' started by xy16644, May 5, 2012.

  1. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    Hello All

    I want to upgrade my FreeBSD 8.2 RELEASE server to 9.0 RELEASE. I do run a custom kernel and I am running a mirrored disk setup with gmirror (from what I understand this can cause issues after the upgrade).

    I have made a full backup of the server as follows:
    dump -0aLf /tmp/backup.tar /dev/mirror/RootMirror4s1a

    Do I need to do anything special before performing the upgrade due to my configuration?

    I was going to run the following commands to do my upgrade:
    Code:
    freebsd-update upgrade -r 9.0-RELEASE
    
    cd /usr/src
    
    make KERNCONF=ALPHA kernel
    
    freebsd-update install
    
    shutdown -r now
    
    freebsd-update install
    
    shutdown -r now
    


    I'm sure I' m missing something here. Do I need to do anything else to make sure my system will still boot up since I use gmirror? Or anything else in general I have missed?

    Many thanks :)
     
  2. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    If you followed the Handbook example to create a mirror in-place on an existing drive, expect problems. There are workarounds. The right way is a full backup, creating the mirror from scratch, then a restore to the mirror. Mirrors created that way will work fine.

    1. Giving dump(8) files a ".tar" extension will cause regret later.
    2. Probably that is not a full backup. s1a is usually just the / filesystem. Remember that dump(8) does not cross filesystems. See Backup Options For FreeBSD for examples.

    Do a full backup (or several) and restore it to another drive. Test that drive, make sure all the files are there and correct. Then do the upgrade.
     
  3. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    Thats got me a bit scared! ;)

    I can't remember where I read about how to create the disk mirror using gmirror but I did document the steps I did:
    Code:
    Ensure:
    echo geom_mirror_load="YES" > /boot/loader.conf
    Create /etc/fstab-mirror as follows:
    /dev/mirror/RootMirror4s1b              none            swap    sw              0       0
    /dev/mirror/RootMirror4s1a              /               ufs     rw              1       1
    /dev/acd0               /cdrom          cd9660  ro,noauto       0       0
    
    Boot into single user mode and then run the following commands:
    
    mount -a
    
    cp /etc/fstab /etc/fstab-nomirror
    
    cp /etc/fstab-mirror /etc/fstab
    
    sysctl kern.geom.debugflags=16
    
    gmirror label -v RootMirror4 /dev/ad4
    
    Reboot in multi user mode and run:
    
    gmirror insert RootMirror4 /dev/ad6
    

    (sorry if this comes out a bit messy)

    Are you saying that the way I am using dump currently is not correct for having a full backup of the system? Currently I am using about 16GB on the server and when the dump command I posted previously is complete the tar file is about 11.5GB in size.

    So I assume I need to get the dump backup command correct first to do a FULL backup of the system and then I need to restore it to a test (maybe VM) server and do a restore? Then do the upgrade?
     
  4. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    For a single filesystem layout like that, that dump is adequate (but testing is still required). Verifying it in a VM is not a bad idea. Better yet, a VM gives the opportunity to rehearse and document the steps so doing it for real is quicker and easier.

    The steps don't say whether ad4 already had data on it. gmirror(8) needs one block for metadata at the end of the drive or partition. If the drive or partition is already full with a filesystem, that procedure turns off the safety with the sysctl. Then the mirror metadata can be written to the last block, which (probably) does not contain filesystem data.

    When the newer geom code sees that problem, one geom (the mirror) inside the boundaries of another (the original partition), it complains and does not boot.

    So again, the right way to do this:

    1. Back up all data. Verify it. Back it up again to other media, and verify that, too.
    2. Create the mirror from scratch. Do not use kern.geom.debugflags, because it will not be needed. This will wipe out all data on the mirror drives!
    # gmirror label mymirror0 ad4 ad6
    3. Set up MBR slices and FreeBSD partitions on the mirror. Remember to make it bootable (Disk Setup On FreeBSD) and load the geom_mirrormodule. (Another option is to mirror partitions. See gmirror With Disk Partitions, but be aware of the downsides.)
    4. Restore backup to the mirror.
     
  5. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    I am busy installing VirtualBox on my laptop now to experiment with.

    A bit of background (which may answer some of your questions). When I initially built the FreeBSD server it was on a single 80GB drive. It was only later on that I added a second drive and then mirrored it. So I think the answer to your question is yes in that ad4 did have data on it before mirroring.

    I have never done a full system restore before using my dump backup file...is this fairly straight forward?

    This is quite overwhelming. Let me just break it down into steps to make sure I fully understand it all:

    1) Install FreeBSD 8.2 into VM with one disk (I assume)

    2) Restore system from dump tar file

    3) Setup disk mirroring (not sure which comes first...step 2 or 3)

    NB: In theory, at this stage, my VM should match my live server

    4) Break the mirror and recreate from scratch

    5) Upgrade to FreeBSD 9

    I'm not 100% sure of all the commands etc at this stage. I just want to understand the steps involved first to make sure I get the logic behind it all.
     
  6. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    1. Boot VM with mfsBSD.
    2. Create mirror. For a test system, a one-disk mirror would be adequate. Use two disks when you do it for real.
    3. fdisk/bsdlabel mirror.
    4. Restore backup onto mirror.
    5. Test VM.
    6. Upgrade VM to FreeBSD 9.
    7. Test VM.
     
  7. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    I didn't get very far with this last night. My VM in VirtualBox kept crashing. So it got me thinking, what are my options to get my system from 8.2 to 9.0:

    1) Remove one of the disks from the gmirror (RAID1). Do the upgrade (on the remaining disk) and hope for the best. If it fails troubleshoot. If all else fails, remove the disk with the failed upgrade on it and put the second disk back in with 8.2 on it.

    2) Build a new server with FreeBSD 9.0 RELEASE and do a full restore from my dump file (is this possible considering I'll be going from FreeBSD 8.2 to 9.0?) I don't have the hardware for this option unfortunately right now.

    3) Clone/image the current system and the do the upgrade. If it fails revert back to a (working tested) copy

    I'm not sure if my thinking is correct regarding this.

    Would my upgrade to 9.0 be easier if I didn't have the system mirrored using gmirror in RAID1? ie: would the upgrade work 100% if I was running the server on a single disk?

    Thanks for all the help. This is the toughest upgrade I have done yet.
     
  8. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    The only difference in setup is the mirror, otherwise it's no harder and no less reliable.

    Make certain you have the latest VirtualBox, and that it's using PIIX4 for the hard drive controller.

    If you install FreeBSD 9 and then restore over it, you'll have FreeBSD 8 again. Unless you only restore user information, not operating system and applications.
     
  9. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    Ok, so what about breaking the mirror? If the 8.2 system has no gmirror RAID 1 on it (ie: single bootable disk), would that simplify things and enable me to perform the upgrade to 9.0?
     
  10. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    Why break the existing mirror? Set up two virtual drives in VirtualBox, or use a scratch drive if you want to do it on real hardware.

    The only difference with the mirror rather than a single disk is that you have to create it first and use the mirror device names instead of raw disk names.
     
  11. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    I have managed to find a physical machine that is indentical to the one I currently have FreeBSD 8.2 installed on. It even has two hard drives of the same capacity.

    So how does this sound:
    1. Install FreeBSD 8.2 on identical machine with two hard drives
    2. Copy dump backup file onto newly built machine
    3. Run full restore on newly built machine (at this stage the newly built machine should have the same config as my current live server)
    4. Update FreeBSD 8.2 to 9 on test machine - test/check that all is working
     
  12. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    I have installed FreeBSD 8.2 RELEASE on a machine that is indentical to what my production machine is. It has two drives that are the same capacity as my production server.

    What I did after installing FreeBSD 8.2 was to mount the second drive as /mnt/seconddrive. I then ran the following while in /mnt/seconddrive:
    restore -rf /tmp/restore.tar

    After it finished running I can see ALL my data on this second drive as it is on my live server.

    I then powered down the server and removed the drive I had installed FreeBSD 8.2 on and rebooted.

    I now have the following error:
    Code:
    FreeBSD/x86 boot
    Default: 0:ad(0,a)/boot/kernel/kernel
    


    I have read a bit about this but I still can't get the newly restored drive to boot. I have tried editing the /etc/fstab file and tried the bsdlabel command but I have not had any luck with it. I also tried making the drive bootable from sysinstall but that didn't work either.

    Any ideas please? I am well stuck! :e

    Once I can get this drive to boot I can perform the upgrade to FreeBSD 9.0 RELEASE as a test.
     
  13. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    Most likely you restored the old /etc/fstab over the one that was installed. That's a problem with brute-force restores. It can be avoided by using labels.

    Let me point out one more time that giving dump files a ".tar" extension will cause you pain, sooner or later.
     
  14. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    Hi Wblock, thanks for the reply :e

    I have taken your advice and stopped using .tar extensions for my backups. I now use .dump. Thank you for pointing that out to me.

    I also had a read through that link you provided that explains labels. These seem very useful. Couple of questions though:

    1) How do I go about using these on a mirrored (RAID1) system using gmirror?

    Currently my /etc/fstab looks like this:
    Code:
    /dev/mirror/RootMirror4s1b              none            swap    sw              0       0
    /dev/mirror/RootMirror4s1a              /               ufs     rw              1       1
    


    and if I run a:
    gmirror status
    I get this:
    Code:
    mirror/RootMirror4  COMPLETE  ad4
                                  ad6
    


    2) If I go ahead and make these changes with labels, does this mean I will be able to successfully upgrade from 8.2 to 9.0 and not experience the boot problems that most people do with gmirror?

    Thanks again.
     
  15. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    Filesystem or glabel(8) labels show up whenever a disk is attached. They can be used in /etc/fstab instead of absolute device names. So the fstab shown above would be
    Code:
    /dev/label/mswap              none            swap    sw              0       0
    /dev/ufs/mrootfs              /               ufs     rw              1       1
    


    That's if glabel(8) was used to assign the name mswap to the swap partition, and tunefs(8) was used to assign mrootfs to the UFS filesystem. These would work regardless of whether there were other labels or other mirrors.

    No, unfortunately that is a different problem, where the mirror and filesystem areas overlap. If the mirror was created as shown in the Handbook, where a drive was added to an existing one with the kern.geom.debugflags workaround, the areas overlap and FreeBSD 9 will refuse to boot. The right way to fix those is to back up everything, then boot with mfsBSD or the FreeBSD 9.0 install CD, go to a shell, and create the mirror from scratch. The workaround is not necessary and the mirror will be correct. Restore all the data to it, then use normally.
     
  16. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    This is what has me nervous and unsure of how to proceed next. I created a fresh dump of my current live server and then did a restore on indentical hardware. It didn't boot after I completed the restore. Now I understand that is due to the disks having different names but I had to "hack" the /etc/fstab file to boot off ONE disk. I didn't have a mirrored pair after the restore. And to boot each time (after a reboot I had to enter:

    Code:
    0:ad(6,d)/boot/loader
    ufs:ad6s1d
    


    (took me ages to figure this out!)

    Sorry for all the questions but if I had to break this down very simply step by step how would it look? This is the most tricky upgrade I have done and am not familiar with using mfsBSD.

    1) Backup current live system with dump (using a .dump file extension!)

    2) Boot with FreeBSD 9 DVD on live system into the shell

    3) This is where I get lost. Must I format the drives and then recreate the mirror from scratch here? And also use labels to make things easier in the future?
     
  17. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    Right. The system sees a mirror as a single drive. Unless you create a mirror and restore onto it, you'll be back to one drive.

    Labels will help with that.

    mfsBSD is just a convenient FreeBSD system that runs from memory. It has all the standard FreeBSD commands.

    Yes. Use dump -C16 -b64 -0uanL, and make sure you back up each filesystem. There might be only one, or there might be /, /var, /tmp, and /usr. Maybe others, like a separate /home. Each has to be dumped separately, dump(8) does not cross filesystems.

    Boot the system with mfsBSD or the FreeBSD 9 CD and create the mirror. Then create the filesystems on it, and restore to those filesystems. It's a lot safer to use new disks, because then the old disk can be disconnected and that data on it isn't endangered.
     
  18. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    I have created a new dump of the system using:
    dump -C16 -b64 -0uanLf /tmp/TestRestore.dump /dev/mirror/RootMIrror4s1a

    At this time, there is only one filesystem and that is for root (/).

    This is the bit I need some help on. Let's assume I do this on my test machine that has identical hardware. Let's also assume that both disks in the test machine are empty/formatted.

    Once I have booted off the FreeBSD 9 DVD, do I do an install? Do I do an fdisk? Do I go into the shell and create the mirror? Surely if I do a full restore of my system once the mirror is created it'll roll back to 8.2? Shouldn't I boot off the 8.2 DVD rather? Am I doing a basic install of 8.2, mirroring it and then restoring over it?

    I understand that I need to create a new mirror and then do a restore using my dump file but I am battling to grasp the exact steps needed to achieve this. I am happy to try anything on the test machine :e
     
  19. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    As the Perl people would say, TMTOWTDI.

    I'd get the 8.2 system onto a properly-created mirror and then use a source upgrade to get to 9-stable. The 9.0 install CD would be used just to provide live CD to create the mirror and restore to it.

    Boot with the 9.0 install CD.
    Select Shell.
    Use glabel(8) to label the two disks. (Same as filesystem labels, these provide a static label that always refers to a single disk.)
    Create the mirror with the two disks, using the labels instead of device names:
    # gmirror label -v mymirror0 /dev/label/mirrordisk0 /dev/label/mirrordisk1
    Use gpart(8) to create an MBR, make it bootable, then create bsdlabel "partitions" on the mirror, just one in this case. (Note: gpart(8) does everything fdisk(8) and bsdlabel(8) did, plus a lot more.)
    glabel(8) the swap area to give it a name.
    newfs(8) the filesystem, giving it a filesystem label at the same time.
    mount(8) the empty filesystem on /mnt.
    cd to /mnt and restore(8) the backup.
    While it's still mounted, edit /mnt/etc/fstab to use the filesystem and swap label.
    Reboot and check that it comes up. Then verify all the data is there.

    After that, update source to FreeBSD 9.0-RELEASE or 9.0-STABLE, buildworld, kernel, installworld, mergemaster -Ui, reboot. Because the mirror was created right, FreeBSD 9 won't have a problem with it.
     
    xy16644 thanks for this.
  20. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    Many thanks for this! I shall be giving this a try and reporting back as to how it goes. I will list the commands in the order I try them. This may be useful for other forum members.
     
  21. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    When I have restored my system, can I use the FreeBSD 9 DVD to perform the upgrade instead of using the following method:
    freebsd-update upgrade -r 9.0-RELEASE

    Also, you mentioned upgrading to FreeBSD 9 STABLE. I am running 8.2 RELEASE with a custom kernel. Will this cause any issues?
     
  22. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    I have never tried it, but have no reason to think it wouldn't work.

    Not really, although I think it means that freebsd-update(8) can't be used. And it may be necessary to adjust the kernel config file for differences in FreeBSD 9.
    Simplifying FreeBSD Kernel Config Files
    Upgrading FreeBSD To -STABLE
     
  23. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    The only reason I want to use the DVD to perform the upgrade is because once I restore the system, it'll have the same IP as my live server. I don't want to change the IP on the freshly restored server so as to keep things "like for like" which means it can't be connected to my network.

    Do I just mount the DVD and run sysinstall to perform the upgrade?

    I think I am going to stick with the FreeBSD 9.0 RELEASE version for now.
     
  24. wblock@

    wblock@ Administrator Staff Member Administrator Moderator Developer

    Messages:
    11,548
    Thanks Received:
    2,222
    IP address will be an issue regardless of whether it's 8.2, 9.0-RELEASE, or 9-STABLE. Unless you mean that the machine would not need to be connected to the net at all if you upgrade from the DVD. Save that until after the mirror conversion is complete.

    Then comment the ifconfig line in /etc/rc.conf, add another to temporarily use DHCP, and do the 8.x to 9.x upgrade. That is probably desirable to be able to test network services on the updated machine anyway.
     
  25. xy16644

    xy16644 New Member

    Messages:
    830
    Thanks Received:
    8
    I realise that the version I upgrade to has nothing to do with the IP address issue. What I meant to say was, I need to keep the restored machine disconnected from the LAN which means that I won't have any internet access. I thought the workaround could be to just use the DVD to perform the upgrade. I don't mind changing the IP address AFTER the upgrade to 9.0.

    So the steps (I think) will be (while the NIC is not connected to the netowork):
    1. Rebuild mirror
    2. Restore system from dump file
    3. Boot off DVD (?) and select upgrade option to upgrade to 9.0
    4. After upgrade, login and change IP in rc.conf
    5. Connect to LAN
    6. Test network services, ports, etc