1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HOWTO: Modern FreeBSD Install RELOADED (vermaden way)

Discussion in 'Howtos and FAQs (Moderated)' started by vermaden, Mar 8, 2010.

  1. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    All these years sysinstall(8) was helping us to install FreeBSD with most options we needed, today with new filesystems/features like GJournal/ZFS/Geli/GMirror/GStripe its no longer up to the task, because it only supports creating installation on UFS filesystem with SoftUpdates turned ON or OFF.

    In this guide you will learn how to setup FreeBSD installation in simple yet flexible setup based on read-only UFS (without SoftUpdates) for 'base system' [1], some SWAP space, /tmp mounted on SWAP and all the other filesystems (/var /usr ...) mounted on ZFS. It will not require rebuilding anything, just simple setup on plain MBR partitions. I should also mention that we would be using AHCI mode for disks. I also provided two versions, for system with one harddisk and with three of them for redundant setup.

    Here is the layout of the system with 1 harddisk:
    Code:
    MBR SLICE 1 |    / | 512 MB | UFS/read-only 
                | SWAP |   2 GB |
                | /tmp | 512 MB | mounted on SWAP with mdmfs(8)
    ------------+------+---------------------------------------
    MBR SLICE 2 | /usr |   REST | ZFS dataset
                | /var |   REST | ZFS dataset


    ... and here layout for single disk for system with 3 disks:
    Code:
    MBR SLICE 1 |    / | 512 MB | UFS/read-only 
    ------------+------+--------+------------------------------
    MBR SLICE 2 | SWAP |   1 GB |
                | /tmp | 512 MB | mounted on SWAP with mdmfs(8)
    ------------+------+--------+------------------------------
    MBR SLICE 3 | /usr |   REST | ZFS dataset
                | /var |   REST | ZFS dataset


    Redundancy planning for system with 3 disks:
    Code:
     [ DISK0 ]           [ DISK1 ]           [ DISK2 ]
     [   /   ] < RAID1 > [   /   ] < RAID1 > [   /   ]
     [ SWAP0 ]           [ SWAP1 ]           [ SWAP2 ]
     [   Z   ] < RAID5 > [   F   ] < RAID5 > [   S   ]


    FreeBSD core, the 'base system' [1] should remain almost unchanged/untouched on daily basis while you can mess all other filesystems, this ensures that when things go wrong, you will be able to fix anything still having working 'base system' [1].

    You will need *-dvd-* disk or *-memstick-* image for this installation, *-disk1-* will not do since it does not contain livefs system.

    Here is the procedude, described as simple as possible.

    1.0. I assume that our disk for the installation would be /dev/ad0 (/dev/ad0 /dev/ad1 /dev/ad2 for system with 3 disks)

    1.1. Boot *-dvd-* from DVD disk or *-memstick-* image from pendrive
    Code:
    Country Selection --> United States
    Fixit --> CDROM/DVD (*-dvd-*) or USB (*-memstick-*)

    1.2. Create your temporary working environment
    Code:
    fixit# /mnt2/bin/csh
    # setenv PATH /mnt2/rescue:/mnt2/usr/bin:/mnt2/sbin
    # set filec
    # set autolist
    # set nobeep

    1.3. Load needed modules
    Code:
    fixit# kldload /mnt2/boot/kernel/geom_mbr.ko
    fixit# kldload /mnt2/boot/kernel/opensolaris.ko
    fixit# kldload /mnt2/boot/kernel/zfs.ko

    1.4. Create/mount needed filesystems
    Code:
    DISKS: 3                                               | DISKS: 1
    # cat > part << __EOF__                                | # cat > part << __EOF__
    p 1 165 63  512M                                       | p 1 165 63  2560M
    p 2 165  * 1024M                                       | p 2 159  *     *
    p 3 159  *     *                                       | p 3   0  0     0
    p 4   0  0     0                                       | p 4   0  0     0
    a 1                                                    | a 1
    __EOF__                                                | __EOF__
                                                           |
    # fdisk -f part ad0                                    | # fdisk -f part ad0
    # fdisk -f part ad1                                    |
    # fdisk -f part ad2                                    |
                                                           |
    # kldload /mnt2/boot/kernel/geom_mirror.ko             |
    # gmirror label  rootfs ad0s1                          |
    # gmirror insert rootfs ad1s1                          |
    # gmirror insert rootfs ad2s1                          |
                                                           |
    # bsdlabel -B -w /dev/mirror/rootfs                    | # cat > label << __EOF__
                                                           | # /dev/ad0s1:
                                                           | 8 partitions:
                                                           |   a: 512m  0 4.2BSD
                                                           |   b: *     * swap
                                                           | __EOF__
                                                           |
                                                           | # bsdlabel -B -w ad0s1
                                                           | # bsdlabel       ad0s1 | tail -1 >> label
                                                           | # bsdlabel -R    ad0s1 label
                                                           |
    # glabel label swap0 ad0s2                             | # glabel label rootfs ad0s1a
    # glabel label swap1 ad1s2                             | # glabel label swap   ad0s1b
    # glabel label swap2 ad2s2                             |
                                                           |
    # newfs /dev/mirror/rootfsa                            | # newfs /dev/label/rootfs
    # zpool create basefs raidz ad0s3 ad1s3 ad2s3          | # zpool create basefs ad0s2
    # zfs create basefs/usr                                | # zfs create basefs/usr
    # zfs create basefs/var                                | # zfs create basefs/var
    # mkdir /NEWROOT                                       | # mkdir /NEWROOT
    # mount /dev/mirror/rootfsa /NEWROOT                   | # mount /dev/label/rootfs /NEWROOT
    # zfs set mountpoint=/NEWROOT/usr basefs/usr           | # zfs set mountpoint=/NEWROOT/usr basefs/usr
    # zfs set mountpoint=/NEWROOT/var basefs/var           | # zfs set mountpoint=/NEWROOT/var basefs/var

    1.5. Actually install needed FreeBSD sets
    Code:
    # setenv DESTDIR /NEWROOT
    # cd /dist/8.0-RELEASE
    
    # cd base
    # ./install.sh (answer 'y' here)
    # cd ..
    
    # cd manpages
    # ./install.sh
    # cd ..
    
    # cd kernels
    # ./install.sh generic
    # cd ..
    
    # cd /NEWROOT/boot
    # rm -r kernel
    # mv GENERIC kernel
     
    tempo, zeissoctopus, Unixworld and 13 others thank for this.
  2. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    1.6. Provide basic configuration needed to boot new system
    1.6.1.
    Code:
    DISKS: 3                                          | DISKS: 1
    # cat > /NEWROOT/etc/fstab << __EOF__             | # cat > /NEWROOT/etc/fstab << __EOF__
    #dev                #mount #fs  #opts #dump #pass | #dev              #mount #fs  #opts #dump #pass
    /dev/mirror/rootfsa /      ufs  rw    1     1     | /dev/label/rootfs /      ufs  rw    1     1
    /dev/label/swap0    none   swap sw    0     0     | /dev/label/swap   none   swap sw    0     0
    /dev/label/swap1    none   swap sw    0     0     | __EOF__
    /dev/label/swap2    none   swap sw    0     0     |
    __EOF__                                           |
                                                      |
    # cat > /NEWROOT/boot/loader.conf << __EOF__      | # cat > /NEWROOT/boot/loader.conf << __EOF__
    zfs_load="YES"                                    | zfs_load="YES"
    ahci_load="YES"                                   | ahci_load="YES"
    geom_mirror_load="YES"                            | __EOF__
    __EOF__                                           |

    1.6.1.
    Code:
    # cat > /NEWROOT/etc/rc.conf << __EOF__
    zfs_enable="YES"
    __EOF__

    1.7. Unmount filesystems and reboot
    Code:
    # cd /
    # zfs umount -a
    # umount /NEWROOT
    # zfs set mountpoint=/usr basefs/usr
    # zfs set mountpoint=/var basefs/var
    # zpool export basefs
    # reboot

    Now lets talk things you will need to do after reboot.

    2.0. At boot loader select boot into single user mode

    4. Boot FreeBSD in single user mode

    Code:
    Enter full pathname of shell or RETURN for /bin/sh: /bin/csh
    % /rescue/mount -w /
    % /rescue/zpool import -D || /rescue/zpool import -f basefs
    % exit

    2.1. Login as root without password
    Code:
    login: root
    password: (just hit ENTER)
    

    2.2. Set root password
    Code:
    # passwd

    2.3. Set hostname
    Code:
    # echo hostname=\"HOSTNAME\" >> /etc/rc.conf

    2.4. Set timezone and date/time
    Code:
    # tzsetup
    # date 201001142240

    2.5. Tune the ZFS filesystem (only for i386)
    Code:
    # cat > /boot/loader.conf << __EOF__
    vfs.zfs.prefetch_disable=0      # enable prefetch
    vfs.zfs.arc_max=134217728       # 128 MB
    vm.kmem_size=536870912          # 512 MB
    vm.kmem_size_max=536870912      # 512 MB
    vfs.zfs.vdev.cache.size=8388608 #   8 MB
    __EOF__

    2.6. Mount /tmp on SWAP
    Code:
    # cat >> /etc/rc.conf << __EOF__
    tmpmfs="YES"
    tmpsize="512m"
    tmpmfs_flags="-m 0 -o async,noatime -S -p 1777"
    __EOF__

    2.7. Move termcap into /etc (instead of useless link on crash)
    Code:
    # rm /etc/termcap
    # mv /usr/share/misc/termcap /etc
    # ln -s /etc/termcap /usr/share/misc/termcap

    2.8. Add latest security patches
    Code:
    # freebsd-update fetch
    # freebsd-update install

    2.9. Make all changes to configuration in /etc, then set / to be mounted read-only in /etc/fstab
    Code:
    DISKS: 3                                           | DISKS: 1
     #dev                #mount #fs  #opts #dump #pass |  #dev              #mount #fs  #opts #dump #pass
    +/dev/mirror/rootfsa /      ufs  ro    1     1     | +/dev/label/rootfs /      ufs  ro    1     1
    -/dev/mirror/rootfsa /      ufs  rw    1     1     | -/dev/label/rootfs /      ufs  rw    1     1
     /dev/label/swap0    none   swap sw    0     0     |  /dev/label/swap   none   swap sw    0     0
     /dev/label/swap1    none   swap sw    0     0     |
     /dev/label/swap2    none   swap sw    0     0     |

    2.10. Reboot and enjoy modern install of FreeBSD system
    Code:
    # shutdown -r now
     
    zeissoctopus, Unixworld, z0ran and 5 others thank for this.
  3. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    To summarise, this setup provides us these things:
    -- bulletproof 'base system' [1] on UFS (w/o SU) mounted read-only
    -- /tmp filesystem mounted on SWAP
    -- usage of new AHCI mode in FreeBSD
    -- flexibility for all other filesystems on ZFS
    -- fully working environment on crash (/etc/termcap)
    -- disks/filesystems mounted by label, possible device name changes are harmless
    -- RAID1 for / and RAID5 for all other systems on setup with 3 disks

    [1] base system is / and /usr while this setup, in context of this setup I name 'base system'
    the most important core of FreeBSD, the / filesystem and its binaries/libraries/configuration
    (thanks to phoenix for reminding me what REAL base system is/means) ;)


    CHANGELOG

    1.0 / 2010-01-14 / initial version
    1.1 / 2010-01-15 / simplified PATH
    +fixit# setenv PATH /mnt2/rescue:/mnt2/usr/bin
    -fixit# setenv PATH /mnt2/bin:/mnt2/sbin:/mnt2/usr/bin:/mnt2/usr/sbin
    1.2 / 2010-01-15 / added link for termcap (instead of duplicate on /etc and /usr) [2.6.]
    .# rm /etc/termcap
    +# mv /usr/share/misc/termcap /etc
    +# ln -s /etc/termcap /usr/share/misc/termcap

    -# cp /usr/share/misc/termcap /etc
    1.3 / 2010-01-21 / removed unneeded mount commands [2.0.]
    -# zfs mount basefs/var
    -# zfs mount basefs/usr
    1.4 / 2010-03-08 / added setup for 3 disks + cleanup
    too much to fit here, we can as well call this new version RELOADED ;)

    MIRROR THREAD: http://daemonforums.org/showthread.php?t=4200
    POLISH VERSION: http://bsdguru.org/dyskusja/viewtopic.php?t=19392

    ADDED: 2010/10/21


    After rethinking setup from my HOWTO and after phoenix thoughts I currently use that setup for most FreeBSD installations that include ZFS.

    LOGICAL SETUP

    Code:
    UFS 512m /           ro
    ZFS *    /home       rw | atime=off
    RAM 128m /tmp        rw | async
    UFS *    /usr        ro | softupdates (mounted r/w only for packages updates)
    ZFS *    /usr/obj    rw | atime=off | checksum=off
    ZFS *    /usr/ports  rw | atime=off
    ZFS *    /usr/src    rw | atime=off
    ZFS *    /var        rw
    UFS 128m /var/db/pkg ro | softupdates (mounted r/w only for packages updates)


    PHYSICAL SETUP (LAPTOP w/ 1 DISK)

    Code:
    p1 8g disk0s1a 512m UFS /           newfs -m 1    /dev/label/root
          disk0s1e 128m UFS /var/db/pkg newfs -m 1 -U /dev/label/pkg
          disk0s1f    * UFS /usr        newfs -m 1 -U /dev/label/usr
    
    p2 *g disk0s2  ZFS/home             zfs create -o mountpoint=/home      pool/home
                   ZFS/var              zfs create -o mountpoint=/var       pool/var
                   ZFS/usr              zfs create -o mountpoint=none       pool/usr
                   ZFS/usr/src          zfs create -o mountpoint=/usr/src   pool/usr/src
                   ZFS/usr/obj          zfs create -o mountpoint=/usr/obj   pool/usr/obj
                   ZFS/usr/ports        zfs create -o mountpoint=/usr/ports pool/usr/ports
    
                   (if You need SWAP, omit on CF/Pendrive/SSD disks)
                   ZFS/SWAP             zfs create -V 2g                    pool/swap
    
    RAM/SWAP 128m  /tmp                 tmpmfs=YES --> /etc/rc.conf


    PHYSICAL SETUP (CF + DISKS)

    Code:
    8g CF    disk0s1a 512m UFS /           newfs -m 1    /dev/label/root
             disk0s1e 128m UFS /var/db/pkg newfs -m 1 -U /dev/label/pkg
             disk0s1f    * UFS /usr        newfs -m 1 -U /dev/label/usr
    
    *g ZFS   ZFS/home                      zfs create -o mountpoint=/home      pool/home
             ZFS/var                       zfs create -o mountpoint=/var       pool/var
             ZFS/usr                       zfs create -o mountpoint=none       pool/usr
             ZFS/usr/src                   zfs create -o mountpoint=/usr/src   pool/usr/src
             ZFS/usr/obj                   zfs create -o mountpoint=/usr/obj   pool/usr/obj
             ZFS/usr/ports                 zfs create -o mountpoint=/usr/ports pool/usr/ports
    
             (if You need SWAP)
             ZFS/SWAP                      zfs create -V 2g                    pool/swap
    
    128M RAM /tmp                          tmpmfs=YES --> /etc/rc.conf


    Of course for serious storage/backup servers it would be 'nice' to have that CF (or pendrive) mirrored via GEOM/mirror.
     
    zeissoctopus, Unixworld, zeroseven and 2 others thank for this.
  4. graudeejs

    graudeejs Well-Known Member

    Messages:
    4,594
    Thanks Received:
    633
    hmm, are you able to boot into single user mode with ZFS?
    I can't for some reason, maybe because my HDD's are encrypted. But there are devices with eli (decrypted), weird
     
  5. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    @killasmurf86

    / is on UFS (with bsdlabel) so no problem to boot into single user mode, I havent played with encrypted / to check here, maybe I will in some free time @ virtualbox.
     
  6. graudeejs

    graudeejs Well-Known Member

    Messages:
    4,594
    Thanks Received:
    633
    Encrypted root with UFS will work :D, bin there done that :D
     
  7. dewarrn1

    dewarrn1 New Member

    Messages:
    14
    Thanks Received:
    0
    Couple of questions about implementation and migration

    First off, thanks to Vermaden for posting this! It's a very slick way to take advantage of the best filesystems that FreeBSD offers. I'm considering migrating a home server to 8.0, and this seems like a great setup for me. I've got a couple of questions before I give it a try, though.

    First, would including a fourth disk be as simple as it looks? I've got 7.2's ZFS spanning 3 disks at the moment, but I've got a fourth sitting around and figure that it might as well be in the server.

    Second, do you have any recommendations for maintaining data integrity during the move? I've got external HDD's that can hold all of my stuff, but they're just FAT32 and lack the kind of checksum protection that ZFS is giving me on the current system. My plan would be to use all three of the current disks and a fourth in the new system, but that will require wiping out the current filesystem.

    Thanks in advance!
     
  8. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    @dewarrn1

    It will fit well on 4 disks, but it will require to recreate the ZFS pool, you will just create raidz over 4 disks.

    You can tar(1) and split(1) all your data into that fat32 filesystem (parts need to be smaller then 4GB), You may as well create TWO copies of your data there, in two folders, or just make UFS filesystem there.
     
    dewarrn1 thanks for this.
  9. dewarrn1

    dewarrn1 New Member

    Messages:
    14
    Thanks Received:
    0
    Very cool, I'll get that process underway. I'll probably generate some par2 data for those split tar files as well, just in case. Thanks!
     
  10. dewarrn1

    dewarrn1 New Member

    Messages:
    14
    Thanks Received:
    0
    This worked almost exactly as advertised! I ended up with 4x500GB HDDs with a 4-way mirrored base system, 4 GB swap spread across the disks, and ~1.3TB ZFS. My only hiccup was the "zpool import -D" bit, which for some reason didn't want to play nice. However, "zpool import basefs" did the trick, and now I'm getting things back onto ZFS. Nice work, V!
     
  11. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    Good to know that it also works for others ;)

    The first version included zpool import basefs, but after messing with 3 disks it imported with zpool import -D so I changed the formula, I think I will include both just in case, thanks.
     
  12. Kami

    Kami New Member

    Messages:
    7
    Thanks Received:
    3
    I've used your guide and had the same problem as dewarrn1 when trying to import the pool ( zpool import -D), but zpool import basefs did the trick here as well :)

    Oh, I installed the system on 2 drives and used raid1 for all the slices (/, /usr and /var).

    I plan to add another disk in the future and I will keep you updated how the process of adding another disk to the zfs pool goes.

    Anyway, great guide.
     
  13. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    But remember that You would have to destroy the current mirror and then create RAIDZ for example.

    Thanks mate.
     
  14. mefizto

    mefizto Member

    Messages:
    133
    Thanks Received:
    0
    Dear vermaden or anybody,

    would it be possible/prudent to modify the installation to have the entire / (USB) and /usr (ZFS) on one (mirrored) disk, in particular flash and having the rest of the file system, i.e., SWAP, /tmp, /var, /home, etc., on RAIDed hard drives using ZFS?

    The motivation would be to further separate the OS/Application (fairly static on my system) from the data.

    Thank you,

    M
     
  15. graudeejs

    graudeejs Well-Known Member

    Messages:
    4,594
    Thanks Received:
    633
    I don't see any reason why this couldn't be done.

    P.S.
    Are you from Latvian Linux center?
     
  16. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    @mefizto

    If You want both / and /usr on separate disks/USB then it would be better, to create RAID 1 with gmirror on that USB drives, and use UFS for /usr on that USB disk, and then put all other filesystems with swap on remaining harddisks.
     
  17. mefizto

    mefizto Member

    Messages:
    133
    Thanks Received:
    0
    Dear killasmurf86,

    thank you for the reply. And, no, I am not from Latvian Linux center.

    Dear vermaden,

    Code:
    . . .it would be better, to create RAID 1 with gmirror on that USB drives, . . .


    That was what I meant by "to have the entire / (USB) and /usr (ZFS) on one (mirrored) disk". Sorry for my imprecise English.

    Code:
    . . .use UFS for /usr on that USB disk. . .


    What would be the advantage of using UFS instead of ZFS for /usr?

    If it is not much bother, could you indicate, at least in general terms, which parts of your procedure have to be changed?

    Thank you,

    M
     
  18. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    Keep small / and /usr (base system) on UFS (in case any problems with ZFS) to have fully working 'repair' environment, put all the rest on ZFS tank.

    About changes, You would have to create additional UFS partition (e) for /usr of course in section 1.4:

    Code:
    # cat > label << __EOF__
    # /dev/ad0s1:
    8 partitions:
      a: 512m  0 4.2BSD
      e: *     * 4.2BSD
    __EOF__
     
  19. mefizto

    mefizto Member

    Messages:
    133
    Thanks Received:
    0
    Dear vermaden,

    thank you for your help.

    Kindest regards,

    M
     
  20. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    @mefizto

    You are welcome mate.
     
  21. Daisuke_Aramaki

    Daisuke_Aramaki New Member

    Messages:
    220
    Thanks Received:
    29
    Great guide vermaden. Worked out perfect setting the system bottom up on a new laptop.
     
  22. vermaden

    vermaden Well-Known Member

    Messages:
    2,361
    Thanks Received:
    677
    @Daisuke_Aramaki

    Thanks 'n' Welcome ;)
     
  23. mystique

    mystique New Member

    Messages:
    13
    Thanks Received:
    0
    Code:
    DISKS: 3                                            
    # cat > part << __EOF__                               
    p 1 165 63  512M                                     
    p 2 165  * 1024M                                      
    p 3 159  *     *                                      
    p 4   0  0     0                                    
    a 1                                                
    __EOF__                                               
                                                          
    # fdisk -f part ad0                               
    # fdisk -f part ad1                               
    # fdisk -f part ad2                                   
    


    My disks are ad4, ad6, ad8 and ad10..

    what would I change here? because this is not working for me;

    I understand fdisk -f part ad4 would be the right command but when I do that I get this error:

    Code:
    ******* Working on device /dev/ad4 *******
    fdisk: invalid fdisk partition table found
    fdisk: geom not found: "ad4"


    thanks in advance..
     
  24. mystique

    mystique New Member

    Messages:
    13
    Thanks Received:
    0
    So it looks like I found that I was not using the full path to add the kernel modules and was getting an error that I over looked.. hence why it could not find ad4; *blush* my bad..

    So after that I was able to finish the install but when I get to the 'reboot in single user mode' I have some issues..

    I have a usb keyboard and can not use the keyboard at boot time; odd..
    when it boots I get the mountroot> prompt; also odd..

    but when I look back through dmesg I do not see my (ad4, ad6, ad8, ad10) but rather I see ada0, ada1, ada2, ada3..

    I am assuming that has *something* to do with ahci (something I've not used yet)

    so what do I do with that?

    Code:
    # fdisk -f part ada0                                    
    # fdisk -f part ada1                               
    # fdisk -f part ada2
    # fidsk -f part ada3


    but do I need to load the ahci module beforehand?

    thanks in advance.
     
  25. zeroseven

    zeroseven New Member

    Messages:
    23
    Thanks Received:
    2
    I may be partially in the wrong place to ask this question. I have a sparc64 system and before I set myself up for a lot of desk to face action, I was wondering if I could accomplish this sort of install using the livefs in combination with disk1?