HOWTO: Modern FreeBSD Install (vermaden way)

All these years sysinstall(8) was helping us to install FreeBSD with most options we needed, today with new filesystems/features like GJournal/ZFS/Geli/GMirror/GStripe its no longer up to the task, because it only supports creating installation on UFS filesystem with SoftUpdates turned ON or OFF.

In this guide you will learn how to setup FreeBSD installation in simple yet flexible setup based on read-only UFS (without SoftUpdates) for 'base system' [1], some SWAP space, /tmp mounted on SWAP and all the other filesystems (/var /usr ...) mounted on ZFS. It will not require rebuilding anything, just simple setup on plain MBR partitions. I should also mention that we would be using AHCI mode for disks.

Here is the layout of the DISK:

Code:
             +       /   512 MB --> UFS/read-only 
MBR SLICE 1  +  [SWAP]     2 GB
             +    /tmp     1 GB --> mounted on SWAP with[B] mdmfs(8)[/B]
-----------------------------------------------------------------
             +    /usr        - --> ZFS
MBR SLICE 2  +    /var        - --> ZFS
             +     ...        - --> ZFS

FreeBSD core, the 'base system' [1] should remain almost unchanged/untouched on daily basis while you can mess all other filesystems, this ensures that when things go wrong, you will be able to fix anything still having working 'base system' [1].

You will need *-dvd-* disk or *-memstick-* image for this installation, *-disk1-* will not do since it does not contain livefs system.

Here is the procedude, described as simple as possible.

1.0. I assume that our disk for the installation would be /dev/ad0

1.1. Boot *-dvd-* from DVD disk or *-memstick-* image from pendrive
Code:
Country Selection --> United States
Fixit --> CDROM/DVD ([file]*-dvd-*[/file]) or USB ([file]*-memstick-*[/file])

1.2. Create your temporary working environment
Code:
fixit# [color="Blue"]/mnt2/bin/csh[/color]
fixit# [color="blue"]setenv PATH /mnt2/rescue:/mnt2/usr/bin[/color]
fixit# [color="blue"]set filec[/color]
fixit# [color="blue"]set autolist[/color]
fixit# [color="blue"]set nobeep[/color]

1.3. Load needed modules
Code:
fixit# [color="#0000ff"]kldload /mnt2/boot/kernel/geom_mbr.ko[/color]
fixit# [color="blue"]kldload /mnt2/boot/kernel/opensolaris.ko[/color]
fixit# [color="blue"]kldload /mnt2/boot/kernel/zfs.ko[/color]

1.4. Create/mount needed filesystems
Code:
fixit# [color="blue"]cat > part << __EOF__
p 1 165 63 2560M
p 2 165  *     *
p 3   0  0     0
p 4   0  0     0
a 1
__EOF__[/color]
fixit# [color="blue"]fdisk -f part ad0[/color]

fixit# [color="blue"]bsdlabel -B -w ad0s1[/color]
fixit# [color="blue"]cat > label << __EOF__
# /dev/ad0s1:
8 partitions:
  a: 512m  0 4.2BSD
  b: *     * swap
__EOF__[/color]
fixit# [color="blue"]bsdlabel ad0s1 | tail -1 >> label[/color]
fixit# [color="blue"]bsdlabel -R ad0s1 label[/color]

fixit# [color="Blue"]glabel label rootfs ad0s1a[/color]
fixit# [color="blue"]glabel label swap ad0s1b[/color]

fixit# [color="blue"]newfs /dev/label/rootfs[/color]
fixit# [color="blue"]zpool create basefs ad0s2[/color]
fixit# [color="blue"]zfs create basefs/usr[/color]
fixit# [color="blue"]zfs create basefs/var[/color]

fixit# [color="blue"]mkdir /NEWROOT[/color]
fixit# [color="blue"]mount /dev/label/rootfs /NEWROOT[/color]
fixit# [color="blue"]zfs set mountpoint=/NEWROOT/usr basefs/usr[/color]
fixit# [color="blue"]zfs set mountpoint=/NEWROOT/var basefs/var[/color]

1.5. Actually install needed FreeBSD system sets
Code:
fixit# [color="blue"]setenv DESTDIR /NEWROOT[/color]
fixit# [color="blue"]cd /dist/8.0-RELEASE[/color]

fixit# [color="blue"]cd base[/color]
fixit# [color="blue"]./install.sh [/color]# answer [I]'y'[/I] here
fixit# [color="blue"]cd ..[/color]

fixit# [color="blue"]cd manpages[/color]
fixit# [color="blue"]./install.sh[/color]
fixit# [color="blue"]cd ..[/color]

fixit# [color="Blue"]cd kernels[/color]
fixit# [color="blue"]./install.sh generic[/color]
fixit# [color="Blue"]cd ..[/color]

fixit# [color="Blue"]cd /NEWROOT/boot[/color]
fixit# [color="Blue"]rm -r kernel[/color]
fixit# [color="Blue"]mv GENERIC kernel[/color]

1.6. Provide basic configuration needed to boot new system
Code:
fixit# [color="Blue"]cat > /NEWROOT/etc/fstab << __EOF__
#dev                #mount   #fs   #opts   #dump   #pass
/dev/label/rootfs   /        ufs   rw      2       2
/dev/label/swap     none     swap  sw      0       0
basefs/var          /var     zfs   rw      0       0
basefs/usr          /usr     zfs   rw      0       0
__EOF__[/color]

fixit# [color="Blue"]cat > /NEWROOT/boot/loader.conf << __EOF__
zfs_load="YES"
ahci_load="YES"
__EOF__
[/color]
fixit# [color="Blue"]zfs set mountpoint=/usr basefs/usr[/color]
fixit# [color="Blue"]zfs set mountpoint=/var basefs/var[/color]

1.7. Unmount filesystems and reboot
Code:
fixit# [color="Blue"]umount /NEWROOT/var[/color]
fixit# [color="Blue"]umount /NEWROOT/usr[/color]
fixit# [color="Blue"]umount /NEWROOT[/color]
fixit# [color="Blue"]reboot[/color]

Now lets talk things you will need to do after reboot.

2.0. If you will be dropped into 'single user mode'
Use these commands to import and mount your basefs zpool:
Code:
newsystem# [color="Green"]zpool import -f basefs[/color]

2.1. Set root password
Code:
newsystem# [color="Green"]passwd[/color]

2.2. Set hostname
Code:
newsystem# [color="Green"]echo hostname=\"HOSTNAME\" >> /etc/rc.conf[/color]

2.3. Set timezone and date
Code:
menu# [color="Green"]tzsetup[/color]
manual# [color="Green"]cat /usr/share/zoneinfo/Europe/Warsaw > /etc/localtime[/color]
time# [color="Green"]date 201001142240[/color]

2.4. Tune ZFS filesystem (only for i386)
Code:
newsystem# [color="Green"]cat > /boot/loader.conf << __EOF__
vfs.zfs.prefetch_disable=0      # enable prefetch
vfs.zfs.arc_max=134217728       # 128 MB
vm.kmem_size=536870912          # 512 MB
vm.kmem_size_max=536870912      # 512 MB
vfs.zfs.vdev.cache.size=8388608 #   8 MB
__EOF__[/color]

2.5. Mount /tmp on swap
Code:
newsystem# [color="Green"]cat >> /etc/rc.conf << __EOF__
tmpmfs="YES"
tmpsize="1024m"
tmpmfs_flags="-m 0 -o async,noatime -S -p 1777"
__EOF__[/color]

2.6. Move termcap into /etc (instead of useless link on crash)
Code:
newsystem# [color="Green"]rm /etc/termcap[/color]
newsystem# [color="Green"]mv /usr/share/misc/termcap /etc[/color]
newsystem# [color="Green"]ln -s /etc/termcap /usr/share/misc/termcap[/color]

2.7. Add latest security patches
Code:
newsystem# [color="Green"]freebsd-update fetch[/color]
newsystem# [color="Green"]freebsd-update install[/color]

2.8. Make all changes to configuration in /etc, then set / to be mounted read-only in /etc/fstab
Code:
 #dev                #mount   #fs   #opts   #dump   #pass
[color="Green"]+/dev/label/rootfs   /        ufs   ro      2       2[/color]
[color="Red"]-/dev/label/rootfs   /        ufs   rw      2       2[/color]
 /dev/label/swap     none     swap  sw      0       0

2.9. Reboot and enjoy modern install of FreeBSD system
Code:
newsystem# [color="Green"]shutdown -r now[/color]

To summarise, this setup provides us these things:
-- bulletproof 'base system' [1] on UFS (w/o SU) mounted read-only
-- /tmp filesystem mounted on SWAP
-- usage of new AHCI mode in FreeBSD
-- flexibility for all other filesystems on ZFS
-- fully working environment on crash (/etc/termcap)
-- disks/filesystems mounted by label, possible device name changes are harmless

[1] base system is / and /usr while this setup, in context of this setup I name 'base system'
the most important core of FreeBSD, the / filesystem and its binaries/libraries/configuration
(thanks to phoenix for reminding me what REAL base system is/means) ;)

CHANGELOG

1.0 / 2010-01-14 / initial version
1.1 / 2010-01-15 / simplified PATH
+fixit# setenv PATH /mnt2/rescue:/mnt2/usr/bin
-fixit# setenv PATH /mnt2/bin:/mnt2/sbin:/mnt2/usr/bin:/mnt2/usr/sbin
1.2 / 2010-01-15 / added link for termcap (instead of duplicate on /etc and /usr) [2.6.]
.newsystem# rm /etc/termcap
+newsystem# mv /usr/share/misc/termcap /etc
+newsystem# ln -s /etc/termcap /usr/share/misc/termcap

-newsystem# cp /usr/share/misc/termcap /etc
1.3 / 2010-01-21 / removed unneeded mount commands [2.0.]
-newsystem# zfs mount basefs/var
-newsystem# zfs mount basefs/usr

MIRROR THREAD: http://daemonforums.org/showthread.php?t=4200
POLISH VERSION: http://bsdguru.org/dyskusja/viewtopic.php?t=19392
 
Thanks anomie.

killasmurf86 said:
Why /tmp on swap?
Its casual for any 'big iron unix' (AIX/Solaris/...) to mount /tmp on SWAP.

What is typical /tmp filesystem utilization, a lot of small files, often modified, its very similar to pagging space behaviour, so that is one of the reasons to mount /tmp on swap.

Also, /tmp is empty for most of the time, why waste space (for example 512m or 1g) and believe that some day some application will eventually use it, its waste.

killasmurf86 said:
And on which arch did you do this on i386? [AMD64 don't need tuning AFAIK]
The guide was done on i386, yes ZFS does not need tuning on amd64, I will add some notice to the guide, thanks.
 
I was jsut wondering, since I use 100% zfs and only boot from ufs flash, I don't care if I have/don't have /tmp on separate FS :)

P.S. I use /tmp quite a lot
 
killasmurf86 said:
I was jsut wondering, since I use 100% zfs and only boot from ufs flash, I don't care if I have/don't have /tmp on separate FS :)
Using /tmp on ZFS solves wasted space problem, but that does not solves fragmentation problem (a lot of small files) ;)

killasmurf86 said:
P.S. I use /tmp quite a lot
Examples?
 
vermaden said:
Using /tmp on ZFS solves wasted space problem, but that does not solves fragmentation problem (a lot of small files) ;)
I haven't noticed so much small files there.
Here's what i have right now:
Code:
$ find /tmp                                                       
/tmp
/tmp/.XIM-unix
/tmp/.font-unix
/tmp/ssh-x5yc3hPQKo
/tmp/ssh-x5yc3hPQKo/agent.1419
/tmp/.X11-unix
/tmp/.X11-unix/X0
/tmp/gpg-PT0KRV
/tmp/gpg-PT0KRV/S.gpg-agent
/tmp/.ICE-unix
/tmp/.X0-lock


vermaden said:
Examples?
Whenever I download something, that I won't save on my PC i stave it to /tmp. [on my desktop]
If I need to burn cd I download image to /tmp....


However I somewhat like the idea about /tmp on swap... will think about that and perhaps try it :D
 
Code:
tmpmfs_flags="-m 0 -o async,noatime [red]-S[/red] -p 1777"
Why soft updates are turned off {-S}? They kinda speed up writing, also you can set -O time
 
killasmurf86 said:
Code:
tmpmfs_flags="-m 0 -o async,noatime [red]-S[/red] -p 1777"
Why soft updates are turned off {-S}? They kinda speed up writing, also you can set -O time
Havent done any benchmarks comparing to SoftUpdates, but its already mounted in asynchronous mode (mount -o async), -o time is default when there is a lot free space and it changes dynamically to -o space when there is less then 8% free space.
 
Well, Soft Updates are on by default... so it's probably good.

I remember writing to flash with and without soft updates.....
Without soft updates it's so slow... uhhh
 
Haven't finished reading it yet, one quick clarification: "base system" refers to everything under / and /usr (except /usr/ports and /usr/local), not just /. Pedantic/semantics, maybe. :)

(Where was this 18 months ago when I started with gmirror for / with ZFS?) ;)
 
Thanks for reminding me, I kind of ?forgot? what the name base system covers.

phoenix said:
(Where was this 18 months ago when I started with gmirror for / with ZFS?) ]
Heh, I had a little break from FreeBSD (playing with Linux/OpenSolaris), hardware problem was also involved, but I must say that after setting 8.0 exclusivelly on my box (and laptop earlier) I must say that I missed the robustness and reliability of FreeBSD, its sollutions, generally the 'FreeBSD way' that all of us know.

IMHO, by messing with Linux, trying newer releases of OpenSolaris alpha/beta builds I feel like I wasted that time/vegetate intead of focusing on to make the job done.
 
Cool guide. I think I'm gonna try it.

Just one issue (and this may be completely wrong..) but if the base system is read only, does this mean its not possible to cvsup and make {build,install}world anymore?

Cheers
 
caesius said:
Cool guide. I think I'm gonna try it.

Just one issue (and this may be completely wrong..) but if the base system is read only, does this mean its not possible to cvsup and make {build,install}world anymore?

Cheers

you can always # mount -uo rw /
 
@mfaridi

Only 2.4. section has info to omit it on amd64 ...
[CMD=""]2.4. Tune ZFS filesystem (only for i386)[/CMD]

@caesius

Thanks mate.

caesius said:
Just one issue (and this may be completely wrong..) but if the base system is read only, does this mean its not possible to cvsup and make {build,install}world anymore?

You will have to remount it as read-write mode like that: # mount -u -w / ... and then bring it back to read-only by # mount -u -r / ([CMD=""]-o rw[/CMD] | [CMD=""]-o ro[/CMD] does the same).

You may also read about some other issues on read-only / on FreeBSD here:
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/questions/2008-06/msg01029.html
 
mfaridi said:
Code:
fdisk: Geom not found : ad4

If you only remembered to # kldload /mnt2/boot/kernel/geom_mbr.ko, then this warning is harmless, and it works (check # fdisk ad4 after that command).
 
in section 1.5
when I go kernels
and type
Code:
./install.sh generic
I see this error
tr:not found
cat: .???: not such file or diretory
 
after reboot and start new , when I run this command
Code:
zpool create basefs ad4s2
I see this error
Code:
invalid vdev soecification
use -f to override the following error
/dev/ad4s2 is part of potentially active pool 'basefs'
 
Back
Top