HOWTO: Install FreeBSD 9.0 RELEASE (Root on UFS + ZFS, RAID1)

[vand777]

I'll show how to install FreeBSD 9.0 RELEASE on the machine with 2 disks (RAID 1). I'll use the following logical layout:


UFS 1G / ro
ZFS * /home rw
ZFS * /tmp rw
UFS * /usr ro | softupdates (mounted r/w only for packages updates)
ZFS * /usr/obj rw
ZFS * /usr/ports rw
ZFS * /usr/src rw
ZFS * /var rw
UFS 1G /var/db/pkg ro | softupdates (mounted r/w only for packages updates)

[vand777]

(1) Boot from a FreeBSD9 installation DVD or memstick and choose "Live CD".

(2) Load the necessary kernel modules.

# kldload geom_mbr.ko
# kldload geom_mirror.ko
# kldload opensolaris.ko
# kldload zfs.ko


(3) Create the necessary partitions on the disks.

# sh
# disk0=da0
# disk1=da1
# gpart create -s gpt $disk0
# gpart add -b 34 -s 94 -t freebsd-boot $disk0
# gpart add -s 1G -t freebsd-ufs $disk0
# gpart add -s 4G -t freebsd-swap $disk0
# gpart add -s 1G -t freebsd-ufs $disk0
# gpart add -s 8G -t freebsd-ufs $disk0
# gpart add -t freebsd-zfs $disk0
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 $disk0
# gpart backup $disk0 | gpart restore -F $disk1


(4) Create RAID1 for UFS and swap filesystems.

# gmirror label root ${disk0}p2 ${disk1}p2
# gmirror label swap ${disk0}p3 ${disk1}p3
# gmirror label pkg ${disk0}p4 ${disk1}p4
# gmirror label usr ${disk0}p5 ${disk1}p5

[vand777]

(5) Create filesystems and mount them.

# newfs -m 1 /dev/mirror/root
# newfs -m 1 -U /dev/mirror/pkg
# newfs -m 1 -U /dev/mirror/usr
# mkdir /tmp/NEWROOT
# mount /dev/mirror/root /tmp/NEWROOT
# mkdir /tmp/NEWROOT/usr
# mount /dev/mirror/usr /tmp/NEWROOT/usr
# zpool create -m /tmp/NEWROOT/basefs basefs mirror ${disk0}p6 ${disk1}p6
# zfs create -o mountpoint=/tmp/NEWROOT/home basefs/home
# zfs create -o mountpoint=/tmp/NEWROOT/var basefs/var
# zfs create -o mountpoint=/tmp/NEWROOT/tmp basefs/tmp
# zfs create -o mountpoint=none basefs/usr
# zfs create -o mountpoint=/tmp/NEWROOT/usr/obj basefs/usr/obj
# zfs create -o mountpoint=/tmp/NEWROOT/usr/ports basefs/usr/ports
# zfs create -o mountpoint=/tmp/NEWROOT/usr/src basefs/usr/src


P.S. Feel free to play with ZFS filesystems options and create additional filesystems if you need/want.

(6) Install FreeBSD.

# cd /usr/freebsd-dist
# export DESTDIR=/tmp/NEWROOT
# for file in base.txz lib32.txz kernel.txz;
# do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

[vand777]

(7) Provide basic configuration.

# cat > /tmp/NEWROOT/etc/fstab << __EOF__
#dev #mount #fs #opts #dump #pass
/dev/mirror/root / ufs rw 1 1
/dev/mirror/swap none swap sw 0 0
/dev/mirror/usr /usr ufs rw 2 2
basefs/var /var zfs rw 0 0
/dev/mirror/pkg /var/db/pkg ufs rw 2 2

basefs /basefs zfs rw 0 0
basefs/home /home zfs rw 0 0
basefs/usr/obj /usr/obj zfs rw 0 0
basefs/usr/ports /usr/ports zfs rw 0 0
basefs/usr/src /usr/src zfs rw 0 0
basefs/tmp /tmp zfs rw 0 0
__EOF__

# cat > /tmp/NEWROOT/boot/loader.conf << __EOF__
zfs_load="YES"
ahci_load="YES"
geom_mirror_load="YES"
__EOF__

# cat > /tmp/NEWROOT/etc/rc.conf << __EOF__
zfs_enable="YES"
__EOF__


(8) Unmount filesystems and reboot.

# cd /
# zfs umount -a
# umount /tmp/NEWROOT/usr
# umount /tmp/NEWROOT
# zfs set mountpoint=/home basefs/home
# zfs set mountpoint=/var basefs/var
# zfs set mountpoint=/tmp basefs/tmp
# zfs set mountpoint=/usr/obj basefs/usr/obj
# zfs set mountpoint=/usr/ports basefs/usr/ports
# zfs set mountpoint=/usr/src basefs/usr/src
# zpool export basefs
# reboot

[vand777]

(9) Boot FreeBSD in single user mode.

# mount -w /
# zpool import -f basefs
# reboot


(10) Set root password

# passwd


(11) Set hostname

# echo hostname=\"HOSTNAME\" >> /etc/rc.conf


(12) Set timezone

# tzsetup


(13) Move termcap into /etc (instead of useless link on crash)

# rm /etc/termcap
# mv /usr/share/misc/termcap /etc
# ln -s /etc/termcap /usr/share/misc/termcap


(14) Setup networking and do any other necessary changes to /etc.

(15) Add latest security patches.

# freebsd-update fetch
# freebsd-update install


(16) Set / to be mounted read-only in /etc/fstab.

+/dev/mirror/root / ufs ro 1 1
-/dev/mirror/root / ufs rw 1 1


(17) Reboot and enjoy.

# reboot

[vand777]

Reserved 1

[vand777]

Reserved 2

[vand777]

Reserved 3

[vand777]

Reserved 4

[phoenix]

Just a quick note: if you have a separate /home, then you don't need the symlink.

[vand777]

Just a quick note: if you have a separate /home, then you don't need the symlink.

Thank you. Will remove it from the guide in a second.

[RusDyr]

I never can understand why do you need create such a numbers of [zfs] partitions like /usr/ports, /usr/obj/, /usr/ports/distfiles and so on. It's excessive, doesn't neccesary for quick installation and obviusly can be done later.
This zfs partitioning way is in almost all howtos. Does anybody think the sense of it?

[matoatlantis]

I never can understand why do you need create such a numbers of [zfs] partitions like /usr/ports, /usr/obj/, /usr/ports/distfiles and so on. It's excessive, doesn't neccesary for quick installation and obviusly can be done later.
This zfs partitioning way is in the almost each howtos. Does anybody think the sense of it?

With ZFS, these tasks are much easier indeed; using UFS-only setup you have to do it beforehand. But I think it's _the_ perfect place to do it during installation. You're not on time trial when installing new BSD, are you? It only takes couple of seconds to do it right.

[RusDyr]

What's the point? There is no difference in /usr/ports, /usr/obj and so on at least in current ZFS configuration.

[gkontos]

What's the point? There is no difference in /usr/ports, /usr/obj and so on at least in current ZFS configuration.

There is a difference because if you want to set specific properties to those datasets, compression for example, you have to do it before they get populated.

/usr/src is a good candidate for compression but if you set this after the installation then you pretty much don't have any.

/usr/ports is another good candidate. But /usr/ports/distfiles is not since it usually contains compressed files.

Of course all those are suggestions and anybody can improvise.

Regards

[matoatlantis]

What's the point? There is no difference in /usr/ports, /usr/obj and so on at least in current ZFS configuration.

For example, when you do a /usr snapshot. I don't care about /usr/ports/* , /usr/src/* at all.
You don't need /usr/ports for kernel debugging; /usr/obj can be shared to others too (when possible). So it does make sense to separate it.


Reasons for separate FS?

* backup/restore (do I need to back it up?) - restore time
* mount options (performance and/or security)
* implications when FS gets full or corrupted (can it be restored online ?)

[bbzz]

All above and zfs list show quickly how much space is taken in each fs :)

[RusDyr]

Yeah, I thought about snapshots after I've wrote it.
Freeing space by compression or deduplication isn't valuable, especially on systems where are ZFS is (storage servers, isn't it?). ;)

[gkontos]

Yeah, I thought about snapshots after I've wrote it.
Freeing space by compression or deduplication isn't valuable, especially on systems where are ZFS is (storage servers, isn't it?). ;)

Your original question was why all guides create so many ZFS datasets during installation, instead of doing this afterwards.

This has nothing to do with snapshots whatsoever. The only reason why you might want to create a dataset before installing the OS, is because you want to assign certain properties such as, compression, dedup, copies, etc.

I hope this clarifies your question.

[beesatmsu]

I am trying this now. First time I tried the # kldload geom_mbr.ko It gave me a bunch of errors. however after I tried the second line, then went back to try it again, it says file aleady exists. So I hope it was loaded.

Now I am stuck at # gpart backup $disk0 | gpart restore -F $disk1 because it says gpart: geom 'ada1': Operation not permitted

dmesg says I have ada0 and ada1, both 1TB drives. It appears ada1 is not writable.

[beesatmsu]

So I proceeded as if I had a single disk and ignored disk1. I had to omit 'mirror' and ${disk1} in the zpool command.

But now I tried exactly:

# cd /usr/freebsd-dist
# export DESTDIR=/tmp/NEWROOT
# for file in base.txz lib32.txz kernel.txz;
# do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

and got:

cat: lib32.txz: no such file or directory.

I am using a freebsd FreeBSD 9.0 bootdisk (i386).

[beesatmsu]

indeed, ls gives:

MANIFEST
doc.txz
kernel.txz
src.txz
base.txz
games.txz
ports.txz

but no lib32.txz.

[matoatlantis]

The only reason why you might want to create a dataset before installing the OS, is because you want to assign certain properties such as, compression, dedup, copies, etc.

Now that's quite some harsh statement (especially "the only" part).

[gkontos]

Now that's quite some harsh statement (especially "the only" part).

Ok, would like to elaborate?

[beesatmsu]

Oops, just found out my boot disk was 9.0-RC3, not 9.0-release as I thought. Will try again tomorrow.

[kpa]

You won't find lib32.txz on i386 install disk, it's amd64 only.

[matoatlantis]

Ok, would like to elaborate?

I think I did in my post above. You are right though - several other ZFS properties can be set, but it's far from being the only reason. And that's why it caught my eye.

[beesatmsu]

Thanks. Should I use amd64? Most of my computers are quadcore amd CPUs. But I used i386 just in case. I have one Intel CPU but it is used for windows.

What is the advantages for using amd64? i386 still works for amd CPU fine.

Zach

You won't find lib32.txz on i386 install disk, it's amd64 only.

[estrabd]

This is very nice, thank you so much!

Question - and this might be a little dense - but is there a write-up anywhere that describes use cases for ZFS and regular maintenance, etc? I am about to be setting up a new server, and would very much like to accurately determine if ZFS/RAID1 is the right set up for me. I'm just a little fuzzy on *what* that gives me exactly. Thanks!

[beesatmsu]

I am using the amd64 9.0 release today. I still get the same error as yesterday after kldload geom_mbr.ko

g_dev_taste: make_dev_p() failed (gp->name=ada0s1, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1a, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1b, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1d, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1e, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1f, error=17)

[beesatmsu]

Strange, I cannot gpart ada0 (operation not permitted), nor can I gpart ada1 (File exists). If I try to zero ada1, it says operation not permitted!

I have ada0 (ad6, SATA, 1TB) and ada1 (ad8, SATA, 1 TB).

On an AMD Phenom 9650 CPU with 4 GB memory. Booted with FreeBSD AMD64 9.0 Release (DVD) in Live CD mode.

[vand777]

What is the result of gpart show? If it already shows something, then you can delete the existing settings with the following commands:

# gpart destroy -F ada0
# gpart destroy -F ada1


And then start from the beginning.

P.S. Sorry, was in a business trip for the whole week.

[vand777]

I never can understand why do you need create such a numbers of [zfs] partitions like /usr/ports, /usr/obj/, /usr/ports/distfiles and so on. It's excessive, doesn't neccesary for quick installation and obviusly can be done later.
This zfs partitioning way is in almost all howtos. Does anybody think the sense of it?

In my guide /usr is sitting on UFS filesystem. If I need /usr/ports to be located on ZFS partition, then I have to do these steps. Yes, they can be done later... But why to postpone it to a later stage?

[beesatmsu]

vand777, thanks.

gpart shows my partitions I made yesterday following your instructions but with 9.0-RC3. I get gpart: Device busy when I tried to do it for ada0. I am using a different machine but the same two HDDs now. Thought there might be something wrong with my MB. So far it seems similar. I burned a i386 9.0_release also and will try that also today.

[beesatmsu]

Strangely the first line does not give me error now for this machine.

[beesatmsu]

vand777, thanks. Now I destroyed both ada0 and ada1 successfully. Last time ada1 was not connected properly (power cable messup). But the geom_mbr error came back. It does not seem to affect things though. I am trying to create partitions now.

[beesatmsu]

When I tried to do gpart backup $disk0 | gpart restore -F $disk1

I got:

gpart: geom 'ada1': Operation not permitted

This seems to be the same error as yesterday (different boot disk, different machine, but the same two drives). It is possible the two drives are different brands (but same cylinders etc) and this caused the error? Freebsd FreeBSD 8.1 did not have a problem with gmirror with two different drives.

[beesatmsu]

I am giving up on AMD64. Got another error with gmirror label root ${disk0}p2 I omitted disk1 since it won't allow me anything. The error I got was gmirror: Can't store metadata on ada0p2: Operation not permitted

Strangely, gpart show says ada1 now is gone (I destroyed it once), but now gpart wont find it.

[beesatmsu]

I started from scratch using the i386 disk (DVD). Still got the same error when I tried to gmirror label root ${disk0}p2 ${disk1}p2

I got: gmirror: Can't store metadata on ada0p2: Operaton not permitted.

I am about to give up zfs on 9.0.

[beesatmsu]

The errors (which happened from p2 to p5) did not seem to affect other things, but now after finishing my network does not work. Yesterday the i386 9.0 RC3 worked fine with network, rc.conf/hosts/resolv.conf files under /etc all looked fine.

[beesatmsu]

It appears the MB's ethernet card is gone while I was installing it. Put another PCI card in, not working. Another one, still not working. Having bad luck today.

[beesatmsu]

I figured out (finally!) that freebsd FreeBSD did not like my wireless router (the wired outlets worked fine with Win, but not with freebsd FreeBSD) -- I figured out because last time a working disk did not have network either. Today I moved the machine to a netgear switch and now network is working...(but Freebsd FreeBSD 9.0 did not see fxp0 but 8.0 saw it).

How do I turn on gmirror? Right now gmirror shows only the 8 gig partition as the master.

gmirror list

Sectorsize: 512
Mode: r0w0e0
Consumers:
1. Name: ada1p5
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 2147549184
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 1770742616

gpart show
=> 34 1953525101 ada0 GPT (931G)
34 94 1 freebsd-boot (47k)
128 2097152 2 freebsd-ufs (1.0G)
2097280 8388608 3 freebsd-swap (4.0G)
10485888 2097152 4 freebsd-ufs (1.0G)
12583040 16777216 5 freebsd-ufs (8.0G)
29360256 1924164879 6 freebsd-zfs (917G)

=> 34 1953525101 ada1 GPT (931G)
34 94 1 freebsd-boot (47k)
128 2097152 2 freebsd-ufs (1.0G)
2097280 8388608 3 freebsd-swap (4.0G)
10485888 2097152 4 freebsd-ufs (1.0G)
12583040 16777216 5 freebsd-ufs (8.0G)
29360256 1924164879 6 freebsd-zfs (917G)

[beesatmsu]

I also got these messages in dmesg:

GEOM_MIRROR: Device mirror/root launched (1/1).
GEOM_MIRROR: Device mirror/swap launched (1/1).
GEOM_MIRROR: Device mirror/pkg launched (1/1).
GEOM_MIRROR: Device mirror/usr launched (1/1).
GEOM_MIRROR: Device root already configured.
GEOM_MIRROR: Device swap already configured.
GEOM_MIRROR: Device pkg already configured.
GEOM_MIRROR: Device mirror/urs launched (1/1).
GEOM_MIRROR: Device root already configured.
GEOM_MIRROR: Device swap already configured.
GEOM_MIRROR: Device pkg already configured.