Root on ZFS mirror recent guide?

M

marvel

Hi all,

I've been searching the Internet for hours but I can't find any guides on how to do a freebsd FreeBSD installation on ZFS with two mirrored drives.

There's this one of course: https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror. But that seems outdated? Like this line: gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad0. It says /mnt2 but I don't have a /mnt2 at all, am I doing something wrong? I just boot 9.2-RELEASE and go directly to the shell. It also talks about choosing the Fixit option during install, but I can't find it?

If someone could point me to a guide for 9.2 I would really appreciate it.

Thanks.
 
You'll have to customize the script a bit below, but it should be mostly straightforward. Check out one of the guides on the wiki for dataset options such as compression.

Another option is to try 10.0 RC4 and then upgrade when the realese is final. It has a root on ZFS option in the installer.

Code: 
# Based on http://www.aisecure.net/2012/01/16/rootzfs/ and 
# @vermaden's guide on the forums

# Choose the "Shell" option in the installer

# Set this to your disks
DISKS="ada0 ada1"

for I in ${DISKS}; do
	NUM=$( echo ${I} | tr -c -d '0-9' )
	gpart destroy -F ${I}
	gpart create -s gpt ${I}
	gpart add -b 34 -s 94 -t freebsd-boot -l bootcode${NUM} ${I}
	gpart add -t freebsd-zfs -l disk${NUM} ${I}
	gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ${I}
	gnop create -S 4096 /dev/gpt/disk${NUM}
done

zpool create -f -o altroot=/mnt -o cachefile=/tmp/zpool.cache zroot mirror /dev/gpt/disk*.nop
zpool export zroot

for I in ${DISKS}; do
	NUM=$( echo ${I} | tr -c -d '0-9' )
	gnop destroy /dev/gpt/disk${NUM}.nop
done

zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache zroot

zpool set bootfs=zroot zroot
zfs set atime=off sys
zfs set checksum=fletcher4 zroot

zfs create zroot/usr
zfs create zroot/usr/home
zfs create zroot/var
zfs create zroot/tmp

chmod 1777 /mnt/tmp
cd /mnt ; ln -s usr/home home
chmod 1777 /mnt/var/tmp

cd /usr/freebsd-dist
export DESTDIR=/mnt
for file in base.txz kernel.txz doc.txz;
do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache

cat << EOF >> /mnt/boot/loader.conf
zfs_load=YES
vfs.root.mountfrom="zfs:zroot"
EOF

cat << EOF >> /mnt/etc/rc.conf
defaultrouter="your default router address here"
hostname="your hostname here"
# replace em0 with your nic
ifconfig_em0="inet address  netmask 255.255.255.0" or ifconconf_em0="SYNCDHCP"
keymap="us.iso"
# if you require nfsd
mountd_flags="-r" # for nfsd
nfs_client_enable="YES"
nfs_server_enable="YES"
rpcbind_enable="YES"
# sendmail
sendmail_enable="NO"
sshd_enable="YES"
zfs_enable=YES
EOF
 
Great thanks I'll give it a shot! I really don't want to use RC versions since it's for production, any idea when 10.0-RELEASE will be available?
 
A lot of those HowTos are written for FreeBSD 8. FreeBSD 9 mounts the install CD differently. Everywhere it says /mnt2 use /.
 
marvel said:
Great thanks I'll give it a shot! I really don't want to use RC versions since it's for production, any idea when 10.0-RELEASE will be available?
Click to expand...
Don't use the .0 releases for production. Wait for 10.1 if you really want to run 10 in production. The reason is that the .0 versions go end-of-life as soon as a .1 is released. The .1 versions all have extended support (2 years).
 
I really recommend the following guide (See the section 'Server with two disks') viewtopic.php?t=31662

Maybe it's just me but having used the above guide on all recent installs (works perfectly for me on 9 & and 10), I much prefer having my ZFS pool nice and clean, with the root filesystem (and any clones or copies) neatly tucked away in mypool/ROOT/bename. (Edit: looking through the bsdinstall source suggests that the new ZFS root install option uses the boot environments layout too. Can anyone confirm the dataset layout created if you use this option?)

Note that in 10 you no longer need the vfs.root.mountfrom option in /boot/loader.conf. Just setting the bootfs property on the pool is enough. This may be the case in 9.2 but i'm I'm not sure whether the change made it in.
 
usdmatt said:
I really recommend the following guide (See the section 'Server with two disks')
viewtopic.php?t=31662

Maybe it's just me but having used the above guide on all recent installs (works perfectly for me on 9 & 10), I much prefer having my ZFS pool nice and clean, with the root fs (and any clones or copies) neatly tucked away in mypool/ROOT/bename.

Note that in 10 you no longer need the vfs.root.mountfrom option in /boot/loader.conf. Just setting the bootfs property on the pool is enough. This may be the case in 9.2 but i'm not sure whether the change made it in.
Click to expand...

Yeah I've seen that but it's such a terrible guide with all the HTML code in it :\

I also think that's too advanced for me, FreeBSD and ZFS are totally new to me so I need to experiment with different configurations first anyway ;)
 
There is a lot in there and it's not helped by the fact that the forum software change has caused a lot of the formatting to be lost, most of which has ended in the formatting tags being visible.

The actual install guide though (the bit I've copied below) is one of the most concise and straight forward methods I've seen.

Code: 
1. Boot from the FreeBSD USB/DVD.
 2. Select the 'Live CD' option.
 3. login: root
 4. # sh
 5. # DISKS="ada0 ada1"
 6. # for I in ${DISKS}; do
    > NUMBER=$( echo ${I} | tr -c -d '0-9' )
    > gpart destroy -F ${I}
    > gpart create -s GPT ${I}
    > gpart add -t freebsd-boot -l bootcode${NUMBER} -s 128k ${I}
    > gpart add -t freebsd-zfs -l sys${NUMBER} ${I}
    > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ${I}
    > done
 7. # zpool create -f -o cachefile=/tmp/zpool.cache sys mirror /dev/gpt/sys*
 8. # zfs set mountpoint=none sys
 9. # zfs set checksum=fletcher4 sys
10. # zfs set atime=off sys
11. # zfs create sys/ROOT
12. # zfs create -o mountpoint=/mnt sys/ROOT/default
13. # zpool set bootfs=sys/ROOT/default sys
14. # cd /usr/freebsd-dist/
15. # for I in base.txz kernel.txz; do
    > tar --unlink -xvpJf ${I} -C /mnt
    > done
16. # cp /tmp/zpool.cache /mnt/boot/zfs/
17. # cat << EOF >> /mnt/boot/loader.conf
    > zfs_load=YES
    > vfs.root.mountfrom="zfs:sys/ROOT/default"
    > EOF
18. # cat << EOF >> /mnt/etc/rc.conf
    > zfs_enable=YES
    > EOF
19. # :> /mnt/etc/fstab
20. # zfs umount -a
21. # zfs set mountpoint=legacy sys/ROOT/default
22. # reboot

The only bit I find that looks a bit daunting if you don't know what's going on is when he uses a for loop to partition the disks. Personally, I just gpart both disks one after the other manually. Once you've done one disk, you only have to use the up arrow to scroll back through the command history, change the device number on the end and re-run the same commands. Same with the tar command - I just run the two tar commands to extract base/kernel.txz manually rather than mess about with a for loop.
 
usdmatt said:
There is a lot in there and it's not helped by the fact that the forum software change has caused a lot of the formatting to be lost, most of which has ended in the formatting tags being visible.

The actual install guide though (the bit I've copied below) is one of the most concise and straight forward methods I've seen?

Code: 
1. Boot from the FreeBSD USB/DVD.
 2. Select the 'Live CD' option.
 3. login: root
 4. # sh
 5. # DISKS="ada0 ada1"
 6. # for I in ${DISKS}; do
    > NUMBER=$( echo ${I} | tr -c -d '0-9' )
    > gpart destroy -F ${I}
    > gpart create -s GPT ${I}
    > gpart add -t freebsd-boot -l bootcode${NUMBER} -s 128k ${I}
    > gpart add -t freebsd-zfs -l sys${NUMBER} ${I}
    > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ${I}
    > done
 7. # zpool create -f -o cachefile=/tmp/zpool.cache sys mirror /dev/gpt/sys*
 8. # zfs set mountpoint=none sys
 9. # zfs set checksum=fletcher4 sys
10. # zfs set atime=off sys
11. # zfs create sys/ROOT
12. # zfs create -o mountpoint=/mnt sys/ROOT/default
13. # zpool set bootfs=sys/ROOT/default sys
14. # cd /usr/freebsd-dist/
15. # for I in base.txz kernel.txz; do
    > tar --unlink -xvpJf ${I} -C /mnt
    > done
16. # cp /tmp/zpool.cache /mnt/boot/zfs/
17. # cat << EOF >> /mnt/boot/loader.conf
    > zfs_load=YES
    > vfs.root.mountfrom="zfs:sys/ROOT/default"
    > EOF
18. # cat << EOF >> /mnt/etc/rc.conf
    > zfs_enable=YES
    > EOF
19. # :> /mnt/etc/fstab
20. # zfs umount -a
21. # zfs set mountpoint=legacy sys/ROOT/default
22. # reboot

The only bit I find that looks a bit daunting if you don't know what's going on is when he uses a for loop to partition the disks. Personally, I just gpart both disks one after the other manually. Once you've done one disk, you only have to use the up arrow to scroll back through the command history, change the device number on the end and re-run the same commands. Same with the tar command - I just run the two tar commands to extract base/kernel.txz manually rather than mess about with a for loop.
Click to expand...

Yeah the loop looks kind of scary to me ;) It could be useful though with a large amount of disks.
 
usdmatt said:
it's not helped by the fact that the forum software change has caused a lot of the formatting to be lost, most of which has ended in the formatting tags being visible.
Click to expand...
I've had a quick look at it, but there are some long posts in there and there's an awful lot of old formatting that needs to be changed/updated/fixed :( Unfortunately it's just not feasible to go and do that by hand, sorry. If I could have, I would have.
 
Ok, so what I did was put all the guides together and I came up with this which now works! So what this does is:
  • Create a mirror of 2 x SATA drives.
  • Create a log mirror on SSDs of 32G 32 GB.
  • Create L2ARC RAID 0 on SSDs of leftover space (448G 448 GB).

I would be happy to hear if I did something wrong, thanks!

Code: 
gpart destroy -F ada0
gpart destroy -F ada1
gpart destroy -F ada2
gpart destroy -F ada3

gpart create -s gpt ada2
gpart create -s gpt ada3

gpart add -s 222 -a 4k -t freebsd-boot -l boot0 ada2
gpart add -a 4k -t freebsd-zfs -l disk0 ada2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2

gpart add -s 222 -a 4k -t freebsd-boot -l boot1 ada3
gpart add -a 4k -t freebsd-zfs -l disk1 ada3
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada3

gnop create -S 4096 /dev/gpt/disk0
gnop create -S 4096 /dev/gpt/disk1

kldload zfs

zpool create -f -o altroot=/mnt -O canmount=off -m none zroot mirror /dev/gpt/disk0.nop /dev/gpt/disk1.nop

zfs set checksum=fletcher4                                           zroot
zfs set atime=off                                                    zroot
zfs create   -o mountpoint=none                                      zroot/ROOT
zfs create   -o mountpoint=/                                         zroot/ROOT/default
zfs create   -o mountpoint=/tmp -o compression=lz4  -o setuid=off   zroot/tmp
chmod 1777 /mnt/tmp
zfs create   -o mountpoint=/usr                                      zroot/usr
zfs create                                                           zroot/usr/local

zfs create   -o mountpoint=/home                     -o setuid=off   zroot/home

zfs create   -o compression=lz4                     -o setuid=off   zroot/usr/ports
zfs create   -o compression=off      -o exec=off     -o setuid=off   zroot/usr/ports/distfiles
zfs create   -o compression=off      -o exec=off     -o setuid=off   zroot/usr/ports/packages
zfs create   -o compression=lz4     -o exec=off     -o setuid=off   zroot/usr/src
zfs create                                                           zroot/usr/obj
zfs create   -o mountpoint=/var                                      zroot/var
zfs create   -o compression=lz4     -o exec=off     -o setuid=off   zroot/var/crash
zfs create                           -o exec=off     -o setuid=off   zroot/var/db
zfs create   -o compression=lz4     -o exec=on      -o setuid=off   zroot/var/db/pkg
zfs create                           -o exec=off     -o setuid=off   zroot/var/empty
zfs create   -o compression=lz4     -o exec=off     -o setuid=off   zroot/var/log
zfs create   -o compression=gzip     -o exec=off     -o setuid=off   zroot/var/mail
zfs create                           -o exec=off     -o setuid=off   zroot/var/run
zfs create   -o compression=lz4     -o exec=on      -o setuid=off   zroot/var/tmp
chmod 1777 /mnt/var/tmp

gpart create -s gpt ada0
gpart create -s gpt ada1
gpart add -t freebsd-zfs -b 2048 -a 4k -l log0 -s 32G ada0
gpart add -t freebsd-zfs -b 2048 -a 4k -l log1 -s 32G ada1
gpart add -t freebsd-zfs -a 4k -l cache0 ada0
gpart add -t freebsd-zfs -a 4k -l cache1 ada1

zpool add -f zroot log mirror gpt/log0 gpt/log1
zpool add -f zroot cache gpt/cache0 gpt/cache1

zpool set bootfs=zroot/ROOT/default zroot

tar --unlink -xvpJf /usr/freebsd-dist/kernel.txz -C /mnt
tar --unlink -xvpJf /usr/freebsd-dist/base.txz -C /mnt

mount -t devfs devfs /dev
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
zfs set readonly=on zroot/var/empty

Edit: I updated it a bit, removed the swap because I have 128 GB RAM (so no need) and forgot loader.conf so the system panicked after boot xD
 
You must log in or register to reply here.
Back
Top