Howto install root on ZFS on a remote server without physical access?

[uisge]

Hi --

I intend to install root on ZFS mirror using GPT on a remote server without physical access. But, it is possible to reboot this server into a FreeBSD 8.1 rescue system that comes with all necessary tools, but without any installable executables.

Please excuse my ignorance, but I couldn't find a solution searching this forum and the net. All found advices come with manual installation of a base system.

Would the following be a feasible strategy for my plan?
- Using one disk for the installation of a basic system.
- Rebooting and loop mounting a downloaded ISO installation file.
- Preparing the other disk according root on ZFS mirror using GPT and installing root on ZFS and the system there.
- Continue with the first disk after rebooting
- Finalise mirroring

Are there other/better strategies, or should I be conservative and create two slices, one for root on UFS the other for data on ZFS?

Thanks in advance.

 

[danbi]

For the purposes of installing zfs-only systems I have built myself a 'bootstrap USB stick'. 4GB in size is sufficient for all intents and purposes. The reason I do it this way and not via the 'official' boot media is that ZFS had some updates recently and it would be better to install the already more current version (ZFS v15) on the new system.

I would prepare the USB stick as single partition, install current system there as outlined in the section 'To cross-install current onto a separate partition' in /usr/src/UPDATING. I do keep a copy of /usr/src on the USB stick as well, to allow of in-place recompilation of the system - so that it can stay current with 8-stable without the need of another already operational FreeBSD computer. The /boot/loader.conf of the boot media has zfs_load="YES" and geom_mirror_load="YES". The later is used for the swap.

If you will do this only once, you may just install FreeBSD in single partition on one of the drives, using any available installation procedure.

First, boot from the USB stick or boot drive.
Then use something like this script:

#!/bin/sh

disk=ada0
swapsize=32g

dd if=/dev/zero of=/dev/$disk bs=1m count=10
gpart create -s GPT $disk 
gpart add -b 34 -s 128 -t freebsd-boot $disk 
gpart add -s $swapsize -t freebsd-swap -l swap0 $disk 
gpart add -t freebsd-zfs -l system0 $disk 
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 $disk

gmirror label swap gpt/swap0
swapon /dev/mirror/swap

zpool create -O mountpoint=/mnt -O atime=off -O canmount=off system /dev/gpt/system0

zfs create -o mountpoint=legacy -o setuid=on system/root 
zpool set bootfs=system/root system 
mount -t zfs system/root /mnt


zfs create -o compress=lzjb system/tmp 
chmod 1777 /mnt/tmp 
zfs create -o canmount=off system/usr 
zfs create -o compress=off system/usr/local
zfs create -o compress=gzip system/usr/src 
zfs create -o compress=lzjb system/usr/obj 
zfs create -o compress=gzip system/usr/ports 
zfs create -o compress=off system/usr/ports/distfiles 
zfs create -o canmount=off system/var 
zfs create -o compress=gzip system/var/log 
zfs create -o compress=lzjb system/var/audit 
zfs create -o compress=lzjb system/var/tmp 
chmod 1777 /mnt/var/tmp 
zfs create -o canmount=off system/usr/home 

cd /
tar -c --one-file-system -f - . | tar xpf - -C /mnt/

echo system/root / zfs rw,noatime 0 0  > /mnt/etc/fstab
echo /dev/mirror/swap none swap sw 0 0 >> /mnt/etc/fstab
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo WRKDIRPREFIX=/usr/obj >> /mnt/etc/make.conf
zfs umount -a
umount /mnt 
zfs set mountpoint=/ system

zpool export system
zpool import system
mount -t zfs system/root /mnt
cp /boot/zfs/zpool.cache /mnt/boot/zfs/
umount /mnt

You need to modify the disk= and swapsize= variables in the script to suit your needs.
You may ignore and not set the various compression options on the filesystems and/or use different filesystem layout.

When you boot from this zfs only disk, you may prepare and add the second disk with this script

#!/bin/sh

disk=ada1
swapsize=32g

dd if=/dev/zero of=/dev/$disk bs=1m count=10
gpart create -s GPT $disk 
gpart add -b 34 -s 128 -t freebsd-boot $disk 
gpart add -s $swapsize -t freebsd-swap -l swap1 $disk 
gpart add -t freebsd-zfs -l system1 $disk 
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 $disk

gmirror insert swap gpt/swap1

zpool attach system gpt/system0 gpt/system1

Wait for the gmirror and zfs resilver to complete, then reboot to verify everything is ok.

I have done this remotely few times, but it is possible things may go bad -- if you do not have remote console/BIOS access it's usually risky idea.

Note: Perhaps you it is better idea to have the swap/zfs partitions aligned, especially if you have some of the new 4k sector drives.

 

[uisge]

I have done this remotely few times, but it is possible things may go bad

Thanks a lot for our reply, and this makes me confident that I will succeed in one way or the other ;-)
I will give your approach a try as soon as I will get access to my new server.

if you do not have remote console/BIOS access it's usually risky idea.

Yeah, but as this server isn't a production server yet, I will have some time to "play" with it. If I will seriously misconfigure or screw it, there's an easy way to reset that server to a fresh 8.1 installation.

Thanks again.

 

[nsayer]

I just set up a rootbsd.net VPS with GPT+ZFS a couple weeks ago. I did it with an amd64 8.1-RELEASE livefs iso mounted (they give you a VNC virtual KVM). I used the fixit shell to do a manual partition and install. It worked ok.

My big problem right now is that if I start to do too much on the machine it will lock up with "swap_pager: indefinite wait" messages. But the actual installation procedure was no problem. I followed, more or less, the procedure outlined here.

 

[uisge]

cp /boot/zfs/zpool.cache /mnt/boot/zfs/

Well, that actually was the trickiest part. I did use a rescue system coming with relevant tools, but /boot/zfs comes mounted read-only. Thus, I did have to use mount_nullfs to create a writable /boot/zfs.

Besides that I was able to follow your instructions, and I succeeded with:

uisge> zpool status
  pool: zp0
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
	still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
	pool will no longer be accessible on older software versions.
 scrub: none requested
config:

	NAME           STATE     READ WRITE CKSUM
	zp0            ONLINE       0     0     0
	  mirror       ONLINE       0     0     0
	    gpt/disk4  ONLINE       0     0     0
	    gpt/disk6  ONLINE       0     0     0

errors: No known data errors

Thank you! That was a very nice lesson for me. Now, I do have to learn more about ZFS' functionalities.

One last question to all:

In quite some HOWTOs (e.g.) one finds:

echo 'zfs_enable="YES"' > /etc/rc.conf

Is that really needed, and if so, why?

 

[DutchDaemon]

echo 'zfs_enable="YES"' > /etc/rc.conf

Is that really needed, and if so, why?

I do hope it's

echo 'zfs_enable="YES"' >> /etc/rc.conf

or we're in a whole new mess pretty quickly ..

 

[nsayer]

Turning the 'zfs_enable' knob on will do a couple things: If you have any zvol's with 'org.freebsd:swap' set to 'on,' then they will be added as swap space. It will also do 'zfs mount -a' and 'zfs share -a,' in case you have filesystems that need either of those steps.

 

[uisge]

I do hope it's

echo 'zfs_enable="YES"' >> /etc/rc.conf

or we're in a whole new mess pretty quickly ..

Yes, it should be rathe a '>>' , and not a '>' like in

echo 'zfs_load="YES"' > /boot/loader.conf

as well. But the Wiki uses '>'.

 

[uisge]

Turning the 'zfs_enable' knob on will do a couple things: If you have any zvol's with 'org.freebsd:swap' set to 'on,' then they will be added as swap space. It will also do 'zfs mount -a' and 'zfs share -a,' in case you have filesystems that need either of those steps.

Understood. After reading /etc/rc.d/zfs (what I should have done before asking) I know that I'll need it.

Thanks.