I solved my problem.
The issue was this:
At install time, I chose zfs-on-root, using a simple 2-way mirror.
bsdinstall partitioned 2 disks, there are 4 in this box, in the usual way. I later added the other two disks, ada2 & ada3, as an additional, mirror vdev, to the same root pool. I added them as RAW disks without partitioning. ZFS is fine with that. So my root pool layout was like this:
Code:
zpool status
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
ada1p3 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
So, just after the install, the physical data distribution (zfs internals) was such that 99% of the base system, including kernel and /boot etc, was on the first vdev: The one with proper partitions, which was created by bsdinstall.
As data got written, ZFS striped it across the 2 vdevs, and favoured the second vdev (ada2/ada3), to even things up. This worked perfectly. I updated the system and rebooted maybe twice over 4 months.
Then, on Thursday, Jan 14 2016, I did another
freesbsd-update fetch install && reboot: BOOM! (see above for detailed errors)
The reason was that, PURELY BY CHANCE, zfs decided to put some of my new /boot and kernel files on the second vdev (ada2/ada3). As reported above, the BTX loader is incapable of properly recognising non-partitioned ZFS disks. It misreported them as having 2 partitions, when they actually had none.
lsdev from the loader prompt was the proof. As a result, the meta-data (all/mostly on 1st vdev), was fine. In other words, I could browse the file/dir tree with
ls from loader prompt. However, attempting to read contents of those files resulted in: "ZFS: i/o error - all block copies unavailable", because the loader could not properly use the second vdev block device.
So, with hindsight, each time I updated the system, I was playing Russian roulette. Each time, I had a 50/50, or worse, chance of zfs deciding to put the new data of /boot onto the 2nd vdev.
The solution was as follows:
- Install a spare drive.
- Create a zpool called zbackup on it, and
zfs send the entire zroot pool to it. Luckily, it was only 10% full. I roughly followed this post.
zpool destroy zroot. Use gpart to create single partitions on ada2 & ada3 of type=freebsd-zfs, taking up almost the whole disk.- Recreate the zroot pool, including both mirrors, using all 4 drives from the start. I could have also added the 2 additional drives later, as long as they had proper partitioning.
zfs receive the backup.
For reference, these are the commands that I used. If useful,
you will need to adapt these:
Code:
# credit to: https://www.dan.me.uk/blog/2012/08/05/full-system-backups-for-freebsd-systems-using-zfs/
# and I obtained a copy of the bsdinstall_log to get the latest, best practice
# to make the backup
gpart destroy -F ada4
dd if=/dev/zero of=/dev/ada4 bs=1m count=128
zpool create zbackup /dev/ada4
zfs set mountpoint=/var/full-backup zbackup
# DONT PIPE THROUGH GZIP, IT WILL LIKELY BE A CPU BOTTLENECK!
zfs snapshot -r zroot@backup
zfs send -Rv zroot@backup > /var/full-backup/full-system-backup.zfs
zfs destroy -r zroot@backup
# ensure our new partitions use 4K sectors
sysctl vfs.zfs.min_auto_ashift=12
# kill the old pool
zpool destroy zroot
# clear the labels of vdev1 drives (not sure if really required)
zpool labelclear -f /dev/ada0p3
zpool labelclear -f /dev/ada1p3
# properly partition the vdev2 drives (one big partition)
gpart destroy -F ada2
gpart destroy -F ada3
zpool labelclear -f /dev/ada2
zpool labelclear -f /dev/ada3
gpart create -s gpt ada2
gpart create -s gpt ada3
gpart add -a 1m -l zfs2 -t freebsd-zfs ada2
gpart add -a 1m -l zfs3 -t freebsd-zfs ada3
# check structure
gpart show
=> 34 7814037101 ada0 GPT (3.6T)
34 6 - free - (3.0K)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 4194304 2 freebsd-swap (2.0G)
4196352 7809839104 3 freebsd-zfs (3.6T)
7814035456 1679 - free - (840K)
=> 34 7814037101 ada1 GPT (3.6T)
34 6 - free - (3.0K)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 4194304 2 freebsd-swap (2.0G)
4196352 7809839104 3 freebsd-zfs (3.6T)
7814035456 1679 - free - (840K)
=> 34 7814037101 ada2 GPT (3.6T)
34 2014 - free - (1.0M)
2048 7814033408 1 freebsd-zfs (3.6T)
7814035456 1679 - free - (840K)
=> 34 7814037101 ada3 GPT (3.6T)
34 2014 - free - (1.0M)
2048 7814033408 1 freebsd-zfs (3.6T)
7814035456 1679 - free - (840K)
zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f zroot mirror ada0p3 ada1p3 mirror ada2p1 ada3p1
# to restore the backup
zfs receive -vdF zroot < /var/full-backup/full-system-backup.zfs
zpool set bootfs=zroot/ROOT/default zroot
# ensure we have a valid zpool.cache
zpool export zroot
zpool import -o altroot=/mnt zroot
mkdir -p /mnt/boot/zfs
zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot
reboot
The new pool structure is now:
Code:
[CMD]zpool status[/CMD]
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
ada1p3 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ada2p1 ONLINE 0 0 0
ada3p1 ONLINE 0 0 0
(note that ada2 and ada3 both have a partition now)
Using
lsdev at the loader prompt, now reports the true pool and partition structure.
And importantly, the system BOOTS FINE, because loader can now understand, that ada2p1 and ada3p1 are part of the pool.
Hope this helps someone. Thanks to
kpa, for steering me in the right direction.
I do wonder, whether this
cp -Rp boot.orig boot technique, mentioned above, is a way of "playing Russian roulette, in reverse". Pull the trigger until you live: ie, keep cp'ing, until your critical boot data is on a vdev/disk, which is healthy, and available at boot time. Perhaps, not a terribly reassuring fix?
Anyway,
the moral of my story is: If you want to grow your root-on-zfs pool, adding a further vdev is a good, and perfectly legitimate way. However, you must make sure,
that the disks, which make up the new vdev, are properly partitioned, with the zfs slice being of type=freebsd-zfs. If not, early loader code will not recognise them, and the whole system will, eventually, fail to boot.
Of course, there is zfs documentation from Sun/Oracle, out there already, that
tells you this:
Using Slices in a ZFS Storage Pool
Disks can be labeled with a traditional Solaris VTOC (SMI) label when you create a storage pool with a disk slice.
For a bootable ZFS root pool, the disks in the pool must contain slices and the disks must be labeled with an SMI label. The simplest configuration would be to put the entire disk capacity in slice 0 and use that slice for the root pool.
It would be nice, if the FreeBSD handbook, were more detailed on the options and pitfalls.
