Solved Adding a New Hard Disk into pool

[Oliver Schonrock]

You don't want to mirror the bootcode, it's meant to be written on each disk separately. Swap on the other hand can be mirrored with gmirror(8).

Sorry, what I wrote was imprecise. I meant:

• By having GPT "bootcode" on ada0 AND ada1 (this is "duplicated" not mirrored) the machine can boot from either disk during a failure at bootime. What happens when there is an update in the bootcode? Is that part of freebsd-update? can it update the partitions on both disks?
  
• The swap is mirrored (I selected that as an option during bsdinstall), so we get data redundancy on swap… nice to have, might prevent a crash when heavily swapping sometime in the far future and then one of the 2 disks with the swap partition dies. Not sure how this actually works, looks like gmirror:

[oliver@zfsroot ~]$ gmirror list
Geom name: swap
State: COMPLETE
Components: 2
Balance: load
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 1798221310
Providers:
1. Name: mirror/swap
  Mediasize: 2147483136 (2.0G)
  Sectorsize: 512
  Mode: r1w1e0

• What I did not decide to do, was to create ADDITIONAL copies of bootcode (duplicate) and swap (mirrored) and the 3rd and 4th disks (ada2 and ada3). This manual page seems to suggest that you might want to that. I didn't because I figured 2 copies was enough, and AKAIK zfs doesn't care that the second vdev (ada2, ada3) is slightly bigger than the ada0p3 & ada1p3 vdev. In fact that's kind of the point of zfs right? Well one of them.

Does that make sense?

 

[Oliver Schonrock]

Edit: One thing I haven't mentioned in here is 4k alignment. You'll probably want to run the following before creating the zpool to , and possibly look at getting the partitions aligned correctly on the SSD.

sysctl vfs.zfs.min_auto_ashift=12

Yeah good point. bsdinstall(8) dealt with that for me (by default) as well. Will bear in mind when I do manually.

 

[Oliver Schonrock]

Yeah good point. bsdinstall dealt with that for me (by default) as well. Will bear in mind when I do manually.

Actually I am wrong, at least partially. bsdinstall(8) forced the 4K sectors (same as ashift right?), at least for the first 2 disks. But looking at this post and testing my system the 3rd and 4th disks have ashift of 9:

root@zfsroot:~ # zdb -l /dev/ada0p3 | grep ashift
  ashift: 12
  ashift: 12
  ashift: 12
  ashift: 12
root@zfsroot:~ # zdb -l /dev/ada1p3 | grep ashift
  ashift: 12
  ashift: 12
  ashift: 12
  ashift: 12
root@zfsroot:~ # zdb -l /dev/ada2 | grep ashift
  ashift: 9
  ashift: 9
  ashift: 9
  ashift: 9
root@zfsroot:~ # zdb -l /dev/ada3 | grep ashift
  ashift: 9
  ashift: 9
  ashift: 9
  ashift: 9

Seems bsdinstall(8) only set the sysctl for ashift=12 during the creation of its zroot vdev. After the reboot, when I made the second mirror the sysctl was at:

root@zfsroot:~ # sysctl vfs.zfs.min_auto_ashift
vfs.zfs.min_auto_ashift: 9

So I will make sure I do that when I build on real disks.

Would it not be sensible for bsdinstall to put the ashift=12 sysctl into /boot/loader.conf (or whereever that goes) if the sysadmin selects "force 4K sectors" in the bsdinstall menu? That way you don't get caught like this? (it's only a vm, so no matter).

 

[kpa]

• By having GPT "bootcode" on ada0 AND ada1 (this is "duplicated" not mirrored) the machine can boot from either disk during a failure at bootime. What happens when there is an update in the bootcode? Is that part of freebsd-update? can it update the partitions on both disks?

The bootcode on the MBR and the first partition is not updated automatically by freebsd-update(8) or make installworld so you'll have to run gpart bootcode yourself. The files under /boot are updated automatically when updates happen though. Unfortunately the only time you're actually notified about the need to update the bootcode is when you run zpool upgrade.