ZFS on raw disk + GRUB2 boot (multiple OS) = working

[free-and-bsd]

As was reported earlier, GRUB2 doesn't support all recent feature flags in zpool. But there's no need to use earlier ZFS tools and create obsolete versions of zpools. We only need to create zpool with features disabled, as in zpool create -d $poolname $device. Where $device is raw unpartitioned device, as in /dev/ada0 or whatever. No boot partitions, no MBR, no GPT labels -- nothing at all.

Alternatively, async_destroy, empty_bpobj and lz4_compress feature flags are recognized by recent git GRUB2 versions and can be enabled at the time of the creation as in zpool create -d -o feature@myfeature ... $poolname $device . LZ4 compression is particularly useful...

Once the pool created and filesystems imported/created/populated/mounted, GRUB2 can be installed. For example, suppose rpool/ROOT/freebsd is mounted at /rpool/ROOT/freebsd -- as it will be at the time of the creation of the datasets, unless other mountpoints are set. Then grub installation will be as follows:

grub-install --modules=zfs --boot-directory=/rpool/ROOT/freebsd/boot /dev/ada0

At this stage it must return "no errors reported" message.

The advantage of this approach is, that if one wants to have Linux installed, it can be installed in a separate dataset, say rpool/ROOT/linux. But if zpool was created on a GPT freebsd-zfs partition, it will be linked to /dev/gptid/XXXX and linux boot scripts will have problems importing such pool. zpool import -d /dev/by-partuuid will have to be added manually to initrd creating scripts, which is additional pain.
With whole disk root pool this problem doesn't exist and everything works smooth. Only I cannot say how it works with EFI.

 

[free-and-bsd]

On the second thought... Is it safe to do this grub-install operation? From man zfsboot it is clear that /boot/zfsboot is installed into a whole-disk rpool in a special way. On the other way, as grub-install doesn't detect GPT on the disk, it most likely treats it like MBR.

Any thoughts, anybody?

 

[free-and-bsd]

UPDATES: the last git version of GRUB2 supports ALL ZFS feature flags introduced in FreeBSD, as of 10.1-RELEASE at least (I'm running STABLE). There is a minor bug there, however, so that grub doesn't see /boot/kernel/zfs.ko module, unless it is copied separately to a custom directory (i.e. /boot/custom/zfs.ko etc.). I'm checking that now.

One other thing in this regard. Before creating a full-disk zpool, make sure the partition table is clean, or else grub installation will fail. That is, fdisk /dev/adaN must give a result like this:

....
Information from DOS bootblock is:
The data for partition 1 is:
<UNUSED>
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

 

[free-and-bsd]

OK, anyone willing to use GRUB2 in its latest version for feature-enabled ZFS pool can follow this (temporary) solution. It's been established now, which changes introduced this bug into GRUB2 code. So, until the developers find an alternative solution, this small patch will revert those changes in grub-core/fs/zfs/zfs.c file:

--- a/grub-core/fs/zfs/zfs.c   2015-05-13 15:15:02.727889975 +0200
+++ b/grub-core/fs/zfs/zfs.c   2015-05-13 15:37:09.401798411 +0200
@@ -2071,14 +2071,12 @@
  */
static grub_err_t
mzap_lookup (mzap_phys_t * zapobj, grub_zfs_endian_t endian,
-    grub_uint16_t objsize, const char *name, grub_uint64_t * value,
+    int objsize, const char *name, grub_uint64_t * value,
    int case_insensitive)
{
-  grub_uint16_t i, chunks;
+  int i, chunks;
  mzap_ent_phys_t *mzap_ent = zapobj->mz_chunk;
-  if (objsize < MZAP_ENT_LEN)
-  return grub_error (GRUB_ERR_FILE_NOT_FOUND, N_("file `%s' not found"), name);
  chunks = objsize / MZAP_ENT_LEN - 1;
  for (i = 0; i < chunks; i++)
  {
@@ -2486,7 +2484,7 @@
    struct grub_zfs_data *data, int case_insensitive)
{
  grub_uint64_t block_type;
-  grub_uint16_t size;
+  int size;
  void *zapbuf;
  grub_err_t err;
  grub_zfs_endian_t endian;

Keep in mind, please, that this is simply the reverse of commit 87a04adb6541c4bf169644bb2f84036a7c8558dd , which introduced this bug. See details here. With this patch applied you have the last version all-zfs-feature-enabled GRUB2 git (working) code. That is, it works with regard to booting FreeBSD (and Linux) from a ZFS pool. Still, it's development code with all that follows.

 

[free-and-bsd]

UPDATE: this has been fixed in 5370dcfdae66b60cff3507ad925300a679fe4117 . So right now it must be working all right as it is, no patch needed.
As an example of a working configuration, here are the features enabled in this pool:

rpool  feature@async_destroy           enabled
rpool  feature@empty_bpobj             active
rpool  feature@lz4_compress            active
rpool  feature@multi_vdev_crash_dump   enabled
rpool  feature@spacemap_histogram      active
rpool  feature@enabled_txg             active
rpool  feature@hole_birth              active
rpool  feature@extensible_dataset      enabled
rpool  feature@embedded_data           active
rpool  feature@bookmarks               enabled
rpool  feature@filesystem_limits       enabled