ZFS boot issues

reuteler · Feb 4, 2015

I've been having some rather odd and very annoying issues w/ ZFS boot on 10.0 and now 10.1. This is a Dell R220 (no RAID card) and it's a fairly simple setup: a mirrored Seagate Constellation/Hitachi Ultrastar (both 3 TB) pair (though it has happened with different drives on the same box, as well, so it doesn't appear to be the drives).

What's happening is that intermittently on reboot it won't come back -- there's nothing obvious going on (/ looks OK, IOW), either. Sometimes I'll bounce the box and the (immediate) second reboot goes into lala land. Sometimes it'll hang after the ZFS boot loader (won't get to the boot menu), other times it'll show a \ and that's it.

Writing out the bootcode:

Code:

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

will usually get me to the point where it'll hang after the ZFS boot loader; but again, no further.

The only way to get it to boot is to boot from USB and "zfs send" the existing zroot/ROOT/default, "zfs destroy" and "zfs recv" it. Fortunately that always works.

Code:

# zfs snapshot zroot/ROOT/default@now
# zfs send zroot/ROOT/default@now > root.send
# zfs destroy -r zroot/ROOT/default
# cat root.send | zfs recv zroot/ROOT/default

But that's obviously suboptimal. There's no compression on / and nothing really out of the ordinary there. I am using compression and copies > 1 elsewhere, though.

This is driving me nuts. Any ideas?

Here's the info.

Code:

[root@panic 0 ~]# zfs get all zroot/ROOT/default
NAME                PROPERTY                        VALUE                           SOURCE
zroot/ROOT/default  type                            filesystem                      -
zroot/ROOT/default  creation                        Wed Feb  4 11:49 2015           -
zroot/ROOT/default  used                            3.94G                           -
zroot/ROOT/default  available                       2.37T                           -
zroot/ROOT/default  referenced                      3.93G                           -
zroot/ROOT/default  compressratio                   1.00x                           -
zroot/ROOT/default  mounted                         yes                             -
zroot/ROOT/default  quota                           none                            default
zroot/ROOT/default  reservation                     none                            default
zroot/ROOT/default  recordsize                      128K                            default
zroot/ROOT/default  mountpoint                      /                               local
zroot/ROOT/default  sharenfs                        off                             default
zroot/ROOT/default  checksum                        on                              default
zroot/ROOT/default  compression                     off                             inherited from zroot/ROOT
zroot/ROOT/default  atime                           off                             inherited from zroot
zroot/ROOT/default  devices                         on                              default
zroot/ROOT/default  exec                            on                              default
zroot/ROOT/default  setuid                          on                              default
zroot/ROOT/default  readonly                        off                             default
zroot/ROOT/default  jailed                          off                             default
zroot/ROOT/default  snapdir                         hidden                          default
zroot/ROOT/default  aclmode                         discard                         default
zroot/ROOT/default  aclinherit                      restricted                      default
zroot/ROOT/default  canmount                        on                              default
zroot/ROOT/default  xattr                           off                             temporary
zroot/ROOT/default  copies                          1                               default
zroot/ROOT/default  version                         5                               -
zroot/ROOT/default  utf8only                        off                             -
zroot/ROOT/default  normalization                   none                            -
zroot/ROOT/default  casesensitivity                 sensitive                       -
zroot/ROOT/default  vscan                           off                             default
zroot/ROOT/default  nbmand                          off                             default
zroot/ROOT/default  sharesmb                        off                             default
zroot/ROOT/default  refquota                        none                            default
zroot/ROOT/default  refreservation                  none                            default
zroot/ROOT/default  primarycache                    all                             default
zroot/ROOT/default  secondarycache                  all                             default
zroot/ROOT/default  usedbysnapshots                 8.21M                           -
zroot/ROOT/default  usedbydataset                   3.93G                           -
zroot/ROOT/default  usedbychildren                  0                               -
zroot/ROOT/default  usedbyrefreservation            0                               -
zroot/ROOT/default  logbias                         latency                         default
zroot/ROOT/default  dedup                           off                             default
zroot/ROOT/default  mlslabel                                                        -
zroot/ROOT/default  sync                            standard                        default
zroot/ROOT/default  refcompressratio                1.00x                           -
zroot/ROOT/default  written                         900K                            -
zroot/ROOT/default  logicalused                     3.58G                           -
zroot/ROOT/default  logicalreferenced               3.57G                           -
zroot/ROOT/default  volmode                         default                         default
zroot/ROOT/default  filesystem_limit                none                            default
zroot/ROOT/default  snapshot_limit                  none                            default
zroot/ROOT/default  filesystem_count                none                            default
zroot/ROOT/default  snapshot_count                  none                            default
zroot/ROOT/default  redundant_metadata              all                             default
zroot/ROOT/default  com.sun:auto-snapshot           true                            local
zroot/ROOT/default  com.sun:auto-snapshot:frequent  false                           inherited from zroot

Code:

[root@panic 0 ~]# zpool get all zroot
NAME   PROPERTY                       VALUE                          SOURCE
zroot  size                           2.66T                          -
zroot  capacity                       3%                             -
zroot  altroot                        -                              default
zroot  health                         ONLINE                         -
zroot  guid                           2274368527045107482            default
zroot  version                        -                              default
zroot  bootfs                         zroot/ROOT/default             local
zroot  delegation                     on                             default
zroot  autoreplace                    off                            default
zroot  cachefile                      -                              default
zroot  failmode                       wait                           default
zroot  listsnapshots                  off                            default
zroot  autoexpand                     off                            default
zroot  dedupditto                     0                              default
zroot  dedupratio                     1.00x                          -
zroot  free                           2.57T                          -
zroot  allocated                      86.4G                          -
zroot  readonly                       off                            -
zroot  comment                        -                              default
zroot  expandsize                     0                              -
zroot  freeing                        0                              default
zroot  fragmentation                  7%                             -
zroot  leaked                         0                              default
zroot  feature@async_destroy          enabled                        local
zroot  feature@empty_bpobj            active                         local
zroot  feature@lz4_compress           active                         local
zroot  feature@multi_vdev_crash_dump  enabled                        local
zroot  feature@spacemap_histogram     active                         local
zroot  feature@enabled_txg            active                         local
zroot  feature@hole_birth             active                         local
zroot  feature@extensible_dataset     enabled                        local
zroot  feature@embedded_data          active                         local
zroot  feature@bookmarks              enabled                        local
zroot  feature@filesystem_limits      enabled                        local

Code:

[reuteler@panic 0 ~]$ zpool status
  pool: zroot
state: ONLINE
  scan: resilvered 88.7G in 0h42m with 0 errors on Wed Feb  4 11:19:11 2015
config:

    NAME                                            STATE     READ WRITE CKSUM
    zroot                                           ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        gptid/e5725b10-06d9-11e4-94a6-a0369f3f8b38  ONLINE       0     0     0
        gptid/b65a6af6-ac93-11e4-8842-a0369f3f8b38  ONLINE       0     0     0

errors: No known data errors

Code:

[root@panic 0 ~]# gpart show ada0 ada1
=>        34  5860533101  ada0  GPT  (2.7T)
          34           6        - free -  (3.0K)
          40        1024     1  freebsd-boot  (512K)
        1064   134217728     2  freebsd-swap  (64G)
  134218792  5726314336     3  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>        34  5860533101  ada1  GPT  (2.7T)
          34           6        - free -  (3.0K)
          40        1024     1  freebsd-boot  (512K)
        1064   134217728     2  freebsd-swap  (64G)
  134218792  5726314336     3  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

reuteler · Feb 10, 2015

Well, FWIW, I played around quite a lot with this (including reinstalling the OS) and was never able to get this working with gptzfsboot (consistently anyhow -- 1 of 15 or so boots would hang/fail). I eventually gave up and created a UFS boot partition that loaded root from the zpool and that's been working no problem. That also allowed me to switch between gptzfsboot and gptboot and it's pretty consistent in results. So, I suppose old school ways have a tendency towards working.

junovitch@ · Feb 13, 2015

Pretty strange issue. I wouldn't know what to suggest next as far as troubleshooting. As far as researching it, a look through bug reports at https://bugs.freebsd.org/bugzilla/ or searching to mailing like may be worth it. Maybe somebody out there has seen the same thing.

free-and-bsd · Feb 24, 2015

Well one thing could be tried that comes to mind. You can try to destroy your existing zpool and recreate it using 9.0-RELEASE installation CD and its earlier version of ZFS tools (not a 9.x-RELEASE, which will have the latest ones), where you can then receive your snapshot and do the same to the second HDD.

That would create a zpool of an older version, without the new features introduced later on. This can be upgraded later on if needed, but before that you might see the difference. In which case you might be the first one to file a bug report.

Another thing you may try (as the cause of the problem is still unknown, it could be anything) is move your swap space inside your ZFS pool. There used to be problems booting when swap partition preceded the root partition... long time ago, with Solaris involved or something.

ZFS boot issues

reuteler

reuteler

junovitch@

free-and-bsd