ZFS gptzfsboot: failed to mount default pool

One of my servers had a reboot few days ago. But after reboot it did not boot.

Last version reported was FreeBSD 10-RELEASE-p1.

zfs.png


Code:
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool tank
gptzfsboot: failed to mount default pool

FreeBSD/x86 boot
.......//asm code here
BTX halted

I looks freebsd-update fetch install was used in a past year weekly but no restarts reported. Also I have found some merger files from FreeBSD 9.0/9.1 in /tmp/ folder so possibly upgraded

I can access server using rescue os and then I can easily mount ZFS partition:

zpool import -o altroot=/mnt -o cachefile=/boot/zfs/zpool.cache tank

Disks are healthy and online. All data is there.


zpool status -x
Code:
all pools are healthy
zpool status
Code:
pool: tank
state: ONLINE
  scan: resilvered 211G in 80406h40m with 0 errors on Wed Mar  5 16:26:48 2014
config:

  NAME  STATE  READ WRITE CKSUM
  tank  ONLINE  0  0  0
  mirror-0  ONLINE  0  0  0
  gptid/40b968be-0983-11e1-872d-14dae9978df7  ONLINE  0  0  0
  gptid/419054da-0983-11e1-872d-14dae9978df7  ONLINE  0  0  0

errors: No known data errors

I tried doing something like this:

gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptzfsboot -i 1 ada0
gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptzfsboot -i 1 ada1
zpool set bootfs=tank/root tank
zpool reguid tank


but this does not help at all. I have copied /boot/ from exactly the same build server. No luck.

I tried checking versions:
zpool get version
Code:
NAME  PROPERTY  VALUE    SOURCE
tank  version   -        default

zdb tank
Code:
Cached configuration:
  version: 5000
  name: 'tank'
  state: 0
  txg: 15557280
  pool_guid: 522225119579774437
  hostid: 1105657835
  hostname: 'rescue'
  vdev_children: 1
  vdev_tree:
  type: 'root'
  id: 0
  guid: 522225119579774437
  children[0]:
  type: 'mirror'
  id: 0
  guid: 14723725333911636736
  metaslab_array: 23
  metaslab_shift: 34
  ashift: 9
  asize: 2983407124480
  is_log: 0
  children[0]:
  type: 'disk'
  id: 0
  guid: 3984000272163324164
  path: '/dev/gptid/40b968be-0983-11e1-872d-14dae9978df7'
  phys_path: '/dev/gptid/40b968be-0983-11e1-872d-14dae9978df7'
  whole_disk: 0
  DTL: 127
  children[1]:
  type: 'disk'
  id: 1
  guid: 13509242023844851023
  path: '/dev/gptid/419054da-0983-11e1-872d-14dae9978df7'
  phys_path: '/dev/gptid/419054da-0983-11e1-872d-14dae9978df7'
  whole_disk: 0
  DTL: 131
  features_for_read:

MOS Configuration:
  version: 5000
  name: 'tank'
  state: 0
  txg: 15557280
  pool_guid: 522225119579774437
  hostid: 1105657835
  hostname: 'rescue'
  vdev_children: 1
  vdev_tree:
  type: 'root'
  id: 0
  guid: 522225119579774437
  children[0]:
  type: 'mirror'
  id: 0
  guid: 14723725333911636736
  metaslab_array: 23
  metaslab_shift: 34
  ashift: 9
  asize: 2983407124480
  is_log: 0
  children[0]:
  type: 'disk'
  id: 0
  guid: 3984000272163324164
  path: '/dev/gptid/40b968be-0983-11e1-872d-14dae9978df7'
  phys_path: '/dev/gptid/40b968be-0983-11e1-872d-14dae9978df7'
  whole_disk: 0
  DTL: 127
  children[1]:
  type: 'disk'
  id: 1
  guid: 13509242023844851023
  path: '/dev/gptid/419054da-0983-11e1-872d-14dae9978df7'
  phys_path: '/dev/gptid/419054da-0983-11e1-872d-14dae9978df7'
  whole_disk: 0
  DTL: 131
  features_for_read:
Uberblock:
  magic = 0000000000bab10c
  version = 5000
  txg = 15557291
  guid_sum = 14292448675790034744
  timestamp = 1415056517 UTC = Mon Nov  3 23:15:17 2014

/mnt/boot/loader.conf:
Code:
zfs_load="YES"
vfs.root.mountfrom="zfs:tank/root"
geom_mirror_load="YES"
vfs.zfs.zio.use_uma="0"
vm.pmap.pcid_enabled="0"

zfs list
Code:
NAME  USED  AVAIL  REFER  MOUNTPOINT
tank  583G  2.09T  21K  none
tank/root  581G  2.09T  547G  /mnt
tank/root/tmp  52.4M  2.09T  52.4M  /mnt/tmp
tank/root/var  33.4G  2.09T  33.4G  /mnt/var

zpool get all tank
Code:
NAME  PROPERTY  VALUE  SOURCE
tank  size  2.70T  -
tank  capacity  21%  -
tank  altroot  /mnt  local
tank  health  ONLINE  -
tank  guid  522225119579774437  default
tank  version  -  default
tank  bootfs  tank/root  local
tank  delegation  on  default
tank  autoreplace  off  default
tank  cachefile  -  default
tank  failmode  wait  default
tank  listsnapshots  off  default
tank  autoexpand  off  default
tank  dedupditto  0  default
tank  dedupratio  1.00x  -
tank  free  2.13T  -
tank  allocated  587G  -
tank  readonly  off  -
tank  comment  -  default
tank  expandsize  0  -
tank  freeing  0  default
tank  feature@async_destroy  enabled  local
tank  feature@empty_bpobj  active  local
tank  feature@lz4_compress  enabled  local
tank  feature@multi_vdev_crash_dump  enabled  local


gpart show
Code:
=>  34  5860533101  ada0  GPT  (2.7T)
  34  2014  - free -  (1.0M)
  2048  128  1  freebsd-boot  (64K)
  2176  33554432  2  freebsd-swap  (16G)
  33556608  5826976527  3  freebsd-zfs  (2.7T)

=>  34  5860533101  ada1  GPT  (2.7T)
  34  2014  - free -  (1.0M)
  2048  128  1  freebsd-boot  (64K)
  2176  33554432  2  freebsd-swap  (16G)
  33556608  5826976527  3  freebsd-zfs  (2.7T)
---> updated to:
Code:
=>        34  5860533101  ada0  GPT  (2.7T)
          34           6        - free -  (3.0K)
          40        1024     1  freebsd-boot  (512K)
        1064        1112        - free -  (556K)
        2176    33554432     2  freebsd-swap  (16G)
    33556608  5826976527     3  freebsd-zfs  (2.7T)

=>        34  5860533101  ada1  GPT  (2.7T)
          34           6        - free -  (3.0K)
          40        1024     1  freebsd-boot  (512K)
        1064        1112        - free -  (556K)
        2176    33554432     2  freebsd-swap  (16G)
    33556608  5826976527     3  freebsd-zfs  (2.7T)

zdb -dddd tank 1
Code:
Dataset mos [META], ID 0, cr_txg 4, 788M, 265 objects, rootbp DVA[0]=<0:15006d2b200:200> DVA[1]=<0:7007a07a00:200> DVA[2]=<0:e8022cd000:200> [L0 DMU objset] fletcher4 lzjb LE contiguous unique triple size=800L/200P birth=15566713L/15566713P fill=265 cksum=ee082d618:5c3ac9516bf:123d4df8262aa:274813f8559465

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         1    1    16K    16K  12.0K    32K  100.00  object directory
        dnode flags: USED_BYTES
        dnode maxblkid: 1
        Fat ZAP stats:
                Pointer table:
                        1024 elements
                        zt_blk: 0
                        zt_numblks: 0
                        zt_shift: 10
                        zt_blks_copied: 0
                        zt_nextblk: 0
                ZAP entries: 14
                Leaf blocks: 1
                Total blocks: 2
                zap_block_type: 0x8000000000000001
                zap_magic: 0x2f52ab2ab
                zap_salt: 0xa6d6f783
                Leafs with 2^n pointers:
                          9:      1 *
                Blocks with n*5 entries:
                          2:      1 *
                Blocks n/10 full:
                          2:      1 *
                Entries with n chunks:
                          3:     13 *************
                          4:      0
                          5:      0
                          6:      0
                          7:      0
                          8:      0
                          9:      1 *
                Buckets with n entries:
                          0:    498 ****************************************
                          1:     14 **

                feature_descriptions = 1205
                history = 22
                errlog_scrub = 0
                root_dataset = 2
                features_for_read = 1203
                errlog_last = 0
                pool_props = 45
                empty_bpobj = 1222
                sync_bplist = 21
                features_for_write = 1204
                free_bpobj = 1199
                deflate = 1
                config = 20
                scan = 1 3 0 0 15558907 0 15558907 1415066043 1415066126 630332826112 103698432 0 0 0 1 3 0 0 0 0 57 0 942 0


Before doing gpart bootcode I had error that version 5000 is not recognized at the boot.

Any hints how to go from here?
 
Last edited:
Booting from the rescue OS, add these to /boot/loader.conf
Code:
zfs_load="YES"
opensolaris_load="YES"
zpool_cache_load="YES"
zpool_cache_type="/boot/zfs/zpool.cache"
zpool_cache_name="/boot/zfs/zpool.cache"


If that fails: Boot from rescue OS and
  • Import the root_pool with -R flag (altroot)
  • # cp /boot/zfs/zpool.cache /<mnt_point>/boot/zfs/
  • zfs set mountpouint=/ [FILE]root_pool[/FILE]
  • zfs umount root_pool* (this may fail and you must unmount each of visible path on df). Do not export the root_pool.
  • Reboot
 
I did a clean installation of FreeBSD10-RELEASE on a dedicated server.
Also I tried to install OS manually - also get the same problem:
http://www.evernote.com/l/ASOE_b_O-0NJU7hwJY3o44EPIao5_DPWF0c/

Problem comes back after doing a restart just after freebsd-update fetch freebsd-update install
Putting details in /boot/loader.conf does not help, but thanks.

I am trying to import root_pool(tank/root) but I am getting this problem:

zpool import -R /mnt tank/root
Code:
cannot import 'tank/root': no such pool available

zpool list
Code:
no pools available

I did this:
Code:
zpool import -f -o altroot=/mnt -o cachefile=/boot/zfs/zpool.cache tank
cp /boot/zfs/zpool.cache /mnt/boot/zfs/
zfs umount /mnt/
##(zfs umont and all folders)
zfs set mountpoint=/ tank/root
reboot
This did not fix the problem. Still cannot boot.

I suspect a hardware problem (requested hardware check) or some FreeBSD 10-p12 problem.
(here is identical hardware spec: http://www.evernote.com/l/ASOMDRkU4WxG_5TOJGOABOyR3uDD4JHdp3Q/ )
 
Any idea how to solve this ?
Same problem while upgrading from 10.2-RELEASE-p14 to 10.3-RELEASE-p16

My server is down :(
Trying copy the cache, upgrade the pool version, ...
 
Back
Top