ZFS ZFS: can't read MOS of pool rpool

Hi,

I've got a problem with booting system FreeBSD 10.3 after then I was removed one disk from pool.

I'm getting:
Code:
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool rpool
gptzfsboot: failed to mount default pool rpool

Command history

Code:
gpart create -s gpt ada1
gpart add -b 40 -s 1024 -t freebsd-boot ada1
gpart add -b 2048 -s 4194304 -t freebsd-swap ada1
gpart add -b 4196352 -s 37744640 -t freebsd-zfs ada1
zpool attach rpool ada0p3 ada1p3
waiting...
zpool detach rpool ada0p3
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
What is wrong ?
 
Looks like you added a striped set (AKA RAID0) to the pool. There's no fault tolerance here, removal (or failure) of a single drive will result in the loss of the entire pool.

Edit: Nope, spoke too soon. But what is the pool type before you added the new drive?
 
Also, I have tested another method:

Code:
root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # zpool replace rpool ada0p3 ada1p3
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'rpool', you may need to update
boot code on newly attached disk 'ada1p3'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Aug  5 14:02:21 2016
        648M scanned out of 1.62G at 40.5M/s, 0h0m to go
        648M resilvered, 38.94% done
config:

        NAME           STATE     READ WRITE CKSUM
        rpool          ONLINE       0     0     0
          replacing-0  ONLINE       0     0     0
            ada0p3     ONLINE       0     0     0
            ada1p3     ONLINE       0     0     0  (resilvering)

errors: No known data errors

root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 1.62G in 0h0m with 0 errors on Fri Aug  5 14:03:02 2016
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
bootcode written to ada1

root@FreeBSD:~ # reboot

And result is the same, ZFS: can't read MOS of pool rpool

On Solaris 11.3 everything is fine.
 
You need to wait for the resilvering to complete before removing ada0p3
 
Your last post shows it's only partially done:
Code:
  scan: resilver in progress since Fri Aug  5 14:02:21 2016
        648M scanned out of 1.62G at 40.5M/s, 0h0m to go
        648M resilvered, 38.94% done
Code:
  scan: resilvered 1.62G in 0h0m with 0 errors on Fri Aug  5 14:03:02 2016
 
Code:
  scan: resilver in progress since Fri Aug  5 14:02:21 2016
        648M scanned out of 1.62G at 40.5M/s, 0h0m to go
        648M resilvered, 38.94% done

This is only example, that resync is working.

Code:
  scan: resilvered 1.62G in 0h0m with 0 errors on Fri Aug  5 14:03:02 2016

This is output after resync.
 
Ok, I have tested another step.

Installing fresh system on zfs mirror with two disk.

Code:
root@FreeBSD:~ # zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  17.9G  1.62G  16.3G         -     5%     9%  1.00x  ONLINE  -

root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0

errors: No known data errors

Let's see what happens when unplugging ada1 disk

Code:
root@FreeBSD:~ # zpool detach rpool ada1p3

root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # reboot

After reeboot, looks fine:

Code:
root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors

Attach again ada1 to mirror.

Code:
root@FreeBSD:~ # zpool attach rpool ada0p3 ada1p3
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'rpool', you may need to update
boot code on newly attached disk 'ada1p3'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
       
root@FreeBSD:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
bootcode written to ada1       

root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 1.62G in 0h0m with 0 errors on Mon Aug  8 08:39:57 2016
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # zpool status -xv
all pools are healthy

Time to detach ada0 disk.

Code:
root@FreeBSD:~ # zpool detach rpool ada0p3

root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 1.62G in 0h0m with 0 errors on Mon Aug  8 08:39:57 2016
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # zpool scrub rpool

root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: scrub in progress since Mon Aug  8 08:46:39 2016
        1.59G scanned out of 1.62G at 81.3M/s, 0h0m to go
        0 repaired, 98.28% done
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors
root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: scrub in progress since Mon Aug  8 08:46:39 2016
        1.61G scanned out of 1.62G at 82.7M/s, 0h0m to go
        0 repaired, 99.89% done
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors
root@FreeBSD:~ # zpool status
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Aug  8 08:46:59 2016
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # reboot

And ..... Still the same.

ZFS: can't read MOS of pool rpool
 
I have the exact same problem. I have a pool consisting of a single SSD for the system. The strange thing is, that I used this very same method some 2(?) years ago to expand the storage of the system SSD. It's still in the history of ZFS. Now, I want to do the same thing again and it fails. Since this is from August of last year... have you made any progress Kajanos?
 
Ok, I have tested another step.

Installing fresh system on zfs mirror with two disk.

Code:
root@FreeBSD:~ # zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  17.9G  1.62G  16.3G         -     5%     9%  1.00x  ONLINE  -

root@FreeBSD:~ # zpool status
  pool: rpool
state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0

errors: No known data errors

Let's see what happens when unplugging ada1 disk

Code:
root@FreeBSD:~ # zpool detach rpool ada1p3

root@FreeBSD:~ # zpool status
  pool: rpool
state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # reboot

After reeboot, looks fine:

Code:
root@FreeBSD:~ # zpool status
  pool: rpool
state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors

Attach again ada1 to mirror.

Code:
root@FreeBSD:~ # zpool attach rpool ada0p3 ada1p3
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'rpool', you may need to update
boot code on newly attached disk 'ada1p3'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
      
root@FreeBSD:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
bootcode written to ada1      

root@FreeBSD:~ # zpool status
  pool: rpool
state: ONLINE
  scan: resilvered 1.62G in 0h0m with 0 errors on Mon Aug  8 08:39:57 2016
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # zpool status -xv
all pools are healthy

Time to detach ada0 disk.

Code:
root@FreeBSD:~ # zpool detach rpool ada0p3

root@FreeBSD:~ # zpool status
  pool: rpool
state: ONLINE
  scan: resilvered 1.62G in 0h0m with 0 errors on Mon Aug  8 08:39:57 2016
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # zpool scrub rpool

root@FreeBSD:~ # zpool status
  pool: rpool
state: ONLINE
  scan: scrub in progress since Mon Aug  8 08:46:39 2016
        1.59G scanned out of 1.62G at 81.3M/s, 0h0m to go
        0 repaired, 98.28% done
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors
root@FreeBSD:~ # zpool status
  pool: rpool
state: ONLINE
  scan: scrub in progress since Mon Aug  8 08:46:39 2016
        1.61G scanned out of 1.62G at 82.7M/s, 0h0m to go
        0 repaired, 99.89% done
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors
root@FreeBSD:~ # zpool status
  pool: rpool
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Aug  8 08:46:59 2016
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors

root@FreeBSD:~ # reboot

And ..... Still the same.

ZFS: can't read MOS of pool rpool
Any updates on how you resolved this?
 
Back
Top