I'm running into an odd thing. Using HAST between two nodes, works fine. I export my zpool on node1, switch HAST roles so that node2 is the primary, then import the zpool. When I check my zpool status, it shows that the hot spare is unavailable.
First Node, current master:
Export zpool on first node and switch role to secondary:
Switch role to primary on second node and import:
As you can see, the ada12 disk is now marked as "UNAVAIL" on the second node.
Does ada12 exist?
...yes it does!
And to get things straightend out, I then have to remove the UNAVAIL spare and add it back again. Weird?
I figure I can script this anomaly, but I'm not sure how that might affect a zpool that is actually using the spare when it fails over ...though I probably should test that before role out.
Oh, and yes, if I switch back to the first node again, the spare is still unavailable.
Actually, this happens even without the switchover to the other node. Just import and export the zpool on the same system and the hot spare will show as UNAVAIL.
Speaking of which, even when the hot spare is seen and a drive fails, it doesn't engage. I've added that issue to my other thread.
Am I experiencing some limitations with the FreeBSD implemenation of ZFS or am I totally missing something?
Any helpful hints would be greatly appreciated!
thanks!
First Node, current master:
Code:
nas1# zpool add zroot spare /dev/hast/ada12
nas1# zpool status
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
hast/ada0 ONLINE 0 0 0
hast/ada1 ONLINE 0 0 0
hast/ada2 ONLINE 0 0 0
hast/ada3 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
hast/ada4 ONLINE 0 0 0
hast/ada5 ONLINE 0 0 0
hast/ada6 ONLINE 0 0 0
hast/ada7 ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
hast/ada8 ONLINE 0 0 0
hast/ada9 ONLINE 0 0 0
hast/ada10 ONLINE 0 0 0
hast/ada11 ONLINE 0 0 0
spares
hast/ada12 AVAIL
errors: No known data errorsExport zpool on first node and switch role to secondary:
Code:
nas1# zpool export -f zroot
nas1# zpool status
no pools available
nas1# hastctl role secondary allSwitch role to primary on second node and import:
Code:
nas2# hastctl role primary all
nas2# zpool import -fa
nas2# zpool status
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
hast/ada0 ONLINE 0 0 0
hast/ada1 ONLINE 0 0 0
hast/ada2 ONLINE 0 0 0
hast/ada3 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
hast/ada4 ONLINE 0 0 0
hast/ada5 ONLINE 0 0 0
hast/ada6 ONLINE 0 0 0
hast/ada7 ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
hast/ada8 ONLINE 0 0 0
hast/ada9 ONLINE 0 0 0
hast/ada10 ONLINE 0 0 0
hast/ada11 ONLINE 0 0 0
spares
dsk/hast/ada12 UNAVAIL cannot open
errors: No known data errorsAs you can see, the ada12 disk is now marked as "UNAVAIL" on the second node.
Does ada12 exist?
Code:
nas2# hastctl status ada12
ada12:
role: primary
provname: ada12
localpath: /dev/ada12
extentsize: 2097152 (2.0MB)
keepdirty: 64
remoteaddr: tcp4://10.10.10.13
replication: fullsync
status: complete
dirty: 0 (0B)
nas2# ls /dev/hast/ada12
/dev/hast/ada12...yes it does!
And to get things straightend out, I then have to remove the UNAVAIL spare and add it back again. Weird?
Code:
nas2# zpool remove zroot dsk/hast/ada12
nas2# zpool status
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
hast/ada0 ONLINE 0 0 0
hast/ada1 ONLINE 0 0 0
hast/ada2 ONLINE 0 0 0
hast/ada3 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
hast/ada4 ONLINE 0 0 0
hast/ada5 ONLINE 0 0 0
hast/ada6 ONLINE 0 0 0
hast/ada7 ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
hast/ada8 ONLINE 0 0 0
hast/ada9 ONLINE 0 0 0
hast/ada10 ONLINE 0 0 0
hast/ada11 ONLINE 0 0 0
errors: No known data errors
nas2# zpool add zroot spare /dev/hast/ada12
nas2# zpool status
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
hast/ada0 ONLINE 0 0 0
hast/ada1 ONLINE 0 0 0
hast/ada2 ONLINE 0 0 0
hast/ada3 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
hast/ada4 ONLINE 0 0 0
hast/ada5 ONLINE 0 0 0
hast/ada6 ONLINE 0 0 0
hast/ada7 ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
hast/ada8 ONLINE 0 0 0
hast/ada9 ONLINE 0 0 0
hast/ada10 ONLINE 0 0 0
hast/ada11 ONLINE 0 0 0
spares
hast/ada12 AVAIL
errors: No known data errorsI figure I can script this anomaly, but I'm not sure how that might affect a zpool that is actually using the spare when it fails over ...though I probably should test that before role out.
Oh, and yes, if I switch back to the first node again, the spare is still unavailable.
Actually, this happens even without the switchover to the other node. Just import and export the zpool on the same system and the hot spare will show as UNAVAIL.
Speaking of which, even when the hot spare is seen and a drive fails, it doesn't engage. I've added that issue to my other thread.
Am I experiencing some limitations with the FreeBSD implemenation of ZFS or am I totally missing something?
Any helpful hints would be greatly appreciated!
thanks!