I'm testing ZED notifications on Linux. ZED apparently isn't a package yet for FreeBSD, but I'll look at /etc/devd/zfs.conf later. For now I'm looking into ZED on Linux.
The only notifications via email I can get is when a pool finished scrub or when I zpool offline -f which offlines a disk and marks it FAULTED.
If I pull a disk, or destroy it and reboot it becomes UNAVAIL but no notifications are triggered. That's kind of useless.
Man page says a top-level vdev can be in a OFFLINE state, as well as DEGRADED and FAULTED.
Linux ZFS manual
Oracle manual
Yeah I don't know if those man pages are from the 70s or what but the only state of a pool or a top level vdev I can get is ONLINE and DEGRADED.
This is a pool I offlined and faulted three disk vdevs inside of each of the top-level vdevs which are mirrors. When I offline and fault two disks from a top-level vdev the second disk goes into DEGRADED state, probably because the pool would dissapear it went into a faulted state so that's good.
So if I pull the DEGRADED /dev/vdisk-2 or in this case delete it and reboot the pool dissapears.
So the pool as far as I know can never be in a FAULTED, OFFLINE or UNAVAIL state. I've seen it DEGRADED, ONLINE and VANISHED or Destroyed.
If I OFFLINE a disk top-level vdev is DEGRADED which makes sense because the other disk is ONLINE. And if I try to offline the other disk it won't allow me because there's no valid replicas. Which is also a good thing as that's essentially a zpool destroy command.
If I destroy the pool by destorying the disk it's not there anymore then I can finally see the pool state as UNAVAIL, but in the context of ZED sending notifications that's too late.
The reason I want to know this is because I'm trying to understand this script https://github.com/openzfs/zfs/blob/master/cmd/zed/zed.d/statechange-notify.sh
Specifically this part
Doesn't this mean in order for this not to exit the STATE has to be all of those at once? Because || operator isn't used.
Trying to understand this. So if ZEVENT isn't FAULTED and DEGRADED ... exit and it can't ever be all of those conditions.
I can't undertand this logic. I do know it works for FAULTED and nothing else, or is it DEGRADED.
That's another part of my problem I'm trying to understand what does it query. The top-level vdev or the disk vdev.
I guess the script queries the state of the disk vdev because when the Disk is FAULTED I gen an email, but not when disk is UNAVAIL or OFFLINE and each of those makes the top-level vdev DEGRADED and stil no email.
I would write this as "if degraded send notification else exit" instead of "if not degraded exit else send notification". Although this way does make sense because it's kind of like "if disk not broken exit" but there's four conditions here.
I run zpool status -x to check for faulty pools but I wonder what piece of code or which script does that call. Would like to check what that looks like.
Any help or pointers would be nice. I have to read about ZEDLETS as I have no idea what that is, and buch of other things.
Even if you have a screenshot of a pool being in anything other than ONLINE or DEGRADED I would like to see that.
And if someone could tell me how to put a pool or a top-level vdev into those states manually so I can test notifications would also be nice.
The only notifications via email I can get is when a pool finished scrub or when I zpool offline -f which offlines a disk and marks it FAULTED.
If I pull a disk, or destroy it and reboot it becomes UNAVAIL but no notifications are triggered. That's kind of useless.
Man page says a top-level vdev can be in a OFFLINE state, as well as DEGRADED and FAULTED.
Linux ZFS manual
Code:
A pool's health status is described by one of three states: online, degraded, or faulted.
A top-level vdev or component device is in one of the following states: DEGRADED, FAULTED, OFFLINE, ONLINE, REMOVED, UNAVAIL.Oracle manual
Code:
You cannot take a pool offline to the point where it becomes faulted. For example, you cannot take offline two devices in a raidz1 configuration, nor can you take offline a top-level virtual device.Yeah I don't know if those man pages are from the 70s or what but the only state of a pool or a top level vdev I can get is ONLINE and DEGRADED.
This is a pool I offlined and faulted three disk vdevs inside of each of the top-level vdevs which are mirrors. When I offline and fault two disks from a top-level vdev the second disk goes into DEGRADED state, probably because the pool would dissapear it went into a faulted state so that's good.
Code:
NAME STATE READ WRITE CKSUM
zed DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
/dev/vdisk-0 ONLINE 0 0 0
/dev/vdisk-1 FAULTED 0 0 0 external device fault
mirror-1 DEGRADED 0 0 0
/dev/vdisk-2 DEGRADED 0 0 0 external device fault
/dev/vdisk-3 FAULTED 0 0 0 external device faultSo if I pull the DEGRADED /dev/vdisk-2 or in this case delete it and reboot the pool dissapears.
So the pool as far as I know can never be in a FAULTED, OFFLINE or UNAVAIL state. I've seen it DEGRADED, ONLINE and VANISHED or Destroyed.
If I OFFLINE a disk top-level vdev is DEGRADED which makes sense because the other disk is ONLINE. And if I try to offline the other disk it won't allow me because there's no valid replicas. Which is also a good thing as that's essentially a zpool destroy command.
Code:
NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
virtio-BHYVE-1BF2-A120-0300-part4 ONLINE 0 0 0
virtio-BHYVE-337A-48DB-7BA0-part4 OFFLINE 0 0 0If I destroy the pool by destorying the disk it's not there anymore then I can finally see the pool state as UNAVAIL, but in the context of ZED sending notifications that's too late.
Code:
gogofc@catsrunning:~$ sudo zpool import
pool: newpooltodestroy
id: 10993580688618051449
state: UNAVAIL
status: One or more devices are faulted.
action: The pool cannot be imported due to damaged devices or data.
config:
newpooltodestroy UNAVAIL insufficient replicas
vdc FAULTED corrupted dataThe reason I want to know this is because I'm trying to understand this script https://github.com/openzfs/zfs/blob/master/cmd/zed/zed.d/statechange-notify.sh
Specifically this part
Bash:
if [ "${ZEVENT_VDEV_STATE_STR}" != "FAULTED" ] \
&& [ "${ZEVENT_VDEV_STATE_STR}" != "DEGRADED" ] \
&& [ "${ZEVENT_VDEV_STATE_STR}" != "REMOVED" ] \
&& [ "${ZEVENT_VDEV_STATE_STR}" != "UNAVAIL" ]; then
exit 3
fiDoesn't this mean in order for this not to exit the STATE has to be all of those at once? Because || operator isn't used.
Trying to understand this. So if ZEVENT isn't FAULTED and DEGRADED ... exit and it can't ever be all of those conditions.
I can't undertand this logic. I do know it works for FAULTED and nothing else, or is it DEGRADED.
That's another part of my problem I'm trying to understand what does it query. The top-level vdev or the disk vdev.
I guess the script queries the state of the disk vdev because when the Disk is FAULTED I gen an email, but not when disk is UNAVAIL or OFFLINE and each of those makes the top-level vdev DEGRADED and stil no email.
I would write this as "if degraded send notification else exit" instead of "if not degraded exit else send notification". Although this way does make sense because it's kind of like "if disk not broken exit" but there's four conditions here.
I run zpool status -x to check for faulty pools but I wonder what piece of code or which script does that call. Would like to check what that looks like.
Any help or pointers would be nice. I have to read about ZEDLETS as I have no idea what that is, and buch of other things.
Even if you have a screenshot of a pool being in anything other than ONLINE or DEGRADED I would like to see that.
And if someone could tell me how to put a pool or a top-level vdev into those states manually so I can test notifications would also be nice.