Solved zfs zabbix monitoring

OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#26
zabbix_agentd -t vfs.zpool.discovery
Code:
vfs.zpool.discovery                           [t|{"data":[
{"{#ZPOOL}":"zroot"}
]}]
I added the template to host
Code:
    production.mydomain.co.uk    Applications 11    Items 1264    Triggers 344    Graphs 221    Discovery 4    Web    195.12.xx.xxx: 10050    Template FS ZFS, Template OS FreeBSD (Template App Zabbix Agent)
I can now see graphs for datasets. Can I see zfs frag ratio?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,422
Messages: 27,743

#27
Not sure if I had added that to this version. But it's fairly easy to add to the template. Just add vfs.zpool.get[#ZPOOL,fragmentation] to the item prototypes of the Zpool discovery.
 
OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#28
Is thsi what you mean?
/usr/local/etc/zabbix/zabbix_agentd.conf.d/userparam_zfs.conf
Code:
UserParameter=vfs.zpool.discovery,/usr/local/etc/zabbix/scripts/zpool-discovery.sh
UserParameter=vfs.zfs.discovery,/usr/local/etc/zabbix/scripts/zfs-discovery.sh
UserParameter=vfs.zfs.get[*],/sbin/zfs get -Hp -o value $2 $1 | sed -e 's/[x%]//'
UserParameter=vfs.zpool.get[*],/sbin/zpool get -Hp -o value $2 $1 | sed -e 's/[x%]//'
UserParameter=vfs.zpool.get[#ZPOOL,fragmentation]
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,422
Messages: 27,743

#29
No, not there. You add it to the template (the XML file you imported). The key vfs.zpool.get[#ZPOOL,fragmentation] is translated by Zabbix. The #ZPOOL is a macro that's used in the LLD (Low Level Discovery). There's some Zabbix magic happening and eventually it's /sbin/zpool get -Hp -o value fragmentation <#ZPOOL> that gets executed. There's no need to change any of the UserParameters, those are already set up correctly.

https://www.zabbix.com/documentation/3.4/manual/discovery/low_level_discovery
https://www.zabbix.com/documentation/3.4/manual/config/templates/template
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,422
Messages: 27,743

#30
I've updated the template, it now definitely does graph fragmentation. I've also added a few things like compression ratio and a whole bunch of ARC related sysctls. The I/O statistics work thanks to the patch provided by Eric A. Borisch.

Please use the updated template and scripts from here: https://github.com/SirDice/zabbix

I have a few other Zabbix scripts I've created over the years for things like MySQL, Apache and HAProxy. I'm going to add some of them there too. But I need to clean those up a bit before I would dare to publish them :D
 
OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#32
# service zabbix_agentd start
Code:
zabbix_agentd [63580]: ERROR: cannot add user parameter "system.os.short=/bin/freebsd-version -u": not comma-separated
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,422
Messages: 27,743

#33
Oh, oops. That line should read:
Code:
UserParameter=system.os.short,/bin/freebsd-version -u
 
OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#34
SirDice
Since installing the new version of the scrip, I cannot see any dataset graph any more (i could on old script) also, It doesn't show any memory usage either ..

Any idea?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,422
Messages: 27,743

#35
Check the "latest data" for the monitored hosts. You should see a lot of data points and check the time. Make sure the data is getting updated.
 
OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#36
I have no data :(
What I did is this:
Zabbix agent
rm -r /usr/local/etc/zabbix/*
cd /tmp/zabbix-master/
cp -r scripts zabbix_agentd.conf.d /usr/local/etc/zabbix
updated file with
Code:
UserParameter=system.os.short,/bin/freebsd-version -u
service zabbix_agentd stop
service zabbix_agentd start
Zabbix Host
Deleted old template
imported new template
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,422
Messages: 27,743

#37
In /usr/local/etc/zabbix34/zabbix_agentd.conf (assuming you're using 3.4) you need to add:
Code:
Include=/usr/local/etc/zabbix/zabbix_agentd.conf.d/*.conf
I moved all my scripts and additions to a more generic zabbix directory because I got tired of having to move things around after each Zabbix update.
 
OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#38
I was on zabbix 32 so updated to 34..
When I run zabbix_agentd -t vfs.zpool.discovery
Code:
vfs.zpool.discovery                           [t|sh: /usr/local/etc/zabbix/scripts/zpool-discovery.sh: Permission denied]
I then ran chmod +x zpool-discovery.sh and now get
Code:
vfs.zpool.discovery                           [t|{"data":[
{"{#ZPOOL}":"zroot"}
]}]
Do i need to make all the file excecutable?
 
OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#40
I have made everything inside sccript/ executable
Is the output bellow look ok?
zabbix_agentd -t vfs.zpool.discovery
Code:
vfs.zpool.discovery                           [t|{"data":[
{"{#ZPOOL}":"zroot"}
]}]
zabbix_agentd -t vfs.zpool-iostat
Code:
vfs.zpool-iostat                              [m|ZBX_NOTSUPPORTED] [Unsupported item key.]
zabbix_agentd -t vfs.zfs-discovery
Code:
vfs.zfs-discovery                             [m|ZBX_NOTSUPPORTED] [Unsupported item key.]
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,422
Messages: 27,743

#41
The keys are called vfs.zfs.discovery, not vfs.zfs-discovery. The vfs.zpool.discovery looks good. With the others a response of ZBX_NOTSUPPORTED means the key is wrong or not working correctly.
 
OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#42
zabbix_agentd -t vfs.zfs.discovery
Code:
vfs.zfs.discovery                             [t|{"data":[
{"{#ZFS}":"zroot"},
{"{#ZFS}":"zroot/ROOT"},
{"{#ZFS}":"zroot/ROOT/default"},
{"{#ZFS}":"zroot/tmp"},
{"{#ZFS}":"zroot/usr"},
{"{#ZFS}":"zroot/usr/home"},
{"{#ZFS}":"zroot/usr/jails"},
{"{#ZFS}":"zroot/usr/jails/bcbsd-ubnt-dsk1.vhd"},
{"{#ZFS}":"zroot/usr/jails/bollenberg"},
{"{#ZFS}":"zroot/usr/jails/childrenandanimals"},
{"{#ZFS}":"zroot/usr/jails/colocation"},
{"{#ZFS}":"zroot/usr/jails/devopslab"},
{"{#ZFS}":"zroot/usr/jails/duramat"},
{"{#ZFS}":"zroot/usr/jails/equifloor"},
{"{#ZFS}":"zroot/usr/jails/leedscircus"},
{"{#ZFS}":"zroot/usr/jails/mariadb"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var/db"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var/db/mysql"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var/db/mysql-innodb"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var/db/mysql-innodb-logs"},
{"{#ZFS}":"zroot/usr/jails/mmamats"},
{"{#ZFS}":"zroot/usr/jails/nginx"},
{"{#ZFS}":"zroot/usr/jails/ns1"},
{"{#ZFS}":"zroot/usr/jails/rddprojects"},
{"{#ZFS}":"zroot/usr/jails/softfloor"},
{"{#ZFS}":"zroot/usr/jails/softfloorkids"},
{"{#ZFS}":"zroot/usr/jails/sophimail"},
{"{#ZFS}":"zroot/usr/jails/trendytikes"},
{"{#ZFS}":"zroot/usr/jails/trinitech"},
{"{#ZFS}":"zroot/usr/jails/wordpress_tpl"},
{"{#ZFS}":"zroot/usr/ports"},
{"{#ZFS}":"zroot/usr/src"},
{"{#ZFS}":"zroot/var"},
{"{#ZFS}":"zroot/var/audit"},
{"{#ZFS}":"zroot/var/crash"},
{"{#ZFS}":"zroot/var/log"},
{"{#ZFS}":"zroot/var/mail"},
{"{#ZFS}":"zroot/var/tmp"}
]}]
zabbix_agentd -t vfs.zpool.discovery
Code:
vfs.zpool.discovery                           [t|{"data":[
{"{#ZPOOL}":"zroot"}
]}]
zabbix_agentd -t vfs.zpool.iostat
Code:
vfs.zpool.iostat                              [t|Usage: /usr/local/etc/zabbix/scripts/zpool-iostat.sh <zpool> <stat>]
Is that correct? not sure aboy the last command I ran
 

Eric A. Borisch

Well-Known Member

Thanks: 212
Messages: 338

#44
zabbix_agentd -t vfs.zfs.discovery
Code:
vfs.zfs.discovery                             [t|{"data":[
{"{#ZFS}":"zroot"},
{"{#ZFS}":"zroot/ROOT"},
{"{#ZFS}":"zroot/ROOT/default"},
{"{#ZFS}":"zroot/tmp"},
{"{#ZFS}":"zroot/usr"},
{"{#ZFS}":"zroot/usr/home"},
{"{#ZFS}":"zroot/usr/jails"},
{"{#ZFS}":"zroot/usr/jails/bcbsd-ubnt-dsk1.vhd"},
{"{#ZFS}":"zroot/usr/jails/bollenberg"},
{"{#ZFS}":"zroot/usr/jails/childrenandanimals"},
{"{#ZFS}":"zroot/usr/jails/colocation"},
{"{#ZFS}":"zroot/usr/jails/devopslab"},
{"{#ZFS}":"zroot/usr/jails/duramat"},
{"{#ZFS}":"zroot/usr/jails/equifloor"},
{"{#ZFS}":"zroot/usr/jails/leedscircus"},
{"{#ZFS}":"zroot/usr/jails/mariadb"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var/db"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var/db/mysql"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var/db/mysql-innodb"},
{"{#ZFS}":"zroot/usr/jails/mariadb/var/db/mysql-innodb-logs"},
{"{#ZFS}":"zroot/usr/jails/mmamats"},
{"{#ZFS}":"zroot/usr/jails/nginx"},
{"{#ZFS}":"zroot/usr/jails/ns1"},
{"{#ZFS}":"zroot/usr/jails/rddprojects"},
{"{#ZFS}":"zroot/usr/jails/softfloor"},
{"{#ZFS}":"zroot/usr/jails/softfloorkids"},
{"{#ZFS}":"zroot/usr/jails/sophimail"},
{"{#ZFS}":"zroot/usr/jails/trendytikes"},
{"{#ZFS}":"zroot/usr/jails/trinitech"},
{"{#ZFS}":"zroot/usr/jails/wordpress_tpl"},
{"{#ZFS}":"zroot/usr/ports"},
{"{#ZFS}":"zroot/usr/src"},
{"{#ZFS}":"zroot/var"},
{"{#ZFS}":"zroot/var/audit"},
{"{#ZFS}":"zroot/var/crash"},
{"{#ZFS}":"zroot/var/log"},
{"{#ZFS}":"zroot/var/mail"},
{"{#ZFS}":"zroot/var/tmp"}
]}]
zabbix_agentd -t vfs.zpool.discovery
Code:
vfs.zpool.discovery                           [t|{"data":[
{"{#ZPOOL}":"zroot"}
]}]
zabbix_agentd -t vfs.zpool.iostat
Code:
vfs.zpool.iostat                              [t|Usage: /usr/local/etc/zabbix/scripts/zpool-iostat.sh <zpool> <stat>]
Is that correct? not sure aboy the last command I ran
Try ‘vfs.zpool.iostat[zroot,rbytes]’ ... note this requires the patched zpool binary to work, too.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,422
Messages: 27,743

#46
Is that correct? not sure aboy the last command I ran
Yes, that looks perfect. The last command 'failed' because it requires some arguments, as Eric A. Borisch already figured out it's going to be something like zabbix_agentd -t 'vfs.zpool.iostat[zroot,rbytes]'. Make sure there are quotes around it or else the shell is going to try and interpret the brackets. But you don't really need to test that, you've already proven the key works through the UserParameter and executes the correct commands. You can try the next command from the Zabbix server itself, something like zabbix_get -s <host_to_check> -k vfs.zpool.discovery. It should output the exact same thing as the test with zabbix_agentd(8). When all that works you've proven that Zabbix can get the 'key', the agent correctly understands that key and translates it to a command (that's what those UserParameters do), and last but not least, the commands themselves are correct. Then in Zabbix itself, "Configuration" -> "Hosts" -> "Templates". Make sure the "Template FS ZFS" is linked to the host.

That should be it actually. Give it a couple of minutes to detect everything, especially those low level discovery rules can take a couple of checks to pick up all the dynamic stuff.

WHere can I find the patch?
Post #20 of this thread :D
 
OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#47
cd cddl/contrib/opensolaris/cmd/zpool/
patch -C < ~/patches/zpool_raw_stats.patch
Code:
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: cddl/contrib/opensolaris/cmd/zpool/zpool_main.c
|===================================================================
|--- cddl/contrib/opensolaris/cmd/zpool/zpool_main.c    (revision 327490)
|+++ cddl/contrib/opensolaris/cmd/zpool/zpool_main.c    (working copy)
--------------------------
Patching file zpool_main.c using Plan A...
Hunk #1 succeeded at 233.
Hunk #2 succeeded at 2324.
Hunk #3 succeeded at 2333.
Hunk #4 succeeded at 2344.
Hunk #5 succeeded at 2362.
Hunk #6 succeeded at 2401.
Hunk #7 succeeded at 2418.
Hunk #8 succeeded at 2480.
Hunk #9 succeeded at 2710.
Hunk #10 succeeded at 2803.
done
Sorry to bug you guys but does this mean I am in the righdirectory to run the patch?
Not sure what the "Hmm... Looks like a unified diff to me..." mean here
 
OP
OP
fred974

fred974

Daemon

Thanks: 34
Messages: 1,503

#49
SirDice, I have been using the scripts you kindly offer and it has been great to look at the outcome.
I just have 2 issue if you don't mind helping me with.
1. I no longer got any data in relation to memory usage in zabbix
2. I applied the patch above and I cannot see any data in I/O statistic
Any idea where I went wrong?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,422
Messages: 27,743

#50
I no longer got any data in relation to memory usage in zabbix
What memory usage? From a host? That should still work, my templates don't touch anything related to this.

I applied the patch above and I cannot see any data in I/O statistic
Check the commands to make sure it's been applied correctly; zpool iostat -H -p zroot for example. This should provide some numbers, not an error message ;)
 
Top