Solved zfs zabbix monitoring

OP
OP
fred974

fred974

Daemon

Reaction score: 36
Messages: 1,557

From the host zpool iostat -H -p zroot return
Code:
invalid option 'H'
usage:
        iostat [-v] [-T d|u] [pool] ... [interval [count]]
What memory usage? From a host?
Yes, the memory from the host is not showing up ..All I have is a blank graph
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,183
Messages: 29,471

From the host zpool iostat -H -p zroot return
Code:
invalid option 'H'
usage:
        iostat [-v] [-T d|u] [pool] ... [interval [count]]
Ok, this tells me the patch isn't applied, or not correctly applied. The patch makes these options available.

Yes, the memory from the host is not showing up ..All I have is a blank graph
Are other graphs from the same host also not showing? Check the host's configuration in Zabbix and make sure it's not disabled.
 
OP
OP
fred974

fred974

Daemon

Reaction score: 36
Messages: 1,557

After cd /usr/src && patch < ~/patches/zpool_raw_stats.patch do I need to reboot for the patch to apply?
Are other graphs from the same host also not showing?
Yes, I can see the dataset graph, network and other. only memory and I/O stats not showing
 
OP
OP
fred974

fred974

Daemon

Reaction score: 36
Messages: 1,557

/usr/local/etc/zabbix34/zabbix_agentd.conf
Code:
# This is a configuration file for Zabbix agent daemon (Unix)
# To get more information about Zabbix, visit http://www.zabbix.com

############ GENERAL PARAMETERS #################

### Option: PidFile
#       Name of PID file.
#
# Mandatory: no
# Default:
# PidFile=/var/run/zabbix/zabbix_agentd.pid

### Option: LogType
#       Specifies where log messages are written to:
#               system  - syslog
#               file    - file specified with LogFile parameter
#               console - standard output
#
# Mandatory: no
# Default:
# LogType=file

### Option: LogFile
#       Log file name for LogType 'file' parameter.
#
# Mandatory: no
# Default:
# LogFile=

LogFile=/tmp/zabbix_agentd.log

### Option: LogFileSize
#       Maximum size of log file in MB.
#       0 - disable automatic log rotation.
#
# Mandatory: no
# Range: 0-1024
# Default:
# LogFileSize=1

### Option: DebugLevel
#       Specifies debug level:
#       0 - basic information about starting and stopping of Zabbix processes
#       1 - critical information
#       2 - error information
#       3 - warnings
#       4 - for debugging (produces lots of information)
#       5 - extended debugging (produces even more information)
#
# Mandatory: no
# Range: 0-5
# Default:
# DebugLevel=3

### Option: SourceIP
#       Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=

### Option: EnableRemoteCommands
#       Whether remote commands from Zabbix server are allowed.
#       0 - not allowed
#       1 - allowed
#
# Mandatory: no
# Default:
# EnableRemoteCommands=0

### Option: LogRemoteCommands
#       Enable logging of executed shell commands as warnings.
#       0 - disabled
#       1 - enabled
#
# Mandatory: no
# Default:
# LogRemoteCommands=0

##### Passive checks related

### Option: Server
#       List of comma delimited IP addresses, optionally in CIDR notation, or hostnames of Zabbix servers and Zabbix proxies.
#       Incoming connections will be accepted only from the hosts listed here.
#       If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally and '::/0' will allow any IPv4 or IPv6 address.
#       '0.0.0.0/0' can be used to allow any IPv4 address.
#       Example: Server=127.0.0.1,192.168.1.0/24,::1,2001:db8::/32,zabbix.domain
#
# Mandatory: no
# Default:
# Server=

Server=20xx.xx.xx.x

### Option: ListenPort
#       Agent will listen on this port for connections from the server.
#
# Mandatory: no
# Range: 1024-32767
# Default:
ListenPort=10050

### Option: ListenIP
#       List of comma delimited IP addresses that the agent should listen on.
#       First IP address is sent to Zabbix server if connecting to it to retrieve list of active checks.
#
# Mandatory: no
# Default:
# ListenIP=0.0.0.0

### Option: StartAgents
#       Number of pre-forked instances of zabbix_agentd that process passive checks.
#       If set to 0, disables passive checks and the agent will not listen on any TCP port.
#
# Mandatory: no
# Range: 0-100
# Default:
# StartAgents=3

##### Active checks related

### Option: ServerActive
#       List of comma delimited IP:port (or hostname:port) pairs of Zabbix servers and Zabbix proxies for active checks.
#       If port is not specified, default port is used.
#       IPv6 addresses must be enclosed in square brackets if port for that host is specified.
#       If port is not specified, square brackets for IPv6 addresses are optional.
#       If this parameter is not specified, active checks are disabled.
#       Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=

ServerActive=208.95.3.6

### Option: Hostname
#       Unique, case sensitive hostname.
#       Required for active checks and must match hostname as configured on the server.
#       Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

#Hostname=Zabbix server

### Option: HostnameItem
#       Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.
#       Does not support UserParameters or aliases.
#
# Mandatory: no
# Default:
HostnameItem=system.hostname

### Option: HostMetadata
#       Optional parameter that defines host metadata.
#       Host metadata is used at host auto-registration process.
#       An agent will issue an error and not start if the value is over limit of 255 characters.
#       If not defined, value will be acquired from HostMetadataItem.
#
# Mandatory: no
# Range: 0-255 characters
# Default:
# HostMetadata=

### Option: HostMetadataItem
#       Optional parameter that defines an item used for getting host metadata.
#       Host metadata is used at host auto-registration process.
#       During an auto-registration request an agent will log a warning message if
#       the value returned by specified item is over limit of 255 characters.
#       This option is only used when HostMetadata is not defined.
#
# Mandatory: no
# Default:
# HostMetadataItem=

### Option: RefreshActiveChecks
#       How often list of active checks is refreshed, in seconds.
#
# Mandatory: no
# Range: 60-3600
# Default:
# RefreshActiveChecks=120

### Option: BufferSend
#       Do not keep data longer than N seconds in buffer.
#
# Mandatory: no
# Range: 1-3600
# Default:
# BufferSend=5

### Option: BufferSize
#       Maximum number of values in a memory buffer. The agent will send
#       all collected data to Zabbix Server or Proxy if the buffer is full.
#
# Mandatory: no
# Range: 2-65535
# Default:
# BufferSize=100

### Option: MaxLinesPerSecond
#       Maximum number of new lines the agent will send per second to Zabbix Server
#       or Proxy processing 'log' and 'logrt' active checks.
#       The provided value will be overridden by the parameter 'maxlines',
#       provided in 'log' or 'logrt' item keys.
#
# Mandatory: no
# Range: 1-1000
# Default:
# MaxLinesPerSecond=20

############ ADVANCED PARAMETERS #################

### Option: Alias
#       Sets an alias for an item key. It can be used to substitute long and complex item key with a smaller and simpler one.
#       Multiple Alias parameters may be present. Multiple parameters with the same Alias key are not allowed.
#       Different Alias keys may reference the same item key.
#       For example, to retrieve the ID of user 'zabbix':
#       Alias=zabbix.userid:vfs.file.regexp[/etc/passwd,^zabbix:.:([0-9]+),,,,\1]
#       Now shorthand key zabbix.userid may be used to retrieve data.
#       Aliases can be used in HostMetadataItem but not in HostnameItem parameters.
#
# Mandatory: no
# Range:
# Default:

### Option: Timeout
#       Spend no more than Timeout seconds on processing
#
# Mandatory: no
# Range: 1-30
# Default:
# Timeout=3

### Option: AllowRoot
#       Allow the agent to run as 'root'. If disabled and the agent is started by 'root', the agent
#       will try to switch to the user specified by the User configuration option instead.
#       Has no effect if started under a regular user.
#       0 - do not allow
#       1 - allow
#
# Mandatory: no
# Default:
# AllowRoot=0

### Option: User
#       Drop privileges to a specific, existing user on the system.
#       Only has effect if run as 'root' and AllowRoot is disabled.
#
# Mandatory: no
# Default:
# User=zabbix

### Option: Include
#       You may include individual files or all files in a directory in the configuration file.
#       Installing Zabbix will create include directory in /usr/local/etc, unless modified during the compile time.
#
# Mandatory: no
# Default:
# Include=

# Include=/usr/local/etc/zabbix3/zabbix4_agentd.userparams.conf
# Include=/usr/local/etc/zabbix3/zabbix4_agentd.conf.d/
# Include=/usr/local/etc/zabbix3/zabbix4_agentd.conf.d/*.conf

####### USER-DEFINED MONITORED PARAMETERS #######

### Option: UnsafeUserParameters
#       Allow all characters to be passed in arguments to user-defined parameters.
#       The following characters are not allowed:
#       \ ' " ` * ? [ ] { } ~ $ ! & ; ( ) < > | # @
#       Additionally, newline characters are not allowed.
#       0 - do not allow
#       1 - allow
#
# Mandatory: no
# Range: 0-1
# Default:
# UnsafeUserParameters=0

### Option: UserParameter
#       User-defined parameter to monitor. There can be several user-defined parameters.
#       Format: UserParameter=<key>,<shell command>
#       See 'zabbix_agentd' directory for examples.
#
# Mandatory: no
# Default:
# UserParameter=

####### LOADABLE MODULES #######

### Option: LoadModulePath
#       Full path to location of agent modules.
#       Default depends on compilation options.
#
# Mandatory: no
# Default:
# LoadModulePath=${libdir}/modules

### Option: LoadModule
#       Module to load at agent startup. Modules are used to extend functionality of the agent.
#       Format: LoadModule=<module.so>
#       The modules must be located in directory specified by LoadModulePath.
#       It is allowed to include multiple LoadModule parameters.
#
# Mandatory: no
# Default:
# LoadModule=

####### TLS-RELATED PARAMETERS #######

### Option: TLSConnect
#       How the agent should connect to server or proxy. Used for active checks.
#       Only one value can be specified:
#               unencrypted - connect without encryption
#               psk         - connect using TLS and a pre-shared key
#               cert        - connect using TLS and a certificate
#
# Mandatory: yes, if TLS certificate or PSK parameters are defined (even for 'unencrypted' connection)
# Default:
# TLSConnect=unencrypted

### Option: TLSAccept
#       What incoming connections to accept.
#       Multiple values can be specified, separated by comma:
#               unencrypted - accept connections without encryption
#               psk         - accept connections secured with TLS and a pre-shared key
#               cert        - accept connections secured with TLS and a certificate
#
# Mandatory: yes, if TLS certificate or PSK parameters are defined (even for 'unencrypted' connection)
# Default:
# TLSAccept=unencrypted

### Option: TLSCAFile
#       Full pathname of a file containing the top-level CA(s) certificates for
#       peer certificate verification.
#
# Mandatory: no
# Default:
# TLSCAFile=

### Option: TLSCRLFile
#       Full pathname of a file containing revoked certificates.
#
# Mandatory: no
# Default:
# TLSCRLFile=

### Option: TLSServerCertIssuer
#      Allowed server certificate issuer.
#
# Mandatory: no
# Default:
# TLSServerCertIssuer=

### Option: TLSServerCertSubject
#      Allowed server certificate subject.
#
# Mandatory: no
# Default:
# TLSServerCertSubject=

### Option: TLSCertFile
#       Full pathname of a file containing the agent certificate or certificate chain.
#
# Mandatory: no
# Default:
# TLSCertFile=

### Option: TLSKeyFile
#       Full pathname of a file containing the agent private key.
#
# Mandatory: no
# Default:
# TLSKeyFile=

### Option: TLSPSKIdentity
#       Unique, case sensitive string used to identify the pre-shared key.
#
# Mandatory: no
# Default:
# TLSPSKIdentity=

### Option: TLSPSKFile
#       Full pathname of a file containing the pre-shared key.
#
# Mandatory: no
# Default:
# TLSPSKFile=

# load SirDice custom script
Include=/usr/local/etc/zabbix/zabbix_agentd.conf.d/*.conf
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,183
Messages: 29,471

do I need to reboot for the patch to apply?
You need to buildworld. You're patching source code, which means you need to actually build that modified code into an executable.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,183
Messages: 29,471

You can remove almost everything from zabbix_agentd.conf. Pretty much everything is commented out and the rest is actually the default. There are really only a few options that are needed:
Code:
Server=<IP address of Zabbix host>
ServerActive=<IP address of Zabbix host>
Hostname=<FQDN of the host>

# Not required but useful:
LogType=file
LogFile=/tmp/zabbix_agentd.log
Everything else can be removed. Leave LogType set to file, the agent will log its actions to /tmp/zabbix_agentd.log. Useful when debugging. For production set LogType to system.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,183
Messages: 29,471

so just before I dive in using .... can I break anything?
Definitely. But in the past 20 years or so I've been building FreeBSD from source it only happened maybe once or twice. And that happened because I screwed something up, not because of anything from the build itself.
Would I need to make another buildworld when I apply the next freebsd-update?
It depends. If the update replaces /sbin/zpool, yes. If not, no.
 
OP
OP
fred974

fred974

Daemon

Reaction score: 36
Messages: 1,557

You need to buildworld. You're patching source code, which means you need to actually build that modified code into an executable.
I just finish doing the buildworld and I still have the same issue # zpool iostat -H -p zroot
Code:
invalid option 'H'
usage:
        iostat [-v] [-T d|u] [pool] ... [interval [count]]
Still no data in zabbix:(
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,183
Messages: 29,471

After building it you need to installworld.
 
OP
OP
fred974

fred974

Daemon

Reaction score: 36
Messages: 1,557

After building it you need to installworld.
I did..Here is the shell history
# rm -rf /usr/obj
# cd /usr/src/
# make -j8 buildworld
# make -j8 kernel
# shutdown -r now
# cd /usr/src/
# make installworld
# mergemaster -Ui
# shutdown -r now
# cd /usr/src
# make check-old
# make delete-old
# make check-old-libs
# make delete-old-libs
# pkg upgrade

Is there a command to see if I am using the new buildworld?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,183
Messages: 29,471

Is there a command to see if I am using the new buildworld?
No, not really. Technically there's no difference, you're still on the exact same version.
 
OP
OP
fred974

fred974

Daemon

Reaction score: 36
Messages: 1,557

zpool_raw_stats.patch
Code:
Index: cddl/contrib/opensolaris/cmd/zpool/zpool_main.c
===================================================================
--- cddl/contrib/opensolaris/cmd/zpool/zpool_main.c     (revision 327490)
+++ cddl/contrib/opensolaris/cmd/zpool/zpool_main.c     (working copy)
@@ -233,7 +233,7 @@
                    "[-R root] [-F [-n]]\n"
                    "\t    <pool | id> [newpool]\n"));
        case HELP_IOSTAT:
-               return (gettext("\tiostat [-v] [-T d|u] [pool] ... [interval "
+               return (gettext("\tiostat [-Hpv] [-T d|u] [pool] ... [interval "
                    "[count]]\n"));
        case HELP_LABELCLEAR:
                return (gettext("\tlabelclear [-f] <vdev>\n"));
@@ -2324,6 +2324,8 @@
        int cb_namewidth;
        int cb_iteration;
        zpool_list_t *cb_list;
+       boolean_t cb_literal;
+       boolean_t cb_scripted;
 } iostat_cbdata_t;

 static void
@@ -2331,6 +2333,9 @@
 {
        int i = 0;

+       if (cb->cb_scripted || cb->cb_literal)
+               return;
+
        for (i = 0; i < cb->cb_namewidth; i++)
                (void) printf("-");
        (void) printf("  -----  -----  -----  -----  -----  -----\n");
@@ -2339,11 +2344,18 @@
 static void
 print_iostat_header(iostat_cbdata_t *cb)
 {
-       (void) printf("%*s     capacity     operations    bandwidth\n",
-           cb->cb_namewidth, "");
-       (void) printf("%-*s  alloc   free   read  write   read  write\n",
-           cb->cb_namewidth, "pool");
-       print_iostat_separator(cb);
+       if (cb->cb_scripted)
+               return;
+
+       if (cb->cb_literal) {
+               (void) printf("pool/dev\talloc\tfree\trops\twops\trbytes\twbytes\n");
+       } else {
+               (void) printf("%*s     capacity     operations    bandwidth\n",
+                   cb->cb_namewidth, "");
+               (void) printf("%-*s  alloc   free   read  write   read  write\n",
+                   cb->cb_namewidth, "pool");
+               print_iostat_separator(cb);
+       }
 }

 /*
@@ -2350,12 +2362,16 @@
  * Display a single statistic.
  */
 static void
-print_one_stat(uint64_t value)
+print_one_stat(uint64_t value, iostat_cbdata_t *cb)
 {
        char buf[64];
-
-       zfs_nicenum(value, buf, sizeof (buf));
-       (void) printf("  %5s", buf);
+
+       if (cb->cb_literal) {
+               (void) printf("\t%lu", value);
+       } else {
+               zfs_nicenum(value, buf, sizeof (buf));
+               (void) printf("  %5s", buf);
+       }
 }

 /*
@@ -2385,11 +2401,15 @@
        verify(nvlist_lookup_uint64_array(newnv, ZPOOL_CONFIG_VDEV_STATS,
            (uint64_t **)&newvs, &c) == 0);

-       if (strlen(name) + depth > cb->cb_namewidth)
-               (void) printf("%*s%s", depth, "", name);
-       else
-               (void) printf("%*s%s%*s", depth, "", name,
-                   (int)(cb->cb_namewidth - strlen(name) - depth), "");
+       if (cb->cb_literal) {
+               (void) printf("%.*s%s", depth, "::::::", name);
+       } else {
+               if (strlen(name) + depth > cb->cb_namewidth)
+                       (void) printf("%*s%s", depth, "", name);
+               else
+                       (void) printf("%*s%s%*s", depth, "", name,
+                           (int)(cb->cb_namewidth - strlen(name) - depth), "");
+       }

        tdelta = newvs->vs_timestamp - oldvs->vs_timestamp;

@@ -2398,25 +2418,31 @@
        else
                scale = (double)NANOSEC / tdelta;

+       if (cb->cb_literal)
+               scale = 1.0;
+
        /* only toplevel vdevs have capacity stats */
        if (newvs->vs_space == 0) {
-               (void) printf("      -      -");
+               if (cb->cb_literal)
+                       (void) printf("\t-\t-");
+               else
+                       (void) printf("      -      -");
        } else {
-               print_one_stat(newvs->vs_alloc);
-               print_one_stat(newvs->vs_space - newvs->vs_alloc);
+               print_one_stat(newvs->vs_alloc, cb);
+               print_one_stat(newvs->vs_space - newvs->vs_alloc, cb);
        }

        print_one_stat((uint64_t)(scale * (newvs->vs_ops[ZIO_TYPE_READ] -
-           oldvs->vs_ops[ZIO_TYPE_READ])));
+           oldvs->vs_ops[ZIO_TYPE_READ])), cb);

        print_one_stat((uint64_t)(scale * (newvs->vs_ops[ZIO_TYPE_WRITE] -
-           oldvs->vs_ops[ZIO_TYPE_WRITE])));
+           oldvs->vs_ops[ZIO_TYPE_WRITE])), cb);

        print_one_stat((uint64_t)(scale * (newvs->vs_bytes[ZIO_TYPE_READ] -
-           oldvs->vs_bytes[ZIO_TYPE_READ])));
+           oldvs->vs_bytes[ZIO_TYPE_READ])), cb);

        print_one_stat((uint64_t)(scale * (newvs->vs_bytes[ZIO_TYPE_WRITE] -
-           oldvs->vs_bytes[ZIO_TYPE_WRITE])));
+           oldvs->vs_bytes[ZIO_TYPE_WRITE])), cb);

        (void) printf("\n");

@@ -2454,9 +2480,11 @@
         */

        if (num_logs(newnv) > 0) {
-               (void) printf("%-*s      -      -      -      -      -      "
-                   "-\n", cb->cb_namewidth, "logs");
-
+               if (!cb->cb_scripted) {
+                       (void) printf("%-*s      -      -      -"
+                           "      -      -      -\n",
+                           cb->cb_namewidth, "logs");
+               }
                for (c = 0; c < children; c++) {
                        uint64_t islog = B_FALSE;
                        (void) nvlist_lookup_uint64(newchild[c],
@@ -2682,14 +2710,20 @@
        unsigned long interval = 0, count = 0;
        zpool_list_t *list;
        boolean_t verbose = B_FALSE;
-       iostat_cbdata_t cb;
+       iostat_cbdata_t cb = {0};

        /* check options */
-       while ((c = getopt(argc, argv, "T:v")) != -1) {
+       while ((c = getopt(argc, argv, "HT:pv")) != -1) {
                switch (c) {
+               case 'H':
+                       cb.cb_scripted = B_TRUE;
+                       break;
                case 'T':
                        get_timestamp_arg(*optarg);
                        break;
+               case 'p':
+                       cb.cb_literal = B_TRUE;
+                       break;
                case 'v':
                        verbose = B_TRUE;
                        break;
@@ -2769,7 +2803,7 @@
                if (npools > 1 && !verbose)
                        print_iostat_separator(&cb);

-               if (verbose)
+               if (verbose && !cb.cb_scripted)
                        (void) printf("\n");

                /*
Did I not apply the correct patch?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,183
Messages: 29,471

That looks like the correct patch. You can tell by the HELP_IOSTAT line, it shows both -H and -p as valid options.
 
OP
OP
fred974

fred974

Daemon

Reaction score: 36
Messages: 1,557

file in post #20 looks a bit different..
doing another buildworld with that version now.. keep you updated
 
OP
OP
fred974

fred974

Daemon

Reaction score: 36
Messages: 1,557

With latest file from post #20 above
# zpool iostat -H -p zroot
Code:
zroot   258127761408    600865697792    47766   32816   838795264       266481664
Thank you very much SirDice Eric A. Borisch
 

Quip

New Member

Reaction score: 3
Messages: 13

Code:
root@hosaka:~ # env ZPOOL_RAW_STATS=1 zpool iostat stor10k
pool/dev,alloc,free,rops,wops,rbytes,wbytes
stor10k,28027212288,1165973696000,43236,51412,261913600,607056896
Still contemplating on how to get it into Zabbix though. As you may or may not know Zabbix expects 1 item, 1 value. Which makes it a little tricky to query as there are 4 values of interest (rops, wops, rbytes and wbytes). But some simple parser should do, I don't think it'll be much of a problem if it's fired off a few times in quick succession once every 5 minutes. It may get inefficient if there's a large number of pools though.
I know I am replying to an old post. I just want to recommend to not call "any" command multiple times just to get different part of the output. Many commands are quick when you test it on an idle machine, but some of them are very slow on a busy machines (with hundreds of snapshots, clones etc.).
It's much better to run the command just once, store its value as "master item" and then get some parts as "dependent items".
There is an example in manual for getting parts of Apache /server-status
https://www.zabbix.com/documentation/3.4/manual/config/items/itemtypes/dependent_items
 
Top