ZFS High Swap Usage, why? Where to start?

megapearl · Sep 21, 2018

Hi,

My SWAP usage on FreeBSD 11.2-RELEASE is 50% and increasing every day, how to find out what is causing it?

uname -a gives:

FreeBSD fileserver.flissinger.local 11.2-RELEASE-p2 FreeBSD 11.2-RELEASE-p2 #103: Wed Sep 5 15:45:01 CEST 2018 donald@fileserver.flissinger.local:/usr/obj/usr/src/sys/FILESERVER amd64

top gives:

Code:

last pid: 13128;  load averages:  0.38,  0.30,  0.29                                                                                                                                                                                                                                                 up 1+23:47:55  16:39:53
84 processes:  1 running, 83 sleeping
CPU:  0.1% user,  0.0% nice,  1.0% system,  0.0% interrupt, 98.9% idle
Mem: 574M Active, 1550M Inact, 5848M Laundry, 54G Wired, 736M Free
ARC: 48G Total, 2903M MFU, 43G MRU, 336K Anon, 244M Header, 1817M Other
     43G Compressed, 46G Uncompressed, 1.09:1 Ratio
Swap: 8192M Total, 4060M Used, 4132M Free, 49% Inuse

swapinfo gives:

Code:

Device          1K-blocks     Used    Avail Capacity
/dev/zvol/system/swap   8388608  4157420  4231188    50%

zpool status gives:

Code:

  pool: storage
state: ONLINE
  scan: scrub repaired 0 in 92h31m with 0 errors on Fri Sep 14 23:32:31 2018
config:

        NAME                                                          STATE     READ WRITE CKSUM
        storage                                                       ONLINE       0     0     0
          raidz2-0                                                    ONLINE       0     0     0
            diskid/DISK-K4K1SGWB%20%20%20%20%20%20%20%20%20%20%20%20  ONLINE       0     0     0
            diskid/DISK-WD-WCC130167463                               ONLINE       0     0     0
            diskid/DISK-WD-WCC130174628                               ONLINE       0     0     0
            diskid/DISK-WD-WCC5D1LJZ313                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E0291512                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E0293887                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E0291558                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E0294217                               ONLINE       0     0     0
          raidz2-2                                                    ONLINE       0     0     0
            diskid/DISK-WD-WCC4E5CY5VZ4                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E5FH401A                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E5FH4L24                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E5UJ0ACK                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E5UJ0PTH                               ONLINE       0     0     0
            diskid/DISK-WD-WCC7K7RZ0XV7                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E3AKJLU6                               ONLINE       0     0     0
            diskid/DISK-WD-WCC4E0029020                               ONLINE       0     0     0

errors: No known data errors

  pool: system
state: ONLINE
  scan: scrub repaired 0 in 0h12m with 0 errors on Sun Aug 26 03:13:32 2018
config:

        NAME        STATE     READ WRITE CKSUM
        system      ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors

sysctl -a gives:
https://pastebin.com/5YvHgCqz

Where to start?
Any help would be appreciated,

Best Regards,
Donald.

SirDice · Sep 21, 2018

What else is running on the machine?

MarcoB · Sep 21, 2018

I found out that periodic daily is using a lot of memory and adding swap every day on my machine. I'm not using ZFS though.

spork · Sep 22, 2018

In top, sort your processes by size: type "o" which will prompt for "order by" and type "size". There you'll see which processes are using the most memory - post the info here for comment.

megapearl · Sep 22, 2018

SirDice said:
What else is running on the machine?

ps aux

Code:

root@fileserver:/home/donald # ps aux
USER           PID   %CPU %MEM      VSZ     RSS TT  STAT STARTED        TIME COMMAND
root            11 1197.0  0.0        0     192  -  RNL  Wed16   46171:07.67 [idle]
asterisk      2318    7.0  0.1   183192   46620  -  Is   Wed16      16:37.26 /usr/local/sbin/asterisk -n -F -U asterisk
root             0    0.0  0.0        0    9632  -  DLs  Wed16     285:42.91 [kernel]
root             1    0.0  0.0     5428     208  -  ILs  Wed16       0:00.44 /sbin/init --
root             2    0.0  0.0        0      16  -  DL   Wed16       0:00.00 [crypto]
root             3    0.0  0.0        0      16  -  DL   Wed16       0:00.00 [crypto returns]
root             4    0.0  0.0        0      64  -  DL   Wed16      47:53.04 [cam]
root             5    0.0  0.0        0      16  -  DL   Wed16       0:00.00 [mpt_recovery0]
root             6    0.0  0.0        0     272  -  DL   Wed16      14:06.08 [zfskern]
root             7    0.0  0.0        0      16  -  DL   Wed16       0:00.00 [sctp_iterator]
root             8    0.0  0.0        0      16  -  DL   Wed16       0:46.15 [pf purge]
root             9    0.0  0.0        0      16  -  DL   Wed16       2:16.30 [rand_harvestq]
root            10    0.0  0.0        0      16  -  DL   Wed16       0:00.00 [audit]
root            12    0.0  0.0        0     432  -  WL   Wed16      94:59.55 [intr]
root            13    0.0  0.0        0      48  -  DL   Wed16       0:00.15 [geom]
root            14    0.0  0.0        0     160  -  DL   Wed16       0:25.51 [usb]
root            15    0.0  0.0        0      16  -  DL   Wed16       0:00.05 [soaiod1]
root            16    0.0  0.0        0      16  -  DL   Wed16       0:00.05 [soaiod2]
root            17    0.0  0.0        0      16  -  DL   Wed16       0:00.05 [soaiod3]
root            18    0.0  0.0        0      16  -  DL   Wed16       0:00.05 [soaiod4]
root            19    0.0  0.0        0      48  -  DL   Wed16       2:48.08 [pagedaemon]
root            20    0.0  0.0        0      16  -  DL   Wed16       0:00.16 [vmdaemon]
root            21    0.0  0.0        0      16  -  DNL  Wed16       0:00.00 [pagezero]
root            22    0.0  0.0        0      16  -  DL   Wed16       0:02.08 [bufdaemon]
root            23    0.0  0.0        0      16  -  DL   Wed16       0:01.55 [bufspacedaemon]
root            24    0.0  0.0        0      16  -  DL   Wed16       5:01.67 [syncer]
root            25    0.0  0.0        0      16  -  DL   Wed16       0:01.85 [vnlru]
root           113    0.0  0.0     6308       0  -  IWs  -           0:00.00 adjkerntz -i
root           412    0.0  0.0     9184     720  -  Ss   Wed16       0:15.32 /sbin/devd
root           495    0.0  0.0     6704    2432  -  Ss   Wed16       3:19.49 /usr/sbin/syslogd -a *.flissinger.local:* -v -v
root           504    0.0  0.0     6652    2000  -  Ss   Wed16       0:00.26 /usr/sbin/rpcbind -h 10.0.0.2
bind           595    0.0  0.1   133400   75632  -  Is   Wed16       4:40.18 /usr/local/sbin/named -u bind -c /usr/local/etc/namedb/named.conf
root           630    0.0  0.0     6976       0  -  IWs  -           0:00.00 nfsuserd: master (nfsuserd)
root           631    0.0  0.0     6980    1064  -  S    Wed16       0:00.36 nfsuserd: slave (nfsuserd)
root           633    0.0  0.0     6980    1036  -  I    Wed16       0:00.35 nfsuserd: slave (nfsuserd)
root           635    0.0  0.0     6980    1048  -  S    Wed16       0:00.38 nfsuserd: slave (nfsuserd)
root           636    0.0  0.0     6980    1048  -  I    Wed16       0:00.32 nfsuserd: slave (nfsuserd)
root           641    0.0  0.0    90668    1900  -  Is   Wed16       0:00.23 /usr/sbin/mountd -l -r -h 10.0.0.2 -p 4045 /etc/exports /etc/zfs/exports
root           647    0.0  0.0    90560    1924  -  Is   Wed16       0:00.25 nfsd: master (nfsd)
root           648    0.0  0.0     6316    1776  -  S    Wed16      87:14.80 nfsd: server (nfsd)
root           661    0.0  0.0    19376   10432  -  I    Wed16       0:32.39 /usr/local/bin/monit -c /usr/local/etc/monitrc
root           672    0.0  0.0    15552    5236  -  S    Wed16       2:39.45 /usr/local/bin/vmtoolsd -c /usr/local/share/vmware-tools/tools.conf -p /usr/local/lib/open-vm-tools/plugins/vmsvc
transmission   688    0.0  0.1    48980   34436  -  Ss   Wed16      77:14.46 /usr/local/bin/transmission-daemon -g /usr/local/etc/transmission -w /mnt/storage/Downloads/Temp/NoCategory -x /var/run/transmission/daemon.pid -e /var/log/transmission.log
root           691    0.0  0.0     6904    2200  -  Is   Wed16       0:00.01 /usr/local/libexec/in.tftpd -s /usr/local/share/tftpboot --secure --verbose --listen --address 10.0.0.2:69 --blocksize 1024 -P /var/run/tftpd.pid -l
root           746    0.0  0.0    12572   12688  -  Ss   Wed16       0:17.07 /usr/sbin/ntpd -g -c /etc/ntp.conf -l /var/log/ntpd.log -p /var/run/ntpd.pid
root           776    0.0  0.0    29004    5512  -  Ss   Wed16       0:03.59 /usr/local/sbin/nmbd --daemon --configfile=/usr/local/etc/smb4.conf
root           780    0.0  0.1   164656   68412  -  Is   Wed16       0:08.25 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
sonarr         791    0.0  0.3   464808  203992  -  Ss   Wed16      12:52.58 /usr/local/bin/mono /usr/local/share/sonarr/NzbDrone.exe --nobrowser --data=/usr/local/sonarr (mono-sgen)
root           952    0.0  0.0   127660   27956  -  S    Wed16       0:01.00 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root           953    0.0  0.0   122072   27372  -  I    Wed16       0:00.27 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root           955    0.0  0.0    10552    1696  -  I    Wed16       0:02.99 /usr/local/sbin/smartd -l local4 -c /usr/local/etc/smartd.conf -p /var/run/smartd.pid
rslsync        965    0.0  0.4   597888  285592  -  Ss   Wed16      81:39.95 /usr/local/bin/rslsync --config /usr/local/etc/rslsync.conf
mysql          973    0.0  0.0     7072       0  -  IWs  -           0:00.00 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/usr/local/etc/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --pid-file=/var/run/mysqld/mysqld.pid --user=mysql
mysql         2053    0.0  8.3 62382564 5590868  -  S    Wed16      54:36.85 /usr/local/libexec/mysqld --defaults-extra-file=/usr/local/etc/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --plugin-dir=/usr/local/lib/mysql/plugin --log-error=/var/log/mysql/mysql-error.log --open-files-limit=65535 --pid-file=/var
radarr        2075    0.0  0.6   573580  399604  -  Ss   Wed16      50:50.20 /usr/local/bin/mono2 /usr/local/share/radarr/Radarr.exe --nobrowser --data=/usr/local/radarr (mono-sgen)
freeradius    2079    0.0  0.0    40588   10100  -  Is   Wed16       0:01.35 /usr/local/sbin/radiusd
postgrey      2080    0.0  0.0    27924    9908  -  Is   Wed16       0:00.46 postgrey --inet=10023 -d --pidfile=/var/run/postgrey/postgrey.pid --dbdir=/var/db/postgrey (perl)
vscan         2086    0.0  0.1   163440   82756  -  Ss   Wed16       0:04.41 /usr/local/sbin/amavisd (master) (perl)
clamav        2096    0.0  0.8   614092  555320  -  Is   Wed16       6:10.34 /usr/local/sbin/clamd
clamav        2103    0.0  0.0    15172    3780  -  Is   Wed16       0:50.81 /usr/local/bin/freshclam --daemon -p /var/run/clamav/freshclam.pid
root          2110    0.0  0.0     8176    1476  -  Ss   Wed16       0:00.39 /usr/local/sbin/dovecot -c /usr/local/etc/dovecot/dovecot.conf
dovecot       2111    0.0  0.0     7844    1208  -  S    Wed16       0:00.06 dovecot/anvil
root          2112    0.0  0.0     7844    1008  -  S    Wed16       0:00.03 dovecot/log
root          2115    0.0  0.0     8240    2156  -  I    Wed16       0:00.18 dovecot/config
postfix       2117    0.0  0.0    15156    5196  -  Ss   Wed16       0:01.84 /usr/local/libexec/dk-filter -C internal=accept -l -p unix:/var/run/milterdk/filter -u postfix -P /var/run/milterdk/pid
postfix       2124    0.0  0.0    10640       0  -  IWs  -           0:00.00 /usr/local/sbin/opendkim -l -p /var/run/milteropendkim/socket -u postfix:mailnull -P /var/run/milteropendkim/pid -x /usr/local/etc/mail/opendkim.conf
postfix       2125    0.0  0.0    16404    6768  -  S    Wed16       0:01.81 /usr/local/sbin/opendkim -l -p /var/run/milteropendkim/socket -u postfix:mailnull -P /var/run/milteropendkim/pid -x /usr/local/etc/mail/opendkim.conf
root          2194    0.0  0.0    16852    1316  -  Ss   Wed16       0:03.34 /usr/local/libexec/postfix/master -w
postfix       2196    0.0  0.0    16916    1896  -  I    Wed16       0:00.82 qmgr -l -t fifo -u
nzbget        2204    0.0  0.2   248608  114048  -  Ss   Wed16     392:42.01 /usr/local/bin/nzbget -D
root          2208    0.0  0.0   224196   23136  -  Is   Wed16       0:00.35 /usr/local/bin/node /usr/local/lib/node_modules/forever/bin/monitor /usr/local/share/biertje/server.js
messagebus    2214    0.0  0.0     7436    2484  -  Is   Wed16       0:00.02 /usr/local/bin/dbus-daemon --system
avahi         2218    0.0  0.0     7696    3272  -  S    Wed16       0:40.06 avahi-daemon: running [Fileserver.local] (avahi-daemon)
root          2219    0.0  0.1   266292   36436  -  I    Wed16       0:01.57 /usr/local/bin/node /usr/local/share/biertje/server.js
root          2223    0.0  0.0    19148    5408  -  S    Wed16       0:08.47 /usr/local/sbin/netatalk
root          2251    0.0  0.0    27824    9312  -  I    Wed16       0:00.72 /usr/local/sbin/afpd -d -F /usr/local/etc/afp.conf
root          2252    0.0  0.0    18436    6676  -  I    Wed16       0:00.12 /usr/local/sbin/cnid_metad -d -F /usr/local/etc/afp.conf
headphones    2256    0.0  0.3   439452  172552  -  S    Wed16     167:07.78 python /usr/local/share/headphones/Headphones.py --daemon --quiet --nolaunch --port 8181 --pidfile /var/run/headphones/headphones.pid --config /usr/local/share/headphones/config.ini (python2.7)
root          2263    0.0  0.1    92164   52140  -  S    Wed16      15:24.83 /usr/local/bin/python2.7 /usr/local/bin/fail2ban-server --async -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid --loglevel INFO --logtarget /var/log/fail2ban.log --syslogsocket auto
root          2291    0.0  0.0    84304    5444  -  Is   Wed16       0:00.21 /usr/local/sbin/cupsd -C /usr/local/etc/cups/cupsd.conf -s /usr/local/etc/cups/cups-files.conf
colord        2303    0.0  0.0    28636    9432  -  I    Wed16       0:00.15 /usr/local/libexec/colord
root          2340    0.0  0.0    12964    5172  -  Is   Wed16       0:00.23 /usr/sbin/sshd
root          2345    0.0  0.0   187472   15836  -  Ss   Wed16       0:07.79 /usr/local/sbin/httpd -DNOHTTPACCEPT
root          2359    0.0  0.0     6748    1324  -  Is   Wed16       0:01.25 /usr/sbin/cron -s
postfix       2424    0.0  0.0    17008    2152  -  S    Wed16       0:00.44 tlsmgr -l -t unix -u
www           2434    0.0  0.0   187536    8716  -  I    Wed16       0:02.41 /usr/local/sbin/httpd -DNOHTTPACCEPT
root          2504    0.0  0.1   167860   68676  -  S    Wed17       0:00.83 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root          3023    0.0  0.1   176928   75116  -  S    Wed19       6:54.93 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
www           7715    0.0  0.0   187536   10952  -  I    Thu17       0:01.71 /usr/local/sbin/httpd -DNOHTTPACCEPT
www          11765    0.0  0.0   187536   18312  -  I    11:12       0:00.44 /usr/local/sbin/httpd -DNOHTTPACCEPT
www          11771    0.0  0.0   187536   18392  -  S    11:12       0:00.47 /usr/local/sbin/httpd -DNOHTTPACCEPT
www          11821    0.0  0.0   187536   18328  -  I    11:26       0:00.36 /usr/local/sbin/httpd -DNOHTTPACCEPT
www          11832    0.0  0.0   187536   18344  -  I    11:31       0:00.49 /usr/local/sbin/httpd -DNOHTTPACCEPT
www          11833    0.0  0.0   187536   18296  -  I    11:31       0:00.40 /usr/local/sbin/httpd -DNOHTTPACCEPT
www          11834    0.0  0.0   187536   18276  -  I    11:31       0:00.58 /usr/local/sbin/httpd -DNOHTTPACCEPT
www          11997    0.0  0.0   187536   18240  -  I    12:03       0:00.28 /usr/local/sbin/httpd -DNOHTTPACCEPT
www          12165    0.0  0.0   187536   18268  -  I    12:57       0:00.18 /usr/local/sbin/httpd -DNOHTTPACCEPT
vscan        12907    0.0  0.1   164832   85196  -  I    01:09       0:00.10 /usr/local/sbin/amavisd (ch19-avail) (perl)
vscan        12966    0.0  0.1   164832   85204  -  I    01:24       0:00.10 /usr/local/sbin/amavisd (ch19-avail) (perl)
tautulli     14015    0.0  0.2   171932  122484  -  S    04:52       0:36.42 /usr/local/bin/python /usr/local/share/tautulli/Tautulli.py --daemon --pidfile /usr/local/share/tautulli/tautulli.pid --quiet --nolaunch (python2.7)
root         15187    0.0  0.0    48336   23580  -  I    10:55       0:00.36 /usr/local/sbin/cnid_dbd -F /usr/local/etc/afp.conf -p /mnt/storage/Backup/TimeMachine -t 7 -l 5 -u donald
root         15213    0.0  0.1   176616   87208  -  I    11:02       0:01.29 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
postfix      15214    0.0  0.0    16864    8056  -  S    11:02       0:00.01 pickup -l -t fifo -u
root         15215    0.0  0.0    13276    8380  -  Is   11:02       0:00.02 sshd: donald [priv] (sshd)
donald       15217    0.0  0.0    13276    8456  -  S    11:02       0:00.02 sshd: donald@pts/0 (sshd)
postfix      15223    0.0  0.0    20868   10972  -  S    11:04       0:00.06 smtpd -n fileserver.flissinger.local:smtp -t inet -u -o stress= -o smtpd_tls_security_level=may -o smtpd_tls_req_ccert=no -o smtpd_sasl_auth_enable=no -o smtpd_tls_wrappermode=no -o syslog_name=postfix-smtp
postfix      15224    0.0  0.0    16868    8072  -  S    11:04       0:00.01 proxymap -t unix -u
postfix      15225    0.0  0.0    20516    9272  -  S    11:04       0:00.02 trivial-rewrite -n rewrite -t unix -u
postfix      15226    0.0  0.0    16860    8044  -  S    11:04       0:00.01 spawn -z -n spf-policy -t unix user=nobody argv=/usr/local/libexec/postfix-policyd-spf-perl
nobody       15227    0.0  0.0    26372   20056  -  Ss   11:04       0:00.23 /usr/local/bin/perl /usr/local/libexec/postfix-policyd-spf-perl
root          2404    0.0  0.0     6416    2156 v0  Is+  Wed16       0:00.00 /usr/libexec/getty Pc ttyv0
root          2405    0.0  0.0     6416    2156 v1  Is+  Wed16       0:00.00 /usr/libexec/getty Pc ttyv1
donald       15218    0.0  0.0     7152    3224  0  Is   11:02       0:00.01 -sh (sh)
root         15219    0.0  0.0     7216    3296  0  I    11:02       0:00.01 su
root         15220    0.0  0.0     9820    5156  0  S    11:02       0:00.03 _su (csh)
root         15229    0.0  0.0     7000    3092  0  R+   11:04       0:00.00 ps aux
root@fileserver:/home/donald #

megapearl · Sep 22, 2018

spork said:
In top, sort your processes by size: type "o" which will prompt for "order by" and type "size". There you'll see which processes are using the most memory - post the info here for comment.

top

Code:

last pid: 15232;  load averages:  0.25,  0.30,  0.30                                                                                                                                                                                                                                                 up 2+18:14:49  11:06:47
89 processes:  1 running, 88 sleeping
CPU:  0.2% user,  0.0% nice,  0.4% system,  0.1% interrupt, 99.3% idle
Mem: 1079M Active, 1916M Inact, 5937M Laundry, 52G Wired, 1775M Free
ARC: 47G Total, 2524M MFU, 42G MRU, 196K Anon, 240M Header, 1864M Other
     42G Compressed, 45G Uncompressed, 1.07:1 Ratio
Swap: 8192M Total, 4737M Used, 3455M Free, 57% Inuse

  PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 2053 mysql           34  20    0 60920M  5460M select  3  54:38   0.45% mysqld
 2096 clamav           2  20    0   600M   542M select  3   6:10   0.00% clamd
  965 rslsync         17  20    0   584M   279M uwait   9  81:40   0.10% rslsync
 2075 radarr          14  20    0   560M   390M uwait   7  50:51   0.03% mono-sgen
  791 sonarr          16  20    0   454M   199M uwait   3  12:53   0.03% mono-sgen
 2256 headphones      23  20    0   429M   169M select  7 167:08   0.08% python2.7
 2219 root            11  20    0   260M 36436K kqread  6   0:02   0.00% node
 2204 nzbget           9  20    0   243M   111M nanslp  4 392:43   0.74% nzbget
 2208 root            11  52    0   219M 23136K kqread  4   0:00   0.00% node
11771 www              1  20    0   183M 18392K kqread  7   0:00   0.00% httpd
11832 www              1  20    0   183M 18344K lockf   1   0:00   0.00% httpd
11821 www              1  20    0   183M 18328K lockf   6   0:00   0.00% httpd
11765 www              1  20    0   183M 18312K lockf   8   0:00   0.00% httpd
11833 www              1  20    0   183M 18296K lockf  11   0:00   0.00% httpd
11834 www              1  20    0   183M 18276K lockf   7   0:01   0.00% httpd
12165 www              1  20    0   183M 18268K lockf   3   0:00   0.00% httpd
11997 www              1  20    0   183M 18240K lockf  11   0:00   0.00% httpd
 7715 www              1  20    0   183M 10952K lockf   0   0:02   0.00% httpd
 2434 www              1  20    0   183M  8716K lockf   9   0:02   0.00% httpd
 2345 root             1  20    0   183M 15836K select  5   0:08   0.00% httpd
 2318 asterisk        63  32    0   179M 46620K select  6  16:38   0.42% asterisk
 3023 root             1  20    0   173M 75116K select  3   6:55   0.19% smbd
15213 root             1  20    0   172M 87208K select  3   0:01   0.00% smbd
14015 tautulli        22  20    0   168M   120M select  5   0:37   0.10% python2.7
 2504 root             1  20    0   164M 68676K select  6   0:01   0.00% smbd
12966 vscan            1  20    0   161M 85204K lockf   9   0:00   0.00% perl
12907 vscan            1  20    0   161M 85196K select  1   0:00   0.00% perl
  780 root             1  20    0   161M 68412K select  4   0:08   0.00% smbd
 2086 vscan            1  20    0   160M 82756K select  7   0:04   0.00% perl
  595 bind            15  52    0   130M 75632K sigwai 11   4:40   0.04% named
  952 root             2  20    0   125M 27956K select  4   0:01   0.00% smbd
  953 root             1  20    0   119M 27372K select  3   0:00   0.00% smbd
 2263 root            15  20    0 92164K 52140K select  6  15:25   0.37% python2.7
  641 root             1  52    0 90668K  1900K select  5   0:00   0.00% mountd
  647 root             1  20    0 90560K  1924K select  3   0:00   0.00% nfsd
 2291 root             2  20    0 84304K  5444K kqread 11   0:00   0.00% cupsd
  688 transmission     4  20    0 48980K 34436K kqread 11  77:15   0.15% transmission-daemon
15187 root             1  20    0 48336K 23580K select  5   0:00   0.00% cnid_dbd
 2079 freeradius       6  20    0 40588K 10100K kqread 10   0:01   0.00% radiusd
  776 root             1  20    0 29004K  5512K select  4   0:04   0.01% nmbd
 2303 colord           3  32    0 28636K  9432K select  2   0:00   0.00% colord
 2080 postgrey         1  20    0 27924K  9908K select  2   0:00   0.00% perl
 2251 root             1  20    0 27824K  9312K select 11   0:01   0.00% afpd
15227 nobody           1  20    0 26372K 20088K sbwait  8   0:00   0.00% perl
15223 postfix          1  20    0 22916K 11020K kqread 10   0:00   0.00% smtpd
15225 postfix          1  20    0 20516K  9272K kqread  8   0:00   0.00% trivial-rewrite
  661 root             3  20    0 19376K 10432K nanslp  1   0:32   0.00% monit
 2223 root             2  20    0 19148K  5408K kqread  2   0:08   0.00% netatalk
 2252 root             1  20    0 18436K  6676K select  9   0:00   0.00% cnid_metad
 2424 postfix          1  20    0 17008K  2152K kqread  6   0:00   0.00% tlsmgr
 2196 postfix          1  20    0 16916K  1896K kqread 11   0:01   0.00% qmgr
15214 postfix          1  20    0 16864K  8056K kqread 11   0:00   0.00% pickup
15226 root             1  21    0 16860K  8044K wait    4   0:00   0.00% spawn
 2194 root             1  20    0 16852K  1316K kqread  6   0:03   0.00% master
 2125 postfix          3  20    0 16404K  6768K select  8   0:02   0.00% opendkim
  672 root             2  20    0 15552K  5236K select  8   2:40   0.05% vmtoolsd
 2103 clamav           1  52    0 15172K  3780K pause   2   0:51   0.00% freshclam
 2117 postfix          3  20    0 15156K  5196K select  5   0:02   0.00% dk-filter
15217 donald           1  20    0 13276K  8456K select 10   0:00   0.01% sshd
15215 root             1  20    0 13276K  8380K select  8   0:00   0.00% sshd
 2340 root             1  20    0 12964K  5172K select  5   0:00   0.00% sshd
  746 root             1  20    0 12572K 12688K select 11   0:17   0.01% ntpd
 2124 postfix          1  52    0 10640K     0K wait   10   0:00   0.00% <opendkim>
  955 root             1  20    0 10552K  1696K nanslp  0   0:03   0.00% smartd
15220 root             1  20    0  9820K  5156K pause   6   0:00   0.00% csh
  412 root             1  20    0  9184K   720K select  3   0:15   0.01% devd
 2115 root             1  20    0  8240K  2156K kqread  8   0:00   0.00% config
15232 root             1  20    0  8196K  4196K CPU0    0   0:00   0.08% top
 2110 root             1  20    0  8176K  1476K kqread  2   0:00   0.00% dovecot
 2111 dovecot          1  20    0  7844K  1208K kqread  7   0:00   0.00% anvil
 2112 root             1  20    0  7844K  1008K kqread 11   0:00   0.00% log
 2218 avahi            1  20    0  7696K  3272K select 11   0:40   0.00% avahi-daemon
 2214 messagebus       1  24    0  7436K  2484K select  0   0:00   0.00% dbus-daemon
15219 donald           1  20    0  7216K  3296K wait    6   0:00   0.00% su
15218 donald           1  20    0  7152K  3224K wait   11   0:00   0.00% sh
  973 mysql            1  52    0  7072K     0K wait    2   0:00   0.00% <sh>
  631 root             1  20    0  6980K  1064K select  7   0:00   0.00% nfsuserd
  635 root             1  20    0  6980K  1048K select 11   0:00   0.00% nfsuserd
  636 root             1  20    0  6980K  1048K sbwait  0   0:00   0.00% nfsuserd
  633 root             1  20    0  6980K  1036K sbwait  2   0:00   0.00% nfsuserd
  630 root             1  52    0  6976K     0K pause  10   0:00   0.00% <nfsuserd>
  691 root             1  20    0  6904K  2200K select  2   0:00   0.00% in.tftpd
 2359 root             1  20    0  6748K  1324K nanslp  6   0:01   0.00% cron
  495 root             1  20    0  6704K  2432K select  0   3:20   0.11% syslogd
  504 root             1  20    0  6652K  2000K select  4   0:00   0.00% rpcbind

megapearl · Sep 22, 2018

I have homeassistant running on another virtual machine, which was logging every second changes from the different domotica entities to MySQL on FreeBSD, I limited that in the configuration of homeassistant and emptied the MySQL tables.
Swap usage is 0% now.

Thanks for your support,

Best regards,
Donald.

spork · Sep 22, 2018

Dang, don't think I've ever seen mysqld get this big: 60920M

megapearl · Sep 23, 2018

Swap usage is back to 34% within 1 day:

top

Code:

last pid:  7650;  load averages:  0.91,  0.69,  0.61                                                                                                                                 up 1+02:27:21  14:23:30
84 processes:  1 running, 83 sleeping
CPU:  3.6% user,  0.0% nice,  4.6% system,  0.0% interrupt, 91.7% idle
Mem: 91M Active, 613M Inact, 4730M Laundry, 56G Wired, 838M Free
ARC: 51G Total, 2717M MFU, 46G MRU, 23M Anon, 220M Header, 1819M Other
     46G Compressed, 52G Uncompressed, 1.13:1 Ratio
Swap: 8192M Total, 2836M Used, 5356M Free, 34% Inuse

  PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
2061 mysql           33  20    0 60864M  2962M select  1   0:47   0.04% mysqld

Why is MySQLd consuming that much of memory?
Total directory is:

Code:

root@fileserver:/var/db/mysql # du -h
470K    ./biertje
10M    ./pydio
687K    ./mysql
9.0M    ./monit
267K    ./performance_schema
177K    ./postfix
13M    ./donald
189K    ./phpmyadmin
4.3M    ./radius
1.9M    ./homeassistant
2.7G    .
root@fileserver:/var/db/mysql # ls -l
total 2768615
-rw-rw----  1 mysql  mysql          56 Dec  4  2013 auto.cnf
drwx------  2 mysql  mysql           5 Oct 21  2017 biertje
drwx------  2 mysql  mysql          83 Dec 20  2017 donald
drwx------  2 mysql  mysql          11 Sep 22 11:17 homeassistant
-rw-rw----  1 mysql  mysql   536870912 Sep 23 14:25 ib_logfile0
-rw-rw----  1 mysql  mysql   536870912 Sep 22 05:12 ib_logfile1
-rw-rw----  1 mysql  mysql   312475648 Sep 23 14:25 ibdata1
drwx------  2 mysql  mysql          97 Apr  2  2017 monit
drwx------  2 mysql  mysql          87 Feb 21  2017 mysql
-rw-rw----  1 mysql  mysql           6 Feb 21  2017 mysql_upgrade_info
-rw-rw----  1 mysql  mysql   628428542 Sep  9 11:27 mysql-bin.000801
-rw-rw----  1 mysql  mysql  1073742493 Sep 10 10:26 mysql-bin.000802
-rw-rw----  1 mysql  mysql   460010747 Sep 10 23:28 mysql-bin.000803
-rw-rw----  1 mysql  mysql    27151427 Sep 11 00:11 mysql-bin.000804
-rw-rw----  1 mysql  mysql    12321747 Sep 11 00:28 mysql-bin.000805
-rw-rw----  1 mysql  mysql   651563087 Sep 11 17:59 mysql-bin.000806
-rw-rw----  1 mysql  mysql         143 Sep 11 18:37 mysql-bin.000807
-rw-rw----  1 mysql  mysql  1073742475 Sep 13 01:10 mysql-bin.000808
-rw-rw----  1 mysql  mysql  1073742735 Sep 14 12:15 mysql-bin.000809
-rw-rw----  1 mysql  mysql  1073741889 Sep 15 16:08 mysql-bin.000810
-rw-rw----  1 mysql  mysql   336815678 Sep 16 00:50 mysql-bin.000811
-rw-rw----  1 mysql  mysql  1073741914 Sep 17 04:40 mysql-bin.000812
-rw-rw----  1 mysql  mysql  1073743702 Sep 18 09:54 mysql-bin.000813
-rw-rw----  1 mysql  mysql  1073743603 Sep 19 13:03 mysql-bin.000814
-rw-rw----  1 mysql  mysql   105548751 Sep 19 16:04 mysql-bin.000815
-rw-rw----  1 mysql  mysql         120 Sep 19 16:13 mysql-bin.000816
-rw-rw----  1 mysql  mysql         143 Sep 19 16:17 mysql-bin.000817
-rw-rw----  1 mysql  mysql  1073741905 Sep 20 21:38 mysql-bin.000818
-rw-rw----  1 mysql  mysql  1073742208 Sep 22 01:21 mysql-bin.000819
-rw-rw----  1 mysql  mysql   355523624 Sep 22 11:18 mysql-bin.000820
-rw-rw----  1 mysql  mysql      833119 Sep 22 11:55 mysql-bin.000821
-rw-rw----  1 mysql  mysql     8910007 Sep 23 14:25 mysql-bin.000822
-rw-rw----  1 mysql  mysql         682 Sep 22 11:56 mysql-bin.index
drwx------  2 mysql  mysql          55 Feb 21  2017 performance_schema
drwx------  2 mysql  mysql          43 Dec  6  2015 phpmyadmin
drwx------  2 mysql  mysql          34 Sep 10 23:34 postfix
drwx------  2 mysql  mysql          46 May 16  2017 pydio
drwx------  2 mysql  mysql          70 Apr 28  2015 radius
root@fileserver:/var/db/mysql #

SirDice · Sep 24, 2018

megapearl said:
Why is MySQLd consuming that much of memory?

Because it's not tuned correctly?

megapearl · Sep 24, 2018

I will look into that... it is out of the scope of FreeBSD and not related, so this topic can be closed.

Have runned this MySQL tuner years ago...

Code:

root@fileserver:/home/donald # ./mysqltuner.pl
 >>  MySQLTuner 1.7.12 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
Please enter your MySQL administrative login: donald
Please enter your MySQL administrative password: [OK] Currently running supported MySQL version 5.6.41-log
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: /var/log/mysql/mysql-error.log(32M)
[OK] Log file /var/log/mysql/mysql-error.log exists
[OK] Log file /var/log/mysql/mysql-error.log is readable.
[OK] Log file /var/log/mysql/mysql-error.log is not empty
[!!] Log file /var/log/mysql/mysql-error.log is bigger than 32 Mb
[!!] /var/log/mysql/mysql-error.log contains 50489 warning(s).
[!!] /var/log/mysql/mysql-error.log contains 703 error(s).
[--] 505 start(s) detected in /var/log/mysql/mysql-error.log
[--] 1) 2018-09-23 20:16:44 12697 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 2) 2018-09-23 20:14:19 11551 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 3) 2018-09-22 11:56:53 2061 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 4) 2018-09-22 11:20:06 16378 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 5) 2018-09-19 16:52:44 2053 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 6) 2018-09-19 16:13:50 2043 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 7) 2018-09-19 16:04:17 2058 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 8) 2018-09-16 00:52:00 2057 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 9) 2018-09-11 18:46:59 2054 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 10) 2018-09-11 18:22:42 2069 [Note] /usr/local/libexec/mysqld: ready for connections.
[--] 291 shutdown(s) detected in /var/log/mysql/mysql-error.log
[--] 1) 2018-09-23 20:16:26 11551 [Note] /usr/local/libexec/mysqld: Shutdown complete
[--] 2) 2018-09-23 20:11:52 2061 [Note] /usr/local/libexec/mysqld: Shutdown complete
[--] 3) 2018-09-22 11:55:40 16378 [Note] /usr/local/libexec/mysqld: Shutdown complete
[--] 4) 2018-09-22 11:19:50 2053 [Note] /usr/local/libexec/mysqld: Shutdown complete
[--] 5) 2018-09-19 16:17:28 2043 [Note] /usr/local/libexec/mysqld: Shutdown complete
[--] 6) 2018-09-11 18:37:10 2069 [Note] /usr/local/libexec/mysqld: Shutdown complete
[--] 7) 2018-09-11 00:11:07 2059 [Note] /usr/local/libexec/mysqld: Shutdown complete
[--] 8) 2018-09-10 23:28:50 2069 [Note] /usr/local/libexec/mysqld: Shutdown complete
[--] 9) 2018-09-03 19:00:33 2039 [Note] /usr/local/libexec/mysqld: Shutdown complete
[--] 10) 2018-09-02 22:28:34 1913 [Note] /usr/local/libexec/mysqld: Shutdown complete

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in InnoDB tables: 123.0M (Tables: 180)
[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] User 'zoneminder@%' does not specify hostname restrictions.
[!!] There is no basic password file list!

-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 20h 20m 56s (14K q [0.196 qps], 413 conn, TX: 412K, RX: 5M)
[--] Reads / Writes: 17% / 83%
[--] Binary logging is enabled (GTID MODE: OFF)
[--] Physical Memory     : 64.0G
[--] Max MySQL memory    : 55.2G
[--] Other process memory: 7.1G
[--] Total buffers: 54.1G global + 1.1M per thread (500 max threads)
[--] P_S Max memory usage: 569M
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 54.6G (85.41% of installed RAM)
[!!] Maximum possible memory usage: 55.2G (86.25% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (18/14K)
[OK] Highest usage of available connections: 2% (11/500)
[!!] Aborted connections: 19.61%  (81/413)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 112 sorts)
[OK] No joins without indexes
[OK] Temporary tables created on disk: 0% (0 on disk / 6 total)
[OK] Thread cache hit rate: 97% (11 created / 413 connections)
[OK] Table cache hit rate: 91% (76 open / 83 opened)
[OK] Open file limit used: 0% (19/1M)
[OK] Table locks acquired immediately: 100% (7K immediate / 7K locks)
[OK] Binlog cache memory access: 100.00% (3251 Memory / 3251 Total)

-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 569.7M
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.2% (6M used / 33M cache)
[OK] Key buffer size / total MyISAM indexes: 32.0M/99.0K

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 54.0G/123.0M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (1.85185185185185 %): 512.0M * 2/54.0G should be equal 25%
[!!] InnoDB buffer pool instances: 8
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 99.37% (117020 hits/ 117756 total)
[!!] InnoDB Write Log efficiency: 66.63% (11214 hits/ 16829 total)
[OK] InnoDB log waits: 0.00% (0 waits / 5615 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    /var/log/mysql/mysql-error.log is > 32Mb, you should analyze why or implement a rotation log strategy such as logrotate!
    Control warning line(s) into /var/log/mysql/mysql-error.log file
    Control error line(s) into /var/log/mysql/mysql-error.log file
    Restrict Host for user@% to user@SpecificDNSorIp
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Reduce or eliminate unclosed connections and network issues
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    Consider installing Sys schema from https://github.com/mysql/mysql-sys
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: http://bit.ly/2wgkDvS
Variables to adjust:
    innodb_log_file_size should be (=6G) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
    innodb_buffer_pool_instances(=54)
root@fileserver:/home/donald #

SirDice · Sep 24, 2018

spork said:
Dang, don't think I've ever seen mysqld get this big: 60920M

That's not even close to what I regularly deal with:

Code:

 2568 mysql        50  20    0   213G   197G select 16 156.2H   0.04% mysqld

Note that this machine has 380GB of RAM.

SirDice · Sep 24, 2018

megapearl said:
Have runned this MySQL tuner years ago...

Loads change over time, so you need to regularly tune it.

megapearl · Sep 24, 2018

SirDice said:
Loads change over time, so you need to regularly tune it.

Will do! thanks for your time.

ZFS High Swap Usage, why? Where to start?

megapearl

SirDice

Administrator

MarcoB

spork

megapearl

megapearl

megapearl

spork

megapearl

SirDice

Administrator

megapearl

SirDice

Administrator

SirDice

Administrator

megapearl