ZFS OS down with repeating swap_pager indefinite wait buffer messages on console

person3876

New Member


Messages: 4

Hello:

I have a FreeBSD 12.0 virtual machine on a KVM virtual private server. I installed from the 12.0-RELEASE installer ISO and chose encrypted ZFS root file system. I believe after that choice I accepted the default partitioning. So SWAP is part of the encrypted ZFS root disk. I'm using the generic kernel and basically running FAMP stack for Nextcloud.

Code:
# cat /etc/fstab
# Device        Mountpoint    FStype    Options        Dump    Pass#
/dev/vtbd0p3.eli        none    swap    sw        0    0
In a couple of months, three times now the machine has locked up, in the sense it didn't serve web pages and could not be contacted via SSH, though it would still respond to pings. In this error state I can look at the console from the provider's web VNC and I see repeating messages about swap_pager, something like:

Code:
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 16888849, size:4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 16867303 size: 8192
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 16888849, size:4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 16889145, size: 12288
(I stole that from someone else's post on the subject, but that's the idea. bufobj is always 0 and the others change)
My only option then is to use the KVM control panel to power off the virtual machine.

I saw someone mentioned the swap pager will wait 20 seconds before getting really upset, and although my VPS is optimized for amount of storage over speed of storage, I can't imagine it prevented a read or write from happening for that long.

Is there anything else that could cause this in my case?

Thanks.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,268
Messages: 29,700

Is there anything else that could cause this in my case?
It's most likely caused by MySQL. Make sure it's tuned correctly and doesn't try to use more memory than the machine actually has. General rule of thumb is to configure it to use a max of 50-75% of memory (RAM; exclude virtual memory) if there are other applications running on the machine (Apache, PHP, etc).

Don't use any of the my.cnf examples floating around on the internet. Measure and base your changes on your situation. I've found databases/mysqltuner quite useful but don't blindly follow all the advice it gives you. Again, measure, set, test, measure, set, test, etc.

You will also more than likely want to limit ARC by setting vfs.zfs.arc_max. You don't want MySQL and ARC battling for the same piece of the pie.
 
OP
OP
P

person3876

New Member


Messages: 4

Thanks SirDice. It saddens me a bit to learn that my very lightly-loaded (virtual) machine could be frozen with essentially default settings!

If you, or anyone, can spare another moment to help with mariadb tuning, I should have provided this additinal info originally:

Simultaneous users is generally no more than ONE.
Machine has 1GB of RAM.
I'm also running syncthing which seems to be using about 150MB of RAM.
With your advice, I just now I changed: vfs.zfs.arc_max: 625868800 -> 312934400
And I just slightly lowered innodb_buffer_pool_size=100M

my.conf contains only:

[server]
skip-name-resolve
slow-query-log = 1
slow-query-log-file = /var/log/mysql/slow.log
long_query_time = 1

[client]
default-character-set = utf8mb4

[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
innodb_file_per_table=1
innodb_buffer_pool_size=100M #<--- just added this
transaction_isolation = READ-COMMITTED


and if I dump mariadb GLOBAL VARIABLES and grep it for 'buffer' I get these (defaults?):


aria_pagecache_buffer_size 134217728
aria_sort_buffer_size 268434432
bulk_insert_buffer_size 8388608
innodb_buffer_pool_chunk_size 104857600
innodb_buffer_pool_dump_at_shutdown ON
innodb_buffer_pool_dump_now OFF
innodb_buffer_pool_dump_pct 25
innodb_buffer_pool_filename ib_buffer_pool
innodb_buffer_pool_instances 1
innodb_buffer_pool_load_abort OFF
innodb_buffer_pool_load_at_startup ON
innodb_buffer_pool_load_now OFF
innodb_buffer_pool_size 104857600
innodb_change_buffer_max_size 25
innodb_change_buffering all
innodb_log_buffer_size 16777216
innodb_sort_buffer_size 1048576
join_buffer_size 262144
join_buffer_space_limit 2097152
key_buffer_size 134217728
mrr_buffer_size 262144
myisam_sort_buffer_size 134216704
net_buffer_length 16384
optimizer_switch
preload_buffer_size 32768
read_buffer_size 131072
read_rnd_buffer_size 262144
sort_buffer_size 2097152
sql_buffer_result OFF



Anything stick out that I should lower there?

Thanks!
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,268
Messages: 29,700

Machine has 1GB of RAM.
You're really low on resources for ZFS and MariaDB. Both like memory. Any chance you could reinstall it with UFS instead of ZFS? That would definitely lower the memory requirements. I have a 2 core, 4GB VPS, it runs on ZFS with MariaDB and that runs fine but it's not that heavily loaded.

Regarding innodb_buffer_pool_size, ideally you want to size this according to the size of the database. Most optimal is when the whole database can fit inside it. But that's obviously not always possible. Never set it to more than the amount of RAM the machine has. Change the settings, restart MariaDB and leave it running for at least a day or two. Then run mysqltuner again.
 

kpa

Beastie's Twin

Reaction score: 1,801
Messages: 6,318

ZFS with 1GB can be made to work but you'll have to sacrifice performance quite a bit and you have to do some serious fine tuning with various tunables. The biggest problem is that ZFS ARC is wired memory that has to stay in physical memory, it can't be swapped out for obvious reasons. So with 1GB of physical memory you can't go over roughly 384MBs dedicated to ARC or you won't have enough memory for other kernel data structures and the actual applications you run.
 
OP
OP
P

person3876

New Member


Messages: 4

Making progress ... thank you! Here's how things are going, sorted by RES (partial). mysqltuner is very helpful.

Code:
last pid: 86449;  load averages:  0.18,  0.31,  0.31                                                up 0+00:21:01  07:29:48
37 processes:  1 running, 36 sleeping
CPU:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle
Mem: 224M Active, 185M Inact, 487M Wired, 64M Free
ARC: 271M Total, 187M MFU, 73M MRU, 32K Anon, 4367K Header, 7816K Other
     233M Compressed, 382M Uncompressed, 1.64:1 Ratio
Swap: 2048M Total, 2048M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
60719 www         201  82    9   248M   160M uwait    2   0:40   0.06% syncthing
60420 mysql        31  20    0   489M   106M select   0   0:34   0.04% mysqld
28481 www           1  22    0   223M    52M lockf    1   0:00   0.00% httpd
33809 www           1  20    0   218M    33M lockf    2   0:00   0.00% httpd
28835 www           1  20    0   218M    33M kqread   0   0:00   0.00% httpd
29647 www           1  52    0   218M    33M lockf    1   0:00   0.00% httpd
29363 www           1  52    0   218M    33M lockf    0   0:00   0.00% httpd
28936 www           1  52    0   218M    33M lockf    1   0:00   0.00% httpd
28202 root          1  20    0   218M    33M select   1   0:00   0.01% httpd
65128 root          5  20    0    40M    21M select   0   0:02   0.20% python3.6
97945 www          12  52    0   116M    19M uwait    1   0:01   0.00% syncthing
74848 ntpd          2  20    0    19M    19M select   1   0:00   0.01% ntpd
16610 _dnscrypt-    8  20    0   121M    10M uwait    0   0:00   0.00% dnscrypt-proxy
...some relevant entries in my.cnf:

Code:
innodb_buffer_pool_size=128M
innodb_log_file_size=16M
innodb_log_files_in_group=2
transaction_isolation = READ-COMMITTED
max_connections=41
key_buffer_size=33554432
join_buffer_size=393216
mysqltuner recommended the join_buffer_size increase but I'm not sure if it was necessary or helped because later it asked to raise it even more. Other suspicious recommendations (I ignorantly deem suspicious) were to increase tmp_table_size and max_heap_table_size higher than the 16M default.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,268
Messages: 29,700

Other suspicious recommendations (I ignorantly deem suspicious) were to increase tmp_table_size and max_heap_table_size higher than the 16M default.
Yeah, you have to strike a performance balance between memory and disk usage. Lower values are going to cause more temp tables being created on disk, which is obviously a little slower than having them created in memory. Because you're limited with memory you're more or less forced to have more temp tables on disk.
 
OP
OP
P

person3876

New Member


Messages: 4

Hello. I wanted to report back. Regarding RAM use, it was necessary to dial back max ARC size and tweaking my.cnf certainly helped some also. I think surges in RAM use by Syncthing was a triggering factor, and there is a setting for Synthing that can help trim RAM use (changing a garbage collection environment setting). In fact it was swapping probably quite a bit, and with my NEW settings it is not swapping at all.

But, I think my feeling that this message swap_pager: indefinite wait buffer: , along with a locked up machine, should not happen was correct because my VPS's host machine went offline due to disk problems! They did some repairs on the RAID but then decided to migrate my VM to another host until the issues were solved. So, I'm disappointed that I detected disk problems before my host did (I reported this problem to them and the response was to send me a screenshot showing that their RAID controller said the volume was 'optimal'); I was seeing IO service times over 1 second. But it maintains my confidence in FreeBSD!
 
Top