13.2-RC5 zfs arc size drop

I

ivosevb

There is already thread on Truenas forum about the problem with ARC size dropping in OpenZFS 2.1.9, which is in the 13.2. Problem is easily reproducible, here is the graph from my test Bhyve server. At 12:15h ARC was completely flushed after

zfs send -R zroot/vm/test@now > test.raw

In /boot/loader.conf i have vfs.zfs.arc.max="32G".

There is a very simple patch, but it will be in the OpenZFS 2.1.10.

Truenas thread: "Odd ARC Memory Usage Behavior in 13.0-U4"
 

Attachments

  • arc_problem.png
    arc_problem.png
    47.1 KB · Views: 89
Not yet observed here, mine is mostly between 16 and 24G (installed: 48G, arc_max: 25G), and I'm running on BETA2 for quite a while now.

But I have observed occasional strange and sharp changes in arc size steering since about when I gave up doing the steering myself by hacking the code (that was around 2012).

What I do observe is strange changes in db query times, like this:
date
secsrows
"2023-03-30 05:18:49.500334+02"7326500000
"2023-03-29 22:40:21.089673+02"1043500000
"2023-03-29 04:17:35.652673+02"3373500000
"2023-03-28 22:27:41.638745+02"923500000
"2023-03-26 03:51:49.305956+02"1806500000
"2023-03-25 22:35:09.237251+01"939500000
"2023-03-25 03:30:19.895712+01"491500000
"2023-03-24 22:27:14.449436+01"419500000
"2023-03-24 03:17:39.447214+01"253500000

The database says the queries are identical, the difference apparently caused by different query-setup times for index scans. Whatever that means - might be seek times (database is on mechanical disks).
So with the database not showing any logical difference, the question would be if these indexes are currently in the ARC (or L2ARC - and if not, why they don't make it to there).

But then, I don't think this has changed in 13.2. It was weird already before, and also I did a lot of optimizing interim.

Also, this is a CoD machine with two unbalanced NUMA domains (16+32 GB), It's quite difficult to figure what the memory management actually does at some moment.
 
ivosevb said:
In /boot/loader.conf i have vfs.zfs.arc.max="32G".
Click to expand...
Not relevant for the topic itself, but specifying max arc with a string never works on my installation:

# sysctl vfs.zfs.arc.max="4G"
sysctl: invalid unsigned long "4G"

Using the value in bytes ( # sysctl vfs.zfs.arc.max=4294967296) works. I see the string notation all over the internet and I sometimes wonder and ask myself, if all these people realise that their arc configuration is broken.

EDIT: typo
 
Hm, never tried to use these suffixes myself. And having a look at the code, indeed, I don't find a trace of support for them. The only suffixes supported seem to be about temperatures :-/
 
steps said:
Not relevant for the topic itself, but specifying max arc with a string never works on my installation:

# sysctl vfs.zfs.arc.max="4G"
sysctl: invalid unsigned long "4G"

Using the value in bytes ( # sysctl vfs.zfs.arc.max=4294967296) works. I see the string notation all over the internet and I sometimes wonder and ask myself, if all these people realise that their arc configuration is broken.
Click to expand...
No. The abbreviation did work when this tunable was still in /boot/loader.conf. But as it is now dynamically changeable, sysctl wants a literal integer.
 
PMc said:
No. The abbreviation did work when this tunable was still in /boot/loader.conf. But as it is now dynamically changeable, sysctl wants a literal integer.
Click to expand...
Ah that's it. I never tried to put it into /boot/loader.conf. Tried it now and yes, it works with the suffix.
Now I can rest at peace and not think constantly about all these people going nuts.

EDIT: typo again...
 
I notice the same issue with 13.2-RC3. I have some servers used to backup other servers with rsync which now takes double the time to complete. With 13.1 the MFU was taking the most RAM and now MRU uses the most RAM. As the daily backups don't have many file changes the MFU is more useful than MRU.

Do you think is possible they add OpenZFS 2.1.10 when released with a 13.2-p1 ? Or we expect they add it only with 13.3 which it will take another year?
 
CyberCr33p said:
I notice the same issue with 13.2-RC3. I have some servers used to backup other servers with rsync which now takes double the time to complete. With 13.1 the MFU was taking the most RAM and now MRU uses the most RAM. As the daily backups don't have many file changes the MFU is more useful than MRU.

Do you think is possible they add OpenZFS 2.1.10 when released with a 13.2-p1 ? Or we expect they add it only with 13.3 which it will take another year?
Click to expand...

For the quarterly 2023Q1 openzfs port returns for me :
zfs --version
zfs-2.1.99-1
zfs-kmod-v2022082900-zfs_2d5622f5b
 
The patch in my previous post fixes the issue:

Code: 
ARC: 51G Total, 44G MFU, 3133M MRU, 768K Anon, 1564M Header, 2750M Other
     44G Compressed, 183G Uncompressed, 4.16:1 Ratio
 
I'm having a similar problem after upgrading my OS to 13.2-RELEASE.
After upgrading our mail server to 13.2, memory usage began to fluctuate up and down and then gradually decreased over time(MemoryUsage-month.png). As a result, the number of IMAP processes waiting for disk I/O gradually increased(Green graph in ImapProcesses-month.png), and finally the situation worsened to the point that the mail list display in the user's client software timed out.

Until 13.1R, I had the following settings in /etc/sysctl.conf
vfs.zfs.arc_max=27917287424
I've tried increasing this, but nothing changed.

When I rebooted (and delete arc_max setting) the server, the memory usage returned to normal, but when the number of users increased at the start of work, the Wired memory (which should be the ARC cache) intermittently became 0 repeatedly(MemoryUsage-4hours.png) and I/O waiting IMAP processes became noticeable(ImapProcess-4hours.png). In addition, the usage amount looks like gradually decreasing, and it seems that the same situation is progressing.

Also, when I ran the 'zfs-stats -M' command just before rebooting the server, 'Gap' was negative, is this normal?
# zfs-stats -M

------------------------------------------------------------------------
ZFS Subsystem Report Tue Jul 11 20:03:32 2023
------------------------------------------------------------------------

System Memory:

4.69% 1.44 GiB Active, 45.37% 13.93 GiB Inact
49.82% 15.30 GiB Wired, 0.00% 0 Bytes Cache
2.81% 883.23 MiB Free, -2.68% -884084736 Bytes Gap

Real Installed: 32.00 GiB
Real Available: 98.69% 31.58 GiB
Real Managed: 97.25% 30.71 GiB

Logical Total: 32.00 GiB
Logical Used: 53.76% 17.20 GiB
Logical Free: 46.24% 14.80 GiB

Kernel Memory: 1.12 GiB
Data: 96.69% 1.08 GiB
Text: 3.31% 37.87 MiB

Kernel Memory Map: 30.71 GiB
Size: 44.85% 13.77 GiB
Free: 55.15% 16.94 GiB

------------------------------------------------------------------------

I think something is wrong with handling ARC cache in ZFS in 13.2R.
Please let me know if you have any hints on how to resolve it or any further information needed.
 

Attachments

  • MemoryUsage-month.png
    MemoryUsage-month.png
    49.5 KB · Views: 58
  • ImapProcesses-month.png
    ImapProcesses-month.png
    45.6 KB · Views: 57
  • MemoryUsage-4hours.png
    MemoryUsage-4hours.png
    40.5 KB · Views: 53
  • ImapProcesses-4hours.png
    ImapProcesses-4hours.png
    34.6 KB · Views: 53
That's the bug in OpenZFS v2.1.9 which 13.2-RELEASE uses. For our production storages we considered three options: to stay on 13.1-RELEASE, 13.2-RELEASE with patch or to go with 13.2-STABLE, and in the end we choose the third one. In 13.2-STABLE OpenZFS is v2.1.12 right now.
For our production web/mail servers we'll stay on 13.1-RELEASE for now.
 
ivosevb said:
That's the bug in OpenZFS v2.1.9 which 13.2-RELEASE uses. For our production storages we considered three options: to stay on 13.1-RELEASE, 13.2-RELEASE with patch or to go with 13.2-STABLE, and in the end we choose the third one. In 13.2-STABLE OpenZFS is v2.1.12 right now.
For our production web/mail servers we'll stay on 13.1-RELEASE for now.
Click to expand...
Thank you for your information!

Fortunately, there doesn't seem to be any catastrophic failures such as file corruption, so I'll try to survive by rebooting every week until the patch release.

I believe the patch will be released by 7/31, the EoL date for 13.1R ;).
 
You must log in or register to reply here.
Back
Top