Sorry if you consider my post OTG.
I decided to write despite using Proxmox version 8.2.4 version I have the exact same problem.
IOPS degradation by 2000% write on one of the arrays (raid5sas) on LSI SAS2008 controller to about 20-30MB/s after filling the ARC buffer.
Reading from the same array is correct >400MB/s I did not measure it exactly.
ARC cache set to:
c 4 67483668480
c_min 4 4217729280
c_max 4 67483668480
128GB RAM.
I do not think that ARC settings have anything to do with it because the second SATA array on the same hardware works correctly for both write and read.
root@pve2:/raid5sas/test/temp# zpool status
pool: raid5low
state: ONLINE
scan: scrub repaired 0B in 02:56:37 with 0 errors on Sun Jul 14 03:20:39 2024
config:
NAME STATE READ WRITE CKSUM
raid5low ONLINE 0 0 0
sdg ONLINE 0 0 0
sdi ONLINE 0 0 0
sdk ONLINE 0 0 0
sdh ONLINE 0 0 0
sdj ONLINE 0 0 0
sdf ONLINE 0 0 0
errors: No known data errors
pool: raid5sas
state: ONLINE
scan: scrub repaired 0B in 01:52:42 with 0 errors on Sun Aug 4 07:11:04 2024
config:
NAME STATE READ WRITE CKSUM
raid5sas ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
wwn-0x5000c500587b5f5f ONLINE 0 0 0
wwn-0x5000c5008591cf77 ONLINE 0 0 0
wwn-0x5000c5008591a4cb ONLINE 0 0 0
wwn-0x5000c50085880b2f ONLINE 0 0 0
errors: No known data errors
root@pve2:/raid5sas/test/temp#
That's more or less how it looks for me.
The history of this raid5sas array is that it was created under the Rocky8 ~=RHEL8 system.
Then the controller and disks were moved to newer hardware for the Proxmox system.
And here, after the first power-up, it did not detect the pool, so I imported it.
During import, it shouted that /dev/sdf /dev/sdg /dev/sdh /dev/sdi was busy (the names of the disks in the system had changed), so according to the guide, I gave the -f option indicating which disks to import, i.e. /dev/sda /dev/sdb /dev/sdc /dev/sdd.
After importing, it wrote that the array was OK and the disks were displayed using WWN numbering.
Apart from that, there was nothing unusual.
Scrub did not detect anything.
Did
mack3457 manage to solve the problem? How?