zfs waits for [zio->io_cv] or [zfs]

ers

ers

What could cause freeze of zfs (done as raidz1 on 4 drives) when files >=70MB are copied?
Small files are processed smoothly but wen processing big one only part of it is writen and zfs stops.
Everything then works except just zfs subsystem.

^T gives inforamtion that zfs is waiting for zio->io_cv (some times for [zfs]).
Code: 
load: 0.02  cmd: ls 1316 [zfs] 143.74r 0.00u 0.00s 0% 1616k
load: 0.02  cmd: ls 1316 [zfs] 144.43r 0.00u 0.00s 0% 1616k
Usually no messages, no panics... just silent freeze.
Sometimes timeout on random channel
Code: 
ahcich6: Timeout on slot 15
ahcich6: is 00000000 cs 00060000 ss 00078000 rs 00078000 tfd 50010040 serr 00000000

System specification > here <.
Yes I know that 1GB of ram is not much but there is no more to install.
vm.kmem_size* set to 512M, prefetch disabled - the rest is default.
The same controller with disks in UFS works with no problems.

"Search" on forum do not gave stright answer.
"Google" says a lot about zio->io_cv as "slow down" but not "freeze".

How to tweak zfs to run stable on this machine and why this and not other tweaks?
 
Sorry for post under post but i have no eddit option.

New inforamtion on topic:
- zfs hans not only raidz1 but on mirror too
- sometimes in mesages is:
Code: 
ahcich7: Timeout on slot 2
ahcich7: is 00000000 cs 00000020 ss 0000003c rs 0000003c tfd 50040040 serr 00000000
- tweaking with timeout, pendigwrites do not change anything.
- vm.kmeme* size change do not solve the problem

Why ZFS hangs?
What to do to make it _stable_ ?
Please anybody help...
 
Well not sure if it is related, but you shouldn't have any timeouts on your SATA slots. Check your disks' SMART information to see if they list UDMA CRC Error Count; if non-zero this will indicate cabling problems. Also you may check for Current Pending Sector.

4GiB RAM should be considered minimum with ZFS, and starting from 8GiB performance should be alot better. 4GiB RAM would still have prefetching disabled by default. Also, i assume your system is 64-bit?
 
You probably do not read previous messages carefully... :D

If you red specification i linked you will have know that this system is i386 (32bit),
with 8.1-release and only 1GB of RAM (hardware max allowed).

Disks working great when they are using UFS and there is no problem with them.
No timeouts, ets. Problems start just when i try to use zfs.
Message saying about drive timeouts are *because zfs do not respond*.
Zfs is between drive itself and system. When zfs layer stop system reports timeouts...

ZFS sould be stable and only slow down when there is not enought of ram,
but should not hangs... (correct me if i am wrong)

There should be solution to this mystery...
 
Galactic_Dominator said:
@ersm, you are correct ZFS should be stable under that environment.
Have you followed the guidelines for tuning ZFS under i386 and low mem systems?
Click to expand...

Sorry, but i do not found aprioprate guidline (i am a bit fresh in this system).
Could you guide me in the right direction? (link?)

AndyUKG said:
This could be the same bug I encounter with the siis driver, the dev said the same bug exists in the ahci driver:
http://forums.freebsd.org/showthread.php?t=17432
ta Andy.
Click to expand...

This is very interesting.
I have read whole thread and thread linked inside about patch (with ahci conjecture).
The question is: could AHCI driver be a problem?
I suspect that this could be in this case... :/

AndyUKG said:
PS are you using SATA port multipliers? The bug specifically relates to problems with PM configs...
Click to expand...

No i do not use port multiplier (or i do not know about it - in case it is build in),
but acording to the upper link(s) there is a similarity
in marvel controller (FIS-based switching) so it could be related.
Look here: http://lists.freebsd.org/pipermail/freebsd-fs/2010-September/009440.html
 
Galactic_Dominator said:
http://wiki.freebsd.org/ZFSTuningGuide

I would follow what AndyUKG has pointed out though as that applies to your config. Perhaps it's been fixed in STABLE, you could try upgrading to it and see.
Click to expand...

Thank you. or now i took settings from AndyUKG post but they are the same - like in guide you sent.
I have also tried before 8.1-stable 201010 and also 9.0-current 201010 but there was no difference.
As a last resort i will try the patch pointed by AndyUKG thrrread.
Now i am doing some test to catch where the problem lies (in ahci drivers, zfs itself, zfs tuning)
 
New information in this topic

I think that the problem is AHCI driver loaded by
Code: 
ahci_load="YES"
On this board there are 2 controllers: VIA VT8251 and Marvell 88SE6145.
Unfortunatelly Marvell is recognized as SX not SE...

First one (via) is seen under default ahci controller (marvell do not work properly this way),
and everything is working well and is stable.
UFS and ZFS were tested for over an hour and >180GB (parallel reading and writing)

When new ahci driver is loaded (using via) again everything is ok.

When new ahci driver is loaded (for marvell), zfs hangs, waits for zio when doing something with size greater than 50MB.
Simple "ls" can cause hang... Hangs are secially seen with parallel reading during write.
When UFS was used the same thing has happened but filesize sould be much bigger >1GB.
(? ram requirement, queue length, buffers ?)

All this proves taht cabling and zfs subsystem can work stable under i386 with 1GB of RAM.
This also proves that disks are ok and working fine. (all hardware are brand new)
ZFS was tunned for low-mem system:
Code: 
vm.kmem_size_max="330M"
vm.kmem_size="330M"
vfs.zfs.arc_max="40M"
vfs.zfs.vdev.cache.size="5M"
Tested on 8.1-release.

To sum up:
most probably there is a bug in new AHCI driver...

Next tests:
- 8.1-stable
- 9.0-current
- driver patch sugested by AndyUKG
 
Hi,

I just asked for some info on which releases have the bug fix I mentioned included from the developer (MAV), and they are (or will be). 8-STABLE, and 8.2-RELEASE.
Patched drivers are, AHCI, SIIS and MVS.

cheers Andy.
 
This problem was solved by disabling NCQ on Marvell 88se6145 controller.
To patch this you should recompile kernel with newest ahci.c.
Originaly solved in revision 218596 by mav@ (use changes in revision 218605).
After aplying patch all filesystems (ZFS, UFS) are stable even on heavy load with multiple threads.
All needed information are available in this thread http://forums.freebsd.org/showthread.php?t=20412
 
You must log in or register to reply here.
Back
Top