zfs waits for [zio->io_cv] or [zfs]

ers · #1 November 8th, 2010, 18:08

What could cause freeze of zfs (done as raidz1 on 4 drives) when files >=70MB are copied?
Small files are processed smoothly but wen processing big one only part of it is writen and zfs stops.
Everything then works except just zfs subsystem.

^T gives inforamtion that zfs is waiting for zio->io_cv (some times for [zfs]).

Code:

load: 0.02  cmd: ls 1316 [zfs] 143.74r 0.00u 0.00s 0% 1616k
load: 0.02  cmd: ls 1316 [zfs] 144.43r 0.00u 0.00s 0% 1616k

Usually no messages, no panics... just silent freeze.
Sometimes timeout on random channel

Code:

ahcich6: Timeout on slot 15
ahcich6: is 00000000 cs 00060000 ss 00078000 rs 00078000 tfd 50010040 serr 00000000

System specification > here <.
Yes I know that 1GB of ram is not much but there is no more to install.
vm.kmem_size* set to 512M, prefetch disabled - the rest is default.
The same controller with disks in UFS works with no problems.

"Search" on forum do not gave stright answer.
"Google" says a lot about zio->io_cv as "slow down" but not "freeze".

How to tweak zfs to run stable on this machine and why this and not other tweaks?

ers · #2 November 11th, 2010, 14:59

Sorry for post under post but i have no eddit option.

New inforamtion on topic:
- zfs hans not only raidz1 but on mirror too
- sometimes in mesages is:

Code:

ahcich7: Timeout on slot 2
ahcich7: is 00000000 cs 00000020 ss 0000003c rs 0000003c tfd 50040040 serr 00000000

- tweaking with timeout, pendigwrites do not change anything.
- vm.kmeme* size change do not solve the problem

Why ZFS hangs?
What to do to make it _stable_ ?
Please anybody help...

sub_mesa · #3 November 11th, 2010, 18:24

Well not sure if it is related, but you shouldn't have any timeouts on your SATA slots. Check your disks' SMART information to see if they list UDMA CRC Error Count; if non-zero this will indicate cabling problems. Also you may check for Current Pending Sector.

4GiB RAM should be considered minimum with ZFS, and starting from 8GiB performance should be alot better. 4GiB RAM would still have prefetching disabled by default. Also, i assume your system is 64-bit?

ers · #4 November 11th, 2010, 19:24

You probably do not read previous messages carefully...

If you red specification i linked you will have know that this system is i386 (32bit),
with 8.1-release and only 1GB of RAM (hardware max allowed).

Disks working great when they are using UFS and there is no problem with them.
No timeouts, ets. Problems start just when i try to use zfs.
Message saying about drive timeouts are *because zfs do not respond*.
Zfs is between drive itself and system. When zfs layer stop system reports timeouts...

ZFS sould be stable and only slow down when there is not enought of ram,
but should not hangs... (correct me if i am wrong)

There should be solution to this mystery...

Galactic_Dominator · #5 November 11th, 2010, 19:36

@ersm, you are correct ZFS should be stable under that environment.

Have you followed the guidelines for tuning ZFS under i386 and low mem systems?

AndyUKG · #6 November 11th, 2010, 19:55

This could be the same bug I encounter with the siis driver, the dev said the same bug exists in the ahci driver:

http://forums.freebsd.org/showthread.php?t=17432

ta Andy.

AndyUKG · #7 November 11th, 2010, 19:57

PS are you using SATA port multipliers? The bug specifically relates to problems with PM configs...

ers · #8 November 12th, 2010, 02:18

Quote:

Originally Posted by Galactic_Dominator

@ersm, you are correct ZFS should be stable under that environment.
Have you followed the guidelines for tuning ZFS under i386 and low mem systems?

Sorry, but i do not found aprioprate guidline (i am a bit fresh in this system).
Could you guide me in the right direction? (link?)

Quote:

Originally Posted by AndyUKG

This could be the same bug I encounter with the siis driver, the dev said the same bug exists in the ahci driver:
http://forums.freebsd.org/showthread.php?t=17432
ta Andy.

This is very interesting.
I have read whole thread and thread linked inside about patch (with ahci conjecture).
The question is: could AHCI driver be a problem?
I suspect that this could be in this case... :/

Quote:

Originally Posted by AndyUKG

PS are you using SATA port multipliers? The bug specifically relates to problems with PM configs...

No i do not use port multiplier (or i do not know about it - in case it is build in),
but acording to the upper link(s) there is a similarity
in marvel controller (FIS-based switching) so it could be related.
Look here: http://lists.freebsd.org/pipermail/f...er/009440.html

Galactic_Dominator · #9 November 12th, 2010, 02:44

http://wiki.freebsd.org/ZFSTuningGuide

I would follow what AndyUKG has pointed out though as that applies to your config. Perhaps it's been fixed in STABLE, you could try upgrading to it and see.

ers · #10 November 12th, 2010, 03:49

Quote:

Originally Posted by Galactic_Dominator

http://wiki.freebsd.org/ZFSTuningGuide

I would follow what AndyUKG has pointed out though as that applies to your config. Perhaps it's been fixed in STABLE, you could try upgrading to it and see.

Thank you. or now i took settings from AndyUKG post but they are the same - like in guide you sent.
I have also tried before 8.1-stable 201010 and also 9.0-current 201010 but there was no difference.
As a last resort i will try the patch pointed by AndyUKG thrrread.
Now i am doing some test to catch where the problem lies (in ahci drivers, zfs itself, zfs tuning)

ers · #11 November 12th, 2010, 15:44

New information in this topic

I think that the problem is AHCI driver loaded by

Code:

ahci_load="YES"

On this board there are 2 controllers: VIA VT8251 and Marvell 88SE6145.
Unfortunatelly Marvell is recognized as SX not SE...

First one (via) is seen under default ahci controller (marvell do not work properly this way),
and everything is working well and is stable.
UFS and ZFS were tested for over an hour and >180GB (parallel reading and writing)

When new ahci driver is loaded (using via) again everything is ok.

When new ahci driver is loaded (for marvell), zfs hangs, waits for zio when doing something with size greater than 50MB.
Simple "ls" can cause hang... Hangs are secially seen with parallel reading during write.
When UFS was used the same thing has happened but filesize sould be much bigger >1GB.
(? ram requirement, queue length, buffers ?)

All this proves taht cabling and zfs subsystem can work stable under i386 with 1GB of RAM.
This also proves that disks are ok and working fine. (all hardware are brand new)
ZFS was tunned for low-mem system:

Code:

vm.kmem_size_max="330M"
vm.kmem_size="330M"
vfs.zfs.arc_max="40M"
vfs.zfs.vdev.cache.size="5M"

Tested on 8.1-release.

To sum up:
most probably there is a bug in new AHCI driver...

Next tests:
- 8.1-stable
- 9.0-current
- driver patch sugested by AndyUKG

AndyUKG · #12 November 15th, 2010, 17:20

Hi,

I just asked for some info on which releases have the bug fix I mentioned included from the developer (MAV), and they are (or will be). 8-STABLE, and 8.2-RELEASE.
Patched drivers are, AHCI, SIIS and MVS.

cheers Andy.

ers · #13 February 18th, 2011, 20:28

This problem was solved by disabling NCQ on Marvell 88se6145 controller.
To patch this you should recompile kernel with newest ahci.c.
Originaly solved in revision 218596 by mav@ (use changes in revision 218605).
After aplying patch all filesystems (ZFS, UFS) are stable even on heavy load with multiple threads.
All needed information are available in this thread http://forums.freebsd.org/showthread.php?t=20412

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
ZFS (V15): zfs userspace -> Operation not supported	interfasys	General	0	August 18th, 2010 02:00
[Solved] New to ZFS	feralape	General	3	March 25th, 2010 19:41
[ZFS] Combine striping and ZFS to use pools more efficient	bugboy	General	3	February 6th, 2010 00:46
Zfs	SuperMiguel	General	16	June 5th, 2009 19:29
zfs - to be or not to be	graudeejs	Off-Topic	47	May 26th, 2009 09:03