• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

ZFS performance degradation over time

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,038
Messages: 3,826

#51
Not sure if this has been mentioned yet, but lighttpd with sendfile support enabled leads to horrible performance with ZFS.

And rtorrent with pre-allocate space enabled will drag down a ZFS pool.

Disabvle those two options (requires re-compiling lighttpd) and see how things go.
 

Matty

Active Member

Thanks: 9
Messages: 168

#52
phoenix said:
And rtorrent with pre-allocate space enabled will drag down a ZFS pool.
How does zfs handel torrents anyway? because I got poor read performance when reading from a torrent files compared to a file that was uploaded by eq samba?

will all the small bits of a torrent be spread all over the harddisk and is this causing the poorer performance?
 

wonslung

Aspiring Daemon

Thanks: 16
Messages: 850

#53
Matty said:
How does zfs handel torrents anyway? because I got poor read performance when reading from a torrent files compared to a file that was uploaded by eq samba?

will all the small bits of a torrent be spread all over the harddisk and is this causing the poorer performance?
It totally depends on hardware and settings.

When i was running on FreeBSD i NEVER had a problem with ZFS and rtorrent and i am a HEAVY user but i also had a lot of i/o to go around (12 7200 RPM drives )

now i'm on opensolaris and i get even better performance but i really miss a lot of the features in FreeBSD.....The only reason i switched was dedup....it was a really good feature for what i needed....

also, opensolaris allows me to run xen which is QUITE cool but i still use freebsd and zfs for a LOT of rtorrent + rutorrent seedboxes so i can ASSURE you if you set things up correctly you will not have problems.


also, zfs tends to cache up stuff in arc and flush to disk every 5-30 seconds so torrents are no problem in THAT manner.
 

xmanpsk

New Member

Thanks: 2
Messages: 2

#54
wonslung said:
you shouldnt' have to patch rtorrent.
It's QUITE simple to just set rtorrent to a maximum amount of memory with a line line this:

Code:
max_memory_usage = 1024M
No, that didn't help me. I had the same problem with Active memory as garrettmoore. Before patching I had 4-5GB Active memory with swapping (6GB total installed), after patching - only 250-300MB and great ZFS perfomance!

Here's instructions for rtorrent-0.8.6:
1). Make libtorrent backup (if something goes wrong):
pkg_create -b libtorrent-0.12.6
2). Extract libtorrent source:
cd /usr/ports/net-p2p/libtorrent
make clean extract
3). Open in your favourite editor file /usr/ports/net-p2p/libtorrent/work/libtorrent-0.12.6/src/data/memory_chunk.cc and find MemoryChunk::unmap() function definition.
4). Make changes:
Code:
  void
  MemoryChunk::unmap() {
    if (!is_valid())
      throw internal_error("MemoryChunk::unmap() called on an invalid object");

+   if (msync(m_ptr, m_end - m_ptr,MS_INVALIDATE) != 0)
+       throw internal_error("MemoryChunk::unmap() - msync() system call failed");
+
    if (munmap(m_ptr, m_end - m_ptr) != 0)
      throw internal_error("MemoryChunk::unmap() system call failed: " + std::string(rak::error_number::current().c_str()));
  }
5). Build and install edited libtorrent:
cd /usr/ports/net-p2p/libtorrent
make && make deinstall reinstall

Thanks Dorlas for the link.
 

tobiastheviking

New Member

Thanks: 1
Messages: 19

#55
I am getting the problem with "Active" memory, not from rtorrent(which i do have, but there is no correlation between usage and memory usage).

During testing i have also disabled rtorrent and sabnzbd+, and just done a cp from a UFS disk to a ZFS disk(or from ZFS to ZFS for that matter), and gotten the problem.

As it is, right now, both rtorrent and sabnzbd+ are running(on a UFS drive), and i had <100M Active. Then i started an rsync of some 5+ gb data. Active memory is now 1160M. Neither rtorrent or sabnzbd+ is actually doing anything right now, they are just running.
 

Matty

Active Member

Thanks: 9
Messages: 168

#56
tobiastheviking said:
I am getting the problem with "Active" memory, not from rtorrent(which i do have, but there is no correlation between usage and memory usage).

During testing i have also disabled rtorrent and sabnzbd+, and just done a cp from a UFS disk to a ZFS disk(or from ZFS to ZFS for that matter), and gotten the problem.

As it is, right now, both rtorrent and sabnzbd+ are running(on a UFS drive), and i had <100M Active. Then i started an rsync of some 5+ gb data. Active memory is now 1160M. Neither rtorrent or sabnzbd+ is actually doing anything right now, they are just running.
try http://forums.freebsd.org/showpost.php?p=63019&postcount=46
 

Matty

Active Member

Thanks: 9
Messages: 168

#58
tobiastheviking said:
Done and done.

Well, i currently have an uptime of 21 days, which is bound to be a record on this system.

I still have 1132M marked as Active, though i can't see what could possibly be using it. But for now, it looks usable.
Samba with sendfile and sharing a zfs filesystem? just a guess ;)
 

garrettmoore

New Member

Thanks: 3
Messages: 17

#59
I'm still having issues. I have 8GB of ram now and it's been good for a while, but I'm downloading a 52GB torrent and my write performance keeps going to hell. Running that perl command to flush memory works, but is both tedious and sloppy.

Has changing min/max vnodes been the fix for everyone else? When I tried lowering my maxvnodes to such a small amount my performance seemed to be much, much worse.
 

Matty

Active Member

Thanks: 9
Messages: 168

#60
garrettmoore said:
I'm still having issues. I have 8GB of ram now and it's been good for a while, but I'm downloading a 52GB torrent and my write performance keeps going to hell. Running that perl command to flush memory works, but is both tedious and sloppy.

Has changing min/max vnodes been the fix for everyone else? When I tried lowering my maxvnodes to such a small amount my performance seemed to be much, much worse.
have you tried another torrent client?
 

xmanpsk

New Member

Thanks: 2
Messages: 2

#61
garrettmoore said:
I'm still having issues. I have 8GB of ram now and it's been good for a while, but I'm downloading a 52GB torrent and my write performance keeps going to hell. Running that perl command to flush memory works, but is both tedious and sloppy.

Has changing min/max vnodes been the fix for everyone else? When I tried lowering my maxvnodes to such a small amount my performance seemed to be much, much worse.
Did you try rtorrent patch?
 

wonslung

Aspiring Daemon

Thanks: 16
Messages: 850

#62
xmanpsk said:
I can vouch for the patch. I was a doubter myself having never really noticed the issue (i was on a high ram system with a lot of fast drives) but when i installed to a 2 gb ram machine it was clearly evident, even on ufs, that rtorrent eventually fills all the ram.

This patch fixes it. I've mailed it to the maintainer of rtorrent, i think it should be included.
 

tobiastheviking

New Member

Thanks: 1
Messages: 19

#64
wonslung said:
I can vouch for the patch. I was a doubter myself having never really noticed the issue (i was on a high ram system with a lot of fast drives) but when i installed to a 2 gb ram machine it was clearly evident, even on ufs, that rtorrent eventually fills all the ram.

This patch fixes it. I've mailed it to the maintainer of rtorrent, i think it should be included.
Is there a bug report for this patch?
 

wonslung

Aspiring Daemon

Thanks: 16
Messages: 850

#65
i opened a bug report with rtorrent's people but thye insist the problem lies with FreeBSD's code, not rtorrent's.

The fact of the matter is, i do not know which is true, all i know is that when i use this patch, i do not notice any adverse effects to my rtorrent experience, only the benifet of my memory not being slowly stuck in ACTIVE.
 

Jago

Member

Thanks: 8
Messages: 91

#67
wonslung said:
i opened a bug report with rtorrent's people but thye insist the problem lies with FreeBSD's code, not rtorrent's.
The hilarious thing is that they insist that rtorrent is basically perfect and that it's just good at unconvering filesystem issues in different operating systems. This was from a developer comment to a single rtorrent bugreport confirmed by people using Linux, FreeBSD, MacOS X and OpenSolaris across multiple versions of reiserfs, ext2, ufs and zfs. The arrogance of rtorrent developers is mindboggling.
 

wonslung

Aspiring Daemon

Thanks: 16
Messages: 850

#68
Jago said:
The hilarious thing is that they insist that rtorrent is basically perfect and that it's just good at unconvering filesystem issues in different operating systems. This was from a developer comment to a single rtorrent bugreport confirmed by people using Linux, FreeBSD, MacOS X and OpenSolaris across multiple versions of reiserfs, ext2, ufs and zfs. The arrogance of rtorrent developers is mindboggling.
I know. While i love rtorrent and think it is the best torrent client out there currently, it has OBVIOUS issues on FreeBSD, not because of FreeBSD but because of how the kernel is different, yet whenever you bring these issues up (this memory issue is one instance, another REALLY good example is slow hashing) they blame it on FreeBSD.

I think it's more likely an issue due to poor understanding of the FreeBSD kernel, of course i am not a developer really so it's easy to sit down here and make judgements, in all honesty, i'm not qualified to make them, but what i CAN do, is see that this patch solves the problem, and with no degregation of rtorrent's torrent seeding/downloading ability.


I'd like to see a FreeBSD dev create a patch to make it hash faster on bsd though.
 

garrettmoore

New Member

Thanks: 3
Messages: 17

#69
I tried the patch supplied for libtorrent and I still have all of my memory eventually being marked as Inact when I download torrents. How can I check to make sure rtorrent is definitely using the version of libtorrent I modified and reinstalled?
 

wonslung

Aspiring Daemon

Thanks: 16
Messages: 850

#70
garrettmoore said:
I tried the patch supplied for libtorrent and I still have all of my memory eventually being marked as Inact when I download torrents. How can I check to make sure rtorrent is definitely using the version of libtorrent I modified and reinstalled?
the latest libtorrent from ports includes the patch
 

chrcol

Well-Known Member

Thanks: 15
Messages: 377

#73
I have been doing dozens of tests on a raid mirror 2 drive zfs setup along with someone else, adjusting many zfs variables.

conclusions so far on the setup.

optimal queue settings seems to be.
Code:
vfs.zfs.vdev.min_pending: 4
vfs.zfs.vdev.max_pending: 8
the default max 35 seems to flood the drive's whilst 1 is defenitly too low to get full speed and also causes the system to lag too much when the drives are under heavy load, however 1 would be a good setting if there is very little multitasking.

prefetch is a toughy, if its on it almost triples sequential read speed, however it increases latency noticebly and has a noticeable affect on system response when the disks are under stress.

compression, gzip default or gzip-9 both cause the server to reboot when stressing the drives, not sure if this is software or hardware yet. Also even under light use gzip causes system lag, using mkdir, ls etc. however the lzjb seems quite impressive and yields very good performance.

disabling data cache so metadata only really improves disk latency however throughput sharply drops.

reducing arc size only hurts performance in our case, however we havent dont much testing on this on yet.

disabling checksum gives about 10% on read speed and about 4% on write speed. this was done really just out of curiousity.

the hardware is 12 gig of ram and a quad core i7.
 

garrettmoore

New Member

Thanks: 3
Messages: 17

#74
Just as an update --

I am no longer running any torrent apps on my system. All I run now is a few python apps (SickBeard / CouchPotato / SABNZBD), Apache22 serving some static html files, and samba.

I am still having the same performance issues as originally brought up in this thread. Performance is good if I force flush everything out of memory, and then degrades again over time.

I wanted to check to see if anyone had any new suggestions, ideas, or had made some headway in figuring out some tuning options.
 

Sebulon

Aspiring Daemon

Thanks: 126
Messages: 706

#75
Hi,

I have personally had issues with WD Green drives- which seems to be one thing in common here, but never's been mentioned. My issues were never with performance, though I never used it primarily, only as a replicating target. Please show output of:
Code:
# zdb <pool>
Can't make any promises but it's worth ruling out. Also worth trying STABLE, now thats it's got ZFS V28. My sequentials, like scrubs and resilvers have gone up dramatically; x4. My primary pool is made up of 8 Samsung Spinpoint F3 raidz2, I have a Supermicro X7SBE, 8GB RAM and a core2 duo 2.13Ghz. I am now scrubbing at 300MB/s and can shuffle data over NFS async at 100MB/s and SAMBA async at 100MB/s

/boot/loader.conf
Code:
ahci_load="YES"
aio_load="YES"
zfs_load="YES"
Vfs.zfs.prefetch_disable=1
/usr/local/etc/smb.conf
Code:
socket options = TCP_NODELAY SO_SNDBUF=131072 SO_RCVBUF=131072
use sendfile = no
min receivefile size = 16384
aio read size = 16384
aio write size = 16384
aio write behind = yes
Awsome SAMBA tuning, in case anyone's interested.

/Sebulon