Performance problem FreeBSD9.0 + ZFS

C

cr4sh

Hello, I have performance problem on one of my production server. I have it now 3 days, and can't solve or find where the problem is.

Hardware: HP ML350 G5, Smart Array E200i with 2 x 64SAS disk, 6GB RAM

System: FreeBSD hydra 9.0-STABLE FreeBSD 9.0-STABLE #1: Mon Apr 9 19:15:53 CEST 2012 cr4sh@hydra:/sys/amd64/compile/Hydra amd64

/boot/loader.conf
Code: 
vfs.zfs.prefetch_disable=1
vfs.zfs.cache_flush_disable=1

[cmd=]zfs list[/cmd]:
Code: 
hydra# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
zroot                      58,6G  4,40G   976M  legacy
zroot/jails                54,1G  4,40G  1,78G  /usr/jails
zroot/jails/dns             416M  4,40G   691M  /usr/jails/dns
zroot/jails/jailbase        317M  4,40G   314M  /usr/jails/jailbase
zroot/jails/mta            32,7G  4,40G  32,4G  /usr/jails/mta
zroot/jails/mysql55         897M  4,40G   851M  /usr/jails/mysql55
zroot/jails/postgresql9     625M  4,40G   669M  /usr/jails/postgresql9
zroot/jails/samba          3,04G  4,40G  3,25G  /usr/jails/samba
zroot/jails/tomcat7        1,01G  4,40G  1,05G  /usr/jails/tomcat7
zroot/jails/windowsXP      13,2G  4,40G  12,8G  /usr/jails/windowsXP
zroot/tmp                   107K  4,40G    73K  /tmp
zroot/usr                  3,33G  4,40G  2,27G  /usr
zroot/usr/home               46K  4,40G    46K  /usr/home
zroot/usr/ports            1,06G  4,40G   965M  /usr/ports
zroot/usr/ports/distfiles   102M  4,40G   102M  /usr/ports/distfiles
zroot/usr/ports/packages   14,0M  4,40G  14,0M  /usr/ports/packages
zroot/var                   226M  4,40G  75,1M  /var
zroot/var/crash            31,5K  4,40G  31,5K  /var/crash
zroot/var/db                136M  4,40G   135M  /var/db
zroot/var/db/pkg            778K  4,40G   778K  /var/db/pkg
zroot/var/empty              31K  4,40G    31K  /var/empty
zroot/var/log              1,66M  4,40G  1,55M  /var/log
zroot/var/mail              428K  4,40G   428K  /var/mail
zroot/var/run                87K  4,40G    64K  /var/run
zroot/var/tmp                32K  4,40G    32K  /var/tmp

[cmd=]jls[/cmd]
Code: 
hydra# jls
   JID  IP Address      Hostname                      Path
     1  192.168.34.10   mx                    /usr/jails/mta
     2  192.168.34.16   dns                   /usr/jails/dns
     3  192.168.34.8    tomcat7               /usr/jails/tomcat7
     4  192.168.34.9    postgresql9           /usr/jails/postgresql9
     5  192.168.34.12   mysql                 /usr/jails/mysql55
     6  192.168.34.11   xp                    /usr/jails/windowsXP
     7  192.168.34.15   samba                 /usr/jails/samba

Issue:
Sometimes I get server lagging so much I can't use it on console.. When i type [cmd=]w -n[/cmd] I need to wait about 20-30 sec. then I get logged users. I think fold was jail xp (nginx + php) but one or two time I get laggy without this jail.

[cmd=]vmstat[/cmd] show nothing, in [cmd=]dmesg[/cmd], messages are any errors. I try memtest memory, but it passes all tests.

I check [CMD=]dd if=/dev/zero of=/tmp/test bs=1m count=100[/cmd]
Sometimes I get this:
Code: 
hydra# dd if=/dev/zero of=/tmp/test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 0.599228 secs (174987776 bytes/sec)
And sometimes this:
Code: 
hydra# dd if=/dev/zero of=/tmp/test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 26.476963 secs (3960333 bytes/sec)

I don't have other ideas. I will be grateful for any help.

Best regards.
 
Are you running low on memory? When I first set up ZFS it would eventually take up ~10GB of memory, which would remain locked for eternity. I can imagine that having as many pools as you do could cause it to take up nearly all of your memory. I limited ARC and total kernel memory and that took care of the problem. Given 16GB of RAM:
Code: 
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
vfs.zfs.arc_max="512M"
vfs.zfs.prefetch_disable="1"
Other values might be appropriate, but I just needed to set limits and those seem to work fine. This page might also be helpful.

If your problem isn't memory, maybe you can give the jails a higher nice or give your own login a lower nice (e.g. :priority=-5:) in login.conf.

Kevin Barry
 
Hey,

@cr4sh

uncomment the entries in loader.conf, you don´t seem to really know what they do any way, considering that you have disabled cache flushing without a proper SLOG. That is outright dangerous.


@ta0kira

By limiting kmem_size_max, you have effectively also limited what all of your other applications could be using. The only tunable you may keep is arc_max, but raise it. With 16GB of RAM, about 15 is really available, consider what other applications you have and make a rough guess how much RAM they´d like, lets say another 4, so you take 15-4=11 should end you up with vfs.zfs.arc_max="11G". Comment out the rest and see your performance shoot through the roof.

/Sebulon
 
This has repeated many times but do not touch kmem_size_max, IT IS NOT WHAT YOU THINK IT IS so leave it alone and don't set it at all. The same goes for most VM and ZFS related tunables on AMD64. The sole exception is the above mentioned vfs.zfs.arc_max.
 
Hello.
My server is now laggy.
About that loader.conf i add this after my server get problems. I can comment it, but this dont help me.

Load :
0,00 0,01 0,00
Click to expand...

Top show :
last pid: 18567; load averages: 0.02, 0.04, 0.01 up 0+07:42:31 08:19:23
203 processes: 1 running, 202 sleeping
CPU: 0.2% user, 0.0% nice, 1.2% system, 0.1% interrupt, 98.5% idle
Mem: 629M Active, 377M Inact, 4695M Wired, 6640K Cache, 221M Free
Swap: 4096M Total, 11M Used, 4085M Free
Click to expand...

gstatus show :
10 301 18 86 14.4 2250 1410 24.2 100.08| da0
10 301 18 86 14.4 2250 1410 24.2 100.08| da0p3
Click to expand...
 
@cr4sh

So it´s not memory related. By the looks of it your disks are working their ass off (probably with your databases). Buy a really good SSD, like D2RSTK251E19-0200 e.g. Make sure to partition it aligned to 1MiB and do proper IO-sizing with gnop. Then add it to your pool as a log device. That is going to offload your disks completely.

/Sebulon
 
It looks like you have the 2 discs attached to the "smart" controller and let that controller do some raid, yes?

The bottleneck seems to be exactly that controller (100.08% load, if I read the last post correctly), so no tuning in ZFS can get that corrected. You might consider to add the disks directly, and skip the controller software, because ZFS does it's own raid management.
 
@Crivens
You have right. ZFS is on RAID0 from E200i. But for now all works fine. I dont add new services or smt. And it crash ;/
I can buy for now ServeRAID M1015 LSI SAS2008 but i dont know how it works with 4K sector disks...
 
cr4sh said:
I can buy for now ServeRAID M1015 LSI SAS2008 but i dont know how it works with 4K sector disks...
Click to expand...

The SAS2008 chip handles 4k disks just fine. Just flash the card with the standard LSI Initiator/Target(IT) firmware to "dumb" it down to a standard HBA, from what I have read about it.

/Sebulon
 
Your hard drives are starving, looks like your files are fragmented
(using database (MySQL p.ex) on ZFS is often the issue).

Try to stop all jails, and check with [CMD="gstat"] -a[/CMD] that
there is no or very low disk activity before testing the raw disk speed
(just to exclude any issue with the controler)
[CMD="dd "] if=/dev/da0 of=/deb/null bs=1m [/CMD] and check the speed.

You are also very low on free space 4,4GB/64GB, when writing or
modifying files, ZFS is seeking free blocks on hard drive which
could explain trashing (busy 100%).

I've had similar issue on my production server, MySQL on ZFS, after a long
time of fighting I've moved all MySQL files to UFS.
 
Im back with my new controller ;)
I reflash it to IT flash.
I test [CMD=""]dd bs=1m count=1000[/CMD] from da0 and to da0 i got ~100-120M that is not bad i think.

Thanks all for help.
 
@cr4sh

Wow, that was fast! Although the test you performed only measures a continous, big read/write, and the performance issues you are probably experiencing are lots of small random writes generated from your databases. That´s why I think what your system really needs is a log device in the pool to use as a write cache for all those sync-writes. The log device also protects the data in the pool from being fragmented:

ZFS Fragmentation

/Sebulon
 
You must log in or register to reply here.
Back
Top