Gmirror with big amount of small files leads to system's freeze

[gps]

Hi all, I have a some problem with high gmirror I/O on one of my web servers. When I generate some copy/move/delete filesystem commands that leads to systems freeze on several seconds up to minute and over so I see big lag in SSH's console. The same thing happens when I try to import a big MySQL script (table about 3 M records) - very very slow, about 5000 records in one minute). It's a big problem for me and for my knowledge of FreeBSD system, I didn't find any solution anywhere. The server has several jails and two of them serve big amount of small files. That's what I have:

root@server5:/root # uname -a
FreeBSD server5 9.1-RELEASE FreeBSD 9.1-RELEASE #1 r254677: Fri Aug 23 03:02:49 CEST 2013  root@server5:/sys/amd64/compile/WH  amd64

root@server5:/root # gmirror status
  Name  Status  Components
mirror/gm0  COMPLETE
ada0 (ACTIVE)
ada1 (ACTIVE)
root@server5:/root # gmirror list
Geom name: gm0
State: COMPLETE
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3954888687
Providers:
1. Name: mirror/gm0
  Mediasize: 3000592981504 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r2w2e3
Consumers:
1. Name: ada0
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  State: ACTIVE
  Priority: 0
  Flags: DIRTY
  GenID: 0
  SyncID: 1
  ID: 2336097205
2. Name: ada1
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  State: ACTIVE
  Priority: 0
  Flags: DIRTY
  GenID: 0
  SyncID: 1
  ID: 2137211893

root@server5:/root # mount
/dev/mirror/gm0p3 on / (ufs, local, noatime, soft-updates)
devfs on /dev (devfs, local, multilabel)
/usr/jails/basejail on /usr/jails/php.server5/basejail (nullfs, local, read-only)
devfs on /usr/jails/php.server5/dev (devfs, local, multilabel)
fdescfs on /usr/jails/php.server5/dev/fd (fdescfs)
procfs on /usr/jails/php.server5/proc (procfs, local)
/usr/jails/basejail on /usr/jails/mysql.server5/basejail (nullfs, local, read-only)
devfs on /usr/jails/mysql.server5/dev (devfs, local, multilabel)
fdescfs on /usr/jails/mysql.server5/dev/fd (fdescfs)
procfs on /usr/jails/mysql.server5/proc (procfs, local)
/usr/jails/basejail on /usr/jails/httpd.server5/basejail (nullfs, local, read-only)
devfs on /usr/jails/httpd.server5/dev (devfs, local, multilabel)
fdescfs on /usr/jails/httpd.server5/dev/fd (fdescfs)
procfs on /usr/jails/httpd.server5/proc (procfs, local)
/usr/jails/basejail on /usr/jails/service.server5/basejail (nullfs, local, read-only)
devfs on /usr/jails/service.server5/dev (devfs, local, multilabel)
fdescfs on /usr/jails/service.server5/dev/fd (fdescfs)
procfs on /usr/jails/service.server5/proc (procfs, local)
/var/www/images on /usr/jails/service.server5/var/www/images (nullfs, local)
/var/www/images on /usr/jails/httpd.server5/var/www/images (nullfs, local)

/usr/jails/httpd.server5 has a big amount of small files - images (nginx proxy cache), about one million. /usr/jails/service.server5 has a big amount of small files too - PHP's Zend cache, 500 K-800 K files, don't know exactly
/usr/jails/service.server5/var/www/images - about one million small files (images)

When I perform some action on the filesystem, clean up /var/www/images for example ( find . -type f -print -delete or rm -rf .), it leads to big lags for all other processes. I know that is normal situation to have some stress for HDD in this case, but I wait about 10 hours and process was not ended and at the same time I saw that access to Zend's cache files takes over 30-60 sec. I had to break it.

Can anyone advise me something to improve that situation? Do I have a bottleneck at filesystem tuning or gmirror (I'll provide any additional info if need)?

 

[jb_fvwm2]

Can only suggest workarounds...

[src] rsync ... ... --bwlimit=1000 ... . /dest

Please note the dot before the destination. Also, that keyword search can lead to full-line examples here on the forum.
..................................
Sometimes a pipe

for f in $(find /src/images -type f ); do (gcp -Rvu "$f" /dest && sleep 1); done
# or /bin/rm -v , as the case may be

The one second lag usually permits other processes to run unabated. Also the 1000 speed is about 1/10 of the usual maximum disk speed from testing here. And please test well beforehand.
These, if fitting, can easily be put into a shell script for unattended runs usually.

 

[wblock@]

Balance: round-robin

This is one of the slower mirror methods, reading from each drive in turn rather than the one with lower load or splitting the request. Depending on the hardware, that might not make a lot of difference.

The number of files could also be a problem. Many filesystems have trouble with huge numbers of files in a single directory causing performance problems. Directory trees are often split, sometimes into subdirectories based on the first character of the filename or some other sorting system.

Note also that you are using an old version of FreeBSD, with support ending in a month.

 

[gps]

Can only suggest workarounds...

[src] rsync ... ... --bwlimit=1000 ... . /dest

Please note the dot before the destination. Also, that keyword search can lead to full-line examples here on the forum.
..................................
Sometimes a pipe

for f in $(find /src/images -type f ); do (gcp -Rvu "$f" /dest && sleep 1); done
# or /bin/rm -v , as the case may be

The one second lag usually permits other processes to run unabated. Also the 1000 speed is about 1/10 of the usual maximum disk speed from testing here. And please test well beforehand.
These, if fitting, can easily be put into a shell script for unattended runs usually.

Thanks for the workaround, but for me this is unacceptable option. Now the load on the server is small, but if it grows, the file system will not allow him to fully work. So I need to resolve the issue with gmirror/UFS.

This is one of the slower mirror methods, reading from each drive in turn rather than the one with lower load or splitting the request. Depending on the hardware, that might not make a lot of difference.

The number of files could also be a problem. Many filesystems have trouble with huge numbers of files in a single directory causing performance problems. Directory trees are often split, sometimes into subdirectories based on the first character of the filename or some other sorting system.

Note also that you are using an old version of FreeBSD, with support ending in a month.

Thanks for the reply. Files are stored already in path on a similar scheme so they are not much within a single directory. If I change the method, whether it will save the system from the entrance to a halt after the removal of 500 files? It makes sense to hope for it?