newfs increase inode

Rani · May 14, 2011

Hi, folks

I have a strange situations. I have a FreeBSD 8.1 stable server machine with only ftp, mysql as main servicies. There is an external application which connects via ftp username/password and for every connections that made creates one subdirectory (into base username home directory) where put 2 small files, an .xml and an .pdf file (both around 100-200K). FTP username home directory looks like this:

Code:

/usr/home/username
   -------- Dir1
            ----  ~15000 monthly subdirs with 2 small files 
   -------- Dir2
            ----  ~17000 monthly subdirs with 2 small files

So at every time when extern application connects to BSD box it creates one subdirectory inside Dir1 and Dir2 where puts two small files. Problem is where in one of those Dir1 or Dir2 it reaches >28000 subdirs proftpd can not make any subdirecotry inside them, even manually with root from console I am not able to make any subdirectory.

I have been googling for some days for this issues and I found out that 99% it might be a inode issue (because if I move 5000-10000 subdirs from one of this dir's to somewhere else, let's say backup_march, everything is working once again).

I have understood that in order to estimate how many inodes I might need, as from my scenario, I should do a simple math like:

2 (dir1 and dir2) * 2 (files, .xml and .pdf) * ( 20000 (max number of subdirs i have estimated into a month) * 12 (for one year) ) = 960 000 inodes

My system has 2 160GB IDE harddisks in mirroring with gmirror

Now i have

Code:

gw# df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
/dev/mirror/gm0s1a    496M    322M    135M    71%    /
devfs                 1.0K    1.0K      0B   100%    /dev
/dev/mirror/gm0s1e    496M     13M    443M     3%    /tmp
/dev/mirror/gm0s1f    141G     60G     70G    46%    /usr
/dev/mirror/gm0s1d    1.4G    528M    822M    39%    /var

and for inodes

Code:

gw# df -i
Filesystem         1K-blocks     Used    Avail Capacity iused    ifree %iused  Mounted on
/dev/mirror/gm0s1a    507630   329264   137756    71%    3178    62612    5%   /
devfs                      1        1        0   100%       0        0  100%   /dev
/dev/mirror/gm0s1e    507630    13572   453448     3%     127    65663    0%   /tmp
/dev/mirror/gm0s1f 147865114 63015148 73020758    46%  857197 18267025    4%   /usr
/dev/mirror/gm0s1d   1502094   540624   841304    39%     981   210985    0%   /var

One of dirs

Code:

gw# ls -la /usr/home/username/dir2/ | wc -l
   14403

(and when it reaches ~28000 I can not create subdir's anymore)

And structural schema is/will be

Code:

/ 
/usr -> ftp user home + mysql DB 
/var -> mysql install and logs  
/tmp ->

I also understood that I must combine block size and frag size along with inode size and for a system with much small files inode size must be smaller like:

newfs -O2 -U -b 4096 -f 512 -i 2048 /dev/xxxx

My question is what parameters should I give when defining partitions like (/usr, /var) in order to assure that it won't be for one year an issue with running out of inodes?

Thank you very much for your patience and understanding.

Best regards.

Rani · May 18, 2011

Nobody? OK, I ask then a simple question. I am going to buy two 500GB HDDs to install FreeBSD in mirroring mode with gmirror.

Which parameters should I put to newfs so that I will make sure that in /usr partition in one directory I could make 100000-200000 subdirectories each of them including 2-3 small files?

Thanks for your patience.

vermaden · May 18, 2011

I would go for ZFS, I see that you use mirror of 2 drives in your system, so you will be able to create same mirror with ZFS.

ZFS would also allow you compression, verifying checksums of files, deduplication (in ZFS v28) and much more.

Rani · May 20, 2011

Thank you vermaden for your answer. Regarding my problem from above, after all, I going to build a new server with some specifications like:

CPU: Core i3 550 3.2GHz - http://ark.intel.com/Product.aspx?id=48505
Mbd: INTEL iQ57 Express (mATX,4 DDR3 SDR,SB,SVGA,1GbitLAN,Serial ATA II/300,6xUSB2.0,HDMI, RAID 0,1,5,10 & Matrix)
RAM: 4GB DDR3
HDD: 2x750GB-7200-32MB-SATA-6Gb Barracuda 7200

I hope that I could overcome my old issues regarding too many subdirectories with this new hardware but still I kindly ask a few questions:

1. Above hardware specs would fit into FreeBSD/amd64 flavor? Would it be safe for a heavy mysql server (50-80 conns/sec) or I should install basic i386 flavor?

2. I intend to make partitions like:

Code:

     /       - 1GB (enough for boot, I think)
     /usr    - 40GB (for kernel compilation and system ports)
     /usr/home - 650GB (for only one ftp account from all system in which I 
      will have those several 100.000 subdirs)
     /var    - 50GB (for mysql and a basic phpmyAdmin)
     /tmp    - 9GB

Since I would go for hardware raid I will not bother with gmirror. As for ZFS I have no experience with it and not much free time left to test. Althought from what I have read from FreeBSD handbook since my mirroring is hardware level I could try a simple non-redundant ZFS pool.

Would be UFS+J a safer way to that beside standard UFS+U (soft updates)?

3. since my /usr/home partition is about 650GB the default news parameters from install are -b 16384 -f 2048 (as I remember from some sysinstall option menu) I am not sure if should I override default slice creation with minimize -i parameter in order to achive for a greater inode number (for slice that will use 95% small files) like:
[cmd=]newfs -O2 -b 4096 -f 512 -i 2048 /dev/xxxx[/cmd]

Thanks very much for your time.

vermaden · May 21, 2011

Rani said:
1. Above hardware specs would fit into FreeBSD/amd64 flavor? Would it be safe for a heavy mysql server (50-80 conns/sec) or I should install basic i386 flavor?

Hi mate, go all the way amd64, these specs are 'safe' for FreeBSD server, you should have very good performance.

2. I intend to make partitions like:

Code:

     /       - 1GB (enough for boot, I think)
     /usr    - 40GB (for kernel compilation and system ports)
     /usr/home - 650GB (for only one ftp account from all system in which I 
      will have those several 100.000 subdirs)
     /var    - 50GB (for mysql and a basic phpmyAdmin)
     /tmp    - 9GB

I would put it a little differently:

Code:

/          1GB  UFS  gmirror of ada0s1 <--> ada1s1
/usr         *  ZFS  dataset
/usr/home    *  ZFS  dataset
/var         *  ZFS  dataset
/tmp         *  ZFS  dataset

By * I mean all space left for ZFS pool, you do not specify SIZE for datasets, you can limit it but it's pointless for most purposes.

(for mysql and a basic phpmyAdmin)

You may want to set ZFS blocksize of that dataset to 4KB or 8KB (default 128KB) so you will get better MySQL performance.

Since I would go for hardware raid I will not bother with gmirror. As for ZFS I have no experience with it and not much free time left to test. Althought from what I have read from FreeBSD handbook since my mirroring is hardware level I could try a simple non-redundant ZFS pool.

Omit that hardware raid mate, seriously. Put gmirror for / for first 1GB partition, then create ZFS mirror from the second partition of the left space, that way you would have RAID1 of / and RAID1 of rest of the data, and as you would like to add additional 2 x 750GB drives for performance increase, you would add another RAID1 (mirror) ZFS from these new two disks and that would 'convert' Your ZFS pol into RAID10 (which would add great performance increase) you can also create 3-way RAID10 with 3 x RAID1 for even more performance increase, you would not be able to do that 'magic' with your hardware controller.

Configure your controller so it will only pass single disks to the host.

Would be UFS+J a safer way to that beside standard UFS+U (soft updates)?

If you would do UFS only for that 1GB / so it does not matter generally, it could be even without U or J and mounted in read-only for most of the time.

Rani · May 22, 2011

I see. Thank very much for your reply. I have read all day long tutorials on zfs, zfs only-mirrored gpt, gmirror including your tutorial with modern FreeBSD install (it has three disks there which does not fit my needs) but I am kind of confused regarding your partition schema, meaning I still do not get how to do it.

1. Could you be kind to show me how to partition your schema? Or at least point me out some tutorials?
2. I also need to recompile GENERIC kernel in order to enable PF (I know what parameters to put in my kernel file for PF). What parameters should I put in kernel file to be assured that after compile and restart my system will still booting into your partition schema?

Thanks in advance.

vermaden · May 22, 2011

Rani said:
1. Could you be kind to show me how to partitioning your schema ? or at least point me out some tutorials ?

Code:

ada0/ada1                DISK1         DISK2
                     +-----------+ +-----------+
ada0s1/ada1s1 SLICE1 | UFS / 1 G | | UFS / 1 G | (GMIRROR on ada0s1/ada1s1)
                     +-----------+ +-----------+
ada0s2/ada1s2 SLICE2 | Z   F   S | | Z   F   S | (ZFS mirror on ada0s2/ada1s2)
                     +-----------+ +-----------+

something like that mate, my tutorial 'modern install' is also useful here, you just use 2 instead of 3 disks, ZFS mirror instead of ZFS RAIDZ (RAID5 equivalent).

2. I also need to recompile GENERIC kernel in order to anable PF (i know what params put in my kernel file for PF). What parameters should i put in kernel file to be assured that after compile&restart my system will still booting into your partition schema ?

Add what You have to add about PF and thats it. Use all the rest as You would without recompile - by modules.