Recommended ZFS/L2ARC configuration

[marvel]

Hi guys,

I want to have your opinion on my configuration. I have a server which runs Linux now and it runs EXT4 on root with XFS for the large storage. I don't like it though, it's slow and it doesn't utilize the full potential of this machine. It has a 10GbE internet connection and right now I get around ~180 MB/s download speeds.

So, I want to speed things up and switch to FreeBSD, and go with ZFS for my slow storage SATA drives. I already added SSD drives to use for the OS and as L2ARC cache.

First my server configuration:

• Dual Xeon E5-2620v2 (2x6-core)
• 128 GB RAM
• 10 x 4 TB SATA WD GREEN (I know this probably sucks).
• 2x 256 GB Plextor M6e PCIe SSD drives.

So I had the following partitions in mind:

• FreeBSD 9.3 OS - UFS 50 GB RAID1 on SSD (Any reason to go with 10 BTW?)
• Data volume - 10 x 4 TB RAIDZ2.
• L2ARC 400 GB SSD RAID0

Would this be a good configuration? Anything else I need to pay attention to? I mean configuration-wise so my memory and L2ARC is fully utilized by ZFS?

I know WD GREEN has that 4K bug, would I be working around this by creating my partitions like this?

gpart create -s gpt ada0
gpart add -a 4k -t freebsd-zfs -l disk0 ada0
gnop create -S 4096 /dev/gpt/disk0

Thanks in advance!

 

[xibo]

What kind of data will you be hosting, and how large is the working set?

Usually for mainstream setups, which is indicated by the WD GREENs, the working set is very large and completely random (with a very limited amount of "hot" data), where it makes more sense to store only metadata on the l2arc and keep non-metadata in primary cache only - 128GB RAM should be enough for that but you can dump in the two SSDs just in case. Also, for a raid with low random write speeds, there's probably more gain in using the SSDs as log devices.

Also, I'd recommend you to use FreeBSD10 for a new setup rather than 9.

 

[marvel]

It's a media server so it's music, films, series etc. File sizes are usually large, anything from 300 mb to 20 gb but there will be a lot of small metadata as well.

Most important to me are write speeds, I usually have several threads where one's downloading, the other unpacking, etc. so yeah using the SSDs as log devices makes much more sense as I never had problems with reading speeds anyway.

Thanks for your advice!

 

[ViktorK]

A separate ZFS intent log will only improve performance if you are performing synchronous writes (i.e. NFS). If you are using samba then you will probably see no performance improvement. Note that a single RAIDZ2 vdev will have the IOPS write performance of a single disk. If you need more IOPS create more vdevs.

 

[Sebulon]

gpart create -s gpt ada0
gpart add -a 4k -t freebsd-zfs -l disk0 ada0
gnop create -S 4096 /dev/gpt/disk0

What about pool configuration?
# zpool status

A separate ZFS intent log will only improve performance if you are performing synchronous writes (i.e. NFS). If you are using samba then you will probably see no performance improvement.

You actually have to run SAMBA with AIO for it to be asychronous and with FreeBSD 10.0 and higher´s ctld(8) that actually listens to sync calls as opposed to net/istgt, a SLOG really helps performance-wise. It´s also a very good thing to have for keeping the file system un-fragmented. Not to be nit-picking here but you always have the ZIL (ZFS Intent Log), just that without a SLOG (Separate LOG device), the ZIL lives in the data disks instead and leaves empty "holes" of fragmentation in the filesystem, which over time is negative on the pool´s performance. A SLOG prevents that.

/Sebulon

 

[t1066]

In addition to what @Sebulon wrote above, there are other programs which may perform synchronous writes. The most common ones are the browsers. Another example would be emulators/virtualbox-ose. On my system without any NFS activity and with uptime around 14 days, about 26GB had already been written on the SLOG.