ZFS on FreeBSD getting knocked hard

D

drhowarddrfine

I don't know enough about this to respond properly but I would like to see answers here, not necessarily responses there, to what the poster "koitsu" has to say about ZFS on FreeBSD. koitsu seems to know quite a bit about what he speaks but I sense a bit of old knowledge or bad past experiences in what he's saying. If the points he makes about FreeBSD are true, I'd like to know.

I'm not trying to start a cross-forum flame war. I'm just trying to understand his points and whether they are correct or not. He also knocks FreeBSD as a server operating system.

http://www.dslreports.com/forum/r27418620-OS-and-file-system-choices?r=996#27422735
 
I'm going to start off by being the black sheep but I agree with what he stated in his second post (the one enumerating the current big ZFS in FreeBSD problems). Ex: I was also bit by the compression issue and it's not nice.

What I'm saying is that ZFS on FreeBSD is not perfect and because we expect so much of it, we tend to be highly aggravated when feature x or y doesn't exist or was not ported or doesn't work exactly or better then on Solaris.

I don't think there's anything wrong in admitting that we have a problem with ZFS as long as people who actually point out this problem are treated with respect and not being mocked.
 
I don't think that having to do some tuning is such a big issue.

It is also kind of funny to say that ZFS works out of the box on OpenIndiana when so many other things just don't work.
 
I agree with koitsu for the most part. You have to recognize that all systems have their strengths and weaknesses. My opinion is that the grass is just another shade of brownish green on the other side.

"The cake is a lie."

The FreeBSD Forums are "meh" when it comes to a good source of information.
Click to expand...
Also true, but only to some degree. Both the forum and lists are populated by professionals and developers, as well as hobbyists of widely varying experience, and the answers/tips you receive will always vary in quality- in my opinion- regardless of what category of people you are receiving from. So if you have a really tough problem/question, I think it´s good you are able to ask here on the forum and on the lists, and hope someone has a clue.


@da1

I haven´t had any issues at all with compression. In our production, compression has always had a positive effect on our workloads. Could you elaborate?

/Sebulon
 
I cannot see any bashing and don't see any value in the thread either :)
Summery:
1. Windows, Linux and BSD have support contracts, offered by several companies
2. Kernel patches will always be there, no system is perfect except the ones out of the basement in Redmond

Facts:
1. ZFS modules (spl/zfs) and program sources come from Oracle with the last CDDL releases and are freely available, they write the code and update the sources with patches
2. any version higher than 28 wasn't freely released by Oracle 'yet', 'ever', 'whatever'
3. the other OS's merely implement the FS into their systems and provide mostly compatitiblity patches
4. I haven't checked, but I didn't read anyone writing any own new features for ZFS other than Oracle with encryption etc. in version 30+ which is not free and needs Solaris with service contract.
 
It's tricky. I wouldn't recommend ZFS from my experience with it. Though I would like it to work stable on FreeBSD as we have always had a trusted stable OS for production.

Generally there is always some sort of passion behind this. As many people were first experimenting with ZFS the last couple of years there have always been issues arise on this forum. In some cases good advice was given but in others stfu style posts suggesting that it works on root on their home pentium era machine.

It also makes it hard whenever the native BSD derived and licensed file system HAMMER is ever brought up in conversation (even in passing) the same attitude surfaces. I can't imagine the politics which spawned Matt Dillon to lose his commit bit and start DragonFly BSD a decade ago are really that relevant today. Also is it hard for it to be ported? It is still BSD so I can't imagine it would be that difficult.

I agree with the post on that forum. That user seems very knowledgeable. I wouldn't mind seeing vimage finished and more server centric areas in FreeBSD worked on. But it's still a killer server OS... with UFS.
 
vfs.zfs.arc_max is not set on my system and I do use ZFS compression on a small Intel Atom with 2 GB RAM and two pools with each 3 x 2 TB disks. Effective FreeBSD is the most memory sparing system I ever used. Usually ~78 MB memory used and if I build some program or other heavy jobs it goes to ~150 MB.

This is something ZFSOnLinux has major problems with, because they cannot include the module into the kernel (licence issues) and therefore cannot apply this setting as kernel parameter. So this cannot be set with ZFS as root filesystem in Linux. And ZFSOnLinux did, on my 'big-top', use up to 14 GB of RAM just for ARC cache.

//edit: deduplication is a very risky thing to use in general. Its not a bad idea but be aware that in case you need to resilver a lot of data it might take 1-2 days. I read horror reports about that...

//edit: Oh and having ZFS as root, I create/destroy automated a snapshot of root every Monday night and it once saved my /etc/group already :)
 
UNIXgod said:
... in others stfu style posts suggesting that it works on root on their home pentium era machine.
Click to expand...

... for a certain value of "works", as usual and as not mentioned in such posts.

Sadly I have to agree with your post as well as with a lot of the arguments from the other forum thread. ZFS is a great file system with features which make it very valuable. But sadly it still feels a little bit alien on FreeBSD, that is where I have to agree with UFS being the file system for FreeBSD.

When it comes to HAMMER, I would like a port of that one. It looks promising and may deliver some enhancements needed. It would certainly be informative to hear all sides of the story which led to the split off of Mat and the fork of DragonFlyBSD. But as long as it is unlikely that all sides may have a free saying I would rather not hear any of it.
 
@drhowarddrfine

Its his personal opinion, for example he messes bugs in drivers with ZFS and so, it remindes me this:
duty_calls.png
 
After reading some of his posts I realize that the guy suffers from verbal diarrhea.

Inconsistent facts and information about FreeBSD, ZFS and UFS2+J

MYTH

ZFS consumes more memory than UFS.

This is graphs are from a very busy server divided into different jails, running apache, mysql, full email server and streaming media.

memory-week.png

apache_volume-week.png


MYTH

ZFS has very bad performance for Innodb databases.

If you adjust the recordset and add a ZIL device the performance improves over UFS.

mysql_innodb_rows-week.png


As a matter of fact, using a CACHE device as well improves the performance so much that you get the feeling that content is being served from a Varnish server.

zfs_stats_l2efficiency-month.png


FACT

ZFS needs more memory than UFS.

Maybe true but ZFS handles memory much better than UFS does.

I could show you more graphs of a new system that we installed with a full ROOT on ZFS running in HA with carp failover. In the beginning I was a bit reluctant as well given the nature of the server. It could not have proved to be more reliable.
 
I have several FreeBSD systems with ZFS in production that works great! These have compression enabled, NFSv4, Samba (2-8GB of memory and with/without cache/zil) and all systems are rock stable.

And I've only had trouble one time doing a rebuild world & kernel on a FreeBSD 9 system with everything installed on /
Not quite sure what exactly caused it, but it was definitely not because of mounting issues.

I also have a system with dedup, it works great. Though write speed, scrub and resilver is annoyingly slow. Read speed is great though.

Linux may have quite a few more active developers. But in my opinion, FreeBSD doesn't suffer from fewer developers at all. A few sharp developers are much better than many medicore.
I li
 
One question I have...
will Oracle release versions higher than 28 with the same licence or will the sources be forked and the community developes their own version of ZFS?
 
I am wondering if the ZFS tuning is needed for 64-bit machines with > 8 GB memory. Have been running quite big MySQL database on top of FreeBSD + ZFS with 48 GB memory, 2 x Xeon E5540 CPUs without any tuning of ZFS kernel parameters. So far the machine has been running very stable since I build it up for about one month. Initially I just setup a swap space of about 4 GB, and the system got some "swap_pager_getswapspace(16): failed" messages, but still no crashes, then I add some 20 GB additional swap space from SSD disks, these messages go away. Like the clean and elegant FreeBSD system and very happy that ZFS is an integrated part of it.
 
c789a123 said:
I am wondering if the ZFS tuning is needed for 64 bit machines with > 8G memory. Have been running quite big mysql database on top of FreeBSD ZFS with 48G memory, 2 x Xeon E5540 cpus without any tuning of ZFS kernel parameters. So far the machine has been running very stable since I build it up for about one month. Initially I just setup a swap space of about 4G, and the system got some "swap_pager_getswapspace(16): failed" messages, but still no crashes, then I add some 20G additional swap space from SSD disks, these messages go away. Like the clean and elegant FreeBSD system and very happy that ZFS is an integrated part of it.
Click to expand...

It depends on the version you are following, the amount of physical RAM and the purpose of the server. I have seen huge improvements in the upcoming 9.1-RELEASE.

For storage servers I perform no ZFS tuning except adding CACHE devices. For other purposes I usually limit the zfs.arc_max to 3/4 of the RAM. But again this is no rule of thumb and it depends a lot on the physical RAM.

On large memory systems I tend to use SWAP on SSD limited to 1/2 of the physical RAM. You might suddenly get a memory burst which will result in messages like you described above.

For mysql databases there is a lot of tuning you need to do in the database settings. Most example configuration files were written ages ago when 4GB of RAM was considered huge.
 
olav said:
Linux may have quite a few more active developers. But in my opinion, FreeBSD doesn't suffer from fewer developers at all. A few sharp developers are much better than many medicore.
Click to expand...

I wondered about that statement he made myself.

Turns out, the guy used to be a FreeBSD dev until 2008 when he left and made quite a few contributions. I'm sure those devs who visit this board know him. He's a sysadmin who knew quite a bit about the kernel but his background is maintaining and monitoring servers.
 
drhowarddrfine said:
I wondered about that statement he made myself.

Turns out, the guy used to be a FreeBSD dev until 2008 when he left and made quite a few contributions. I'm sure those devs who visit this board know him. He's a sysadmin who knew quite a bit about the kernel but his background is maintaining and monitoring servers.
Click to expand...

Thank you for that link. It was interesting to read.
 
A happy FreeBSD user here.

Previously I was suffering with the poor performance of our Linux storage box, but after switching to FreeBSD with ZFS all of my concern gone. Performance wise it works great on the same box with 1 SSD as L2ARC and ZIL, serving more than 30 VMs on a 4 disk system with snapshot + replication. With Linux it was only able to serve less than 8 I/O-intensive VMs.

I have bad experience with OpenIndianna and Nexenta because those servers can't even survive for more than 60 days even in very low utilization. FreeBSD is much stable so far on the same hardware, seems it to be more "hardware friendly" than the rest.
 
disi said:
Facts:
1. ZFS modules (spl/zfs) and program sources come from Oracle with the last CDDL releases and are freely available, they write the code and update the sources with patches
2. any version higher than 28 wasn't freely released by Oracle 'yet', 'ever', 'whatever'
3. the other OS's merely implement the FS into their systems and provide mostly compatitiblity patches
4. I haven't checked, but I didn't read anyone writing any own new features for ZFS other than Oracle with encryption etc. in version 30+ which is not free and needs Solaris with service contract.
Click to expand...

ZFSv28 was the last release by SUN/Oracle under the CDDL. However, it is not the end of development for the open-source version of ZFS.

The "main"/central code repository for open-source ZFS is illumos-gate. Many different OS developer groups are working together to improve open-source ZFS (Illumos, OpenSolaris, OpenIndiana, FreeBSD, NetBSD, Linux, and many others).

The major new feature for OSS ZFS is "feature flags". This allows for new features to implemented in ZFS without re-using any of the ZFS version numbers, so that versions won't clash with Oracle ZFS. FreeBSD devs have imported this feature into FreeBSD -current, and it should be available in FreeBSD 10; possibly in FreeBSD 9.2.

ZFS development has not stopped. No single OS has "the best" implementation of ZFS. Each has their own strengths and weaknesses.

As with any filesystems, you need to know the limitations and featureset, and how it maps to your requirements.

UFS isn't perfect. Ext3/4 aren't perfect. XFS isn't perfect. Btrfs is far from production-ready. ZFS isn't perfect. Each has it's role, and each can be used for various things.

Pick the OS, FS, and volume manager that works for your needs.
 
disi said:
One question I have...
will Oracle release versions higher than 28 with the same licence or will the sources be forked and the community developes their own version of ZFS?
Click to expand...

See my other reply in this thread.
 
c789a123 said:
I am wondering if the ZFS tuning is needed for 64-bit machines with > 8 GB memory. Have been running quite big MySQL database on top of FreeBSD + ZFS with 48 GB memory, 2 x Xeon E5540 CPUs without any tuning of ZFS kernel parameters. So far the machine has been running very stable since I build it up for about one month. Initially I just setup a swap space of about 4 GB, and the system got some "swap_pager_getswapspace(16): failed" messages, but still no crashes, then I add some 20 GB additional swap space from SSD disks, these messages go away. Like the clean and elegant FreeBSD system and very happy that ZFS is an integrated part of it.
Click to expand...

You will want to create a separate filesystem for the database files to be stored on. And you'll want to set the recordsize property of the filesystem to match the block size of the database engine. If you do that, things will improve immensely.
 
You must log in or register to reply here.
Back
Top