ZFS vs HAMMER

Status
Not open for further replies.
Oko

Oko

I am starting this thread in desire to document ZFS (FreeBSD) and HAMMER1 (DragonFlyBSD) feature differences for people who are interested in serious storage solutions. As a disclaimer I have been using ZFS almost exclusively for the past year and the half as a storage solution at my work but should be considered ZFS n00b. I tried to use DragonFly but I bailed out about 3 months ago mostly out of frustration with monitoring tools ( I could not compile Monit on DF and net-snmp didn't work the way I wanted).

For starters ZFS is a file system and a volume manager in one. HAMMER is just a file system. If you want to use ZFS, you should get a decent Host Bus Adapter (HBA) like LSI 9211-8i. If you want to use HAMMER as a serious storage solution you should get a good hardware RAID controller, LSI and Areca come to mind as very well supported (not cheap with the price over $600 in U.S.).

Both systems are in practical terms 64 bit only and they are resources hungry. If you want ZFS you need lots of expensive ECC RAM. The fact that somebody is running ZFS in her/his laptop on 2GB of RAM will be of no importance to you when that 16TB storage pool fails during the ZFS scrub or de-duplication because you have insufficient RAM or the RAM is of low quality. The rule of the thumb is that you typically need about 16GB of RAM to begin with ZFS and then about 1-2 GB for each additional TB of data. You also need a really good Intel processor. FreeNAS officially doesn't support any AMD hardware and those people know something about storage solutions. Please refer to mailing lists to all sorts of problems with AMD processors. Long story short you are looking at the very serious initial investment with ZFS.

HAMMER is not a cheap file system too, but it is far cheaper than ZFS. While you probably need a little bit more than 2 GB of RAM with 16TB of RAID 6 with HAMMER1 on the top of it, you probably can get a way with it. Notice that HAMMER is not enterprise file system suitable for large data farms according to Matt himself.

When is come to storage solutions, I like to install both FreeBSD and DF on small 32 GB SSD drives (preferably 2 in ZFS mirror configuration or HAMMER mirror-stream). Using SSDs has actual technical advantages. I hope people seriously help me complete these lists.

ZFS good
  1. Self healing.
  2. ZFS Clones are writable (ideal for hot migration).
  3. Fully journaling system using ZFS snapshots.
  4. Uses compression to store data I like (lzmi).
  5. Portable storage: ZFS pool created in Linux can be imported into FreeBSD.
  6. "Portable" in general as it runs on Solaris, FreeBSD, and Linux. The truth of the matter is that large parts of Solaris kernel have been reimplemented in FreeBSD to add native support for ZFS. Linux implementation relays on FUSE.
  7. It is possible to use ZFS Volume as a iSCSI Target
  8. It is possible to share ZFS via NFS and SMB

ZFS bad
  1. Well known data base degradation (not suitable to keep SQL data bases).
  2. No file grained journaling like hammer history.
  3. No volume growing at least FreeBSD version.

ZFS ugly
  1. CDDL license.
  2. Extremely complicated peace of software. List of files required to compile ZFS into FreeBSD kernel support is very loooong.
  3. Legally associated with Oracle.
  4. Native encryption for ZFS has been only available in Oracle version.
  5. Upstream is dead. I don't know much about you but OpenZFS doesn't inspire lots of confidence in me.
HAMMER good
  1. Fine grained journaling with HAMMER history beat a pants out of everything around.
  2. HAMMER has high focus on data integrity.
  3. Backup ready having network aware hammer-snapshot/hammer-stream.
  4. Phenomenal performance with with SQL.
  5. Much lower RAM requirements for HAMMER de-duplication than ZFS.
  6. Pseudo-Filesystems (PFSs).
  7. Fully open source with nice BSD license attached to it
  8. It is possible to share HAMMER via NFS.

HAMMER bad
  1. No self healing.
  2. No compression.
  3. No volume growing.
  4. mirror-snapshot/stream are read only and require human intervention to be deployed.
  5. PFS can't be used as iSCSI target.

HAMMER ugly
  1. HAMMER 1 is dead end. The file system has reached limitations of its original design and no new features are planned/possible.
  2. Not portable https://wiki.freebsd.org/PortingHAMMERFS.

Finally, the DragonFly community is really tiny and charismatic. People are really cool but there are too few of them to stabilize infrastructure which is needed for enterprise solutions. FreeBSD also has a meaningful vendor support with HBA cards and RAID cards for that matter which should be used only with UFS. DF jail infrastructure is stuck in 2005 although there is a Google Summer Code project 2015 to sync DF jails to FreeBSD 9.xxx.

As a matter of subjective opinion I will tell you that DF new network stack and the whole OS feels smoking fast comparing to FreeBSD. As a matter of fact I have not seeing anything faster than DF on my hardware period (I have Red Hat computing nodes and OpenBSD too beside FreeBSD).
 
Last edited:
Oko

Oko said:
ZFS bad
3. No volume growing.
Click to expand...

You can grow ZFS pool, with these:
# zpool set autoexpand=on zroot
# zpool online -e zroot ada3 ada3

I would also add these:

ZFS good
5. LZ4 compression which is even better then LZMA.
6. Deduplication.
7. ZFS Send/Receive with deduplication and compression on the fly.
8. Boot Environments.
9. Can 'do' block devices so can be used as backend for iSCSI or SWAP directly.

ZFS bad
4. Lots of RAM needed for deduplication.
5. No 'offline' deduplicaction like with HAMMER.
 
The comparison makes me want to see HAMMER2 on FreeBSD. Wasn't there some effort to have it ported to FreeBSD? That would be one point which I would add to the "HAMMER Ugly" list, if it is so deeply integrated into the kernel that it can not be ported to it's sibling operating systems. But that is only me.
 
One major difference is that ZFS is cross-platform. Is HAMMER available on anything other than DragonFly?
 
Crivens said:
The comparison makes me want to see HAMMER2 on FreeBSD. Wasn't there some effort to have it ported to FreeBSD? That would be one point which I would add to the "HAMMER uUgly" list, if it is so deeply integrated into the kernel that it can not be ported to it's sibling operating systems. But that is only me.
Click to expand...
Done! HAMMER2 should rectify most if not all problems with HAMMER1 including portability, lack of compression, snapshots will be alive and it will be true clustering file system. Readers be aware after two years of work HAMMER2 is not even close to being usable let production ready in DF. It will take years if ever that thing stabilize and get ported to other OSs.

wblock@ said:
One major difference is that ZFS is cross-platform. Is HAMMER available on anything other than DragonFly?
Click to expand...
Nope! See Matt's answer I linked above from FreeBSD mailing lists to see how difficult/impossible would be porting HAMMER to FreeBSD without essentially replicating parts of DF kernel. DF is "the logical continuation of the FreeBSD 4.x series" and as such at this point is really very different OS than FreeBSD with whom is sharing common ancestry. The amount of work which went into rewriting kernel, network stack and for that all major parts of the OS is incredible in particularly in the light of the fact that it has been done by such a tiny group of developers. In BSDtalk248 Matt talks about it and makes the following interesting observation. At the moment of DF forking FreeBSD was faster, leaner and betting the pants out of Linux in all categories. According to him and in my experience FreeBSD is now one super server bloated OS requiring huge resources while DF is just a smoking fast research OS.
 
Almost all the bullet points in the original post are false. ZFS is active upstream, ZFS allows native encryption and it is possible to grow ZFS volumes. Also, ZFS runs fine with all sorts of RAM and doesn't require much of it. I often use ZFS on machines with less than 2GB of RAM and it runs smoothly. Granted, if you want deduplication or massive RAID arrays then, yes, more RAM is good, but it's not at all required for most home and small office scenarios.
 
NewGuy said:
Almost all the bullet points in the original post are false. ZFS is activ upstream, ZFS allows native encryption and it is possible to grow ZFS volumes. Also, ZFS runs fine with all sorts of RAM and doesn't require much of it. I often use ZFS on machines with less than 2GB of RAM and it runs smoothly. Granted, if you want deduplication or massive RAID arrays then, yes, more RAM is good, but it's not at all required for most home and small office scenarios.
Click to expand...
Please fix it. Samples of code (how do you do for example native ZFS pool encryption) are especially appreciated. I am not an expert by any stretch of imagination. The idea of original post is to try to objectively compare two most sophisticated file systems in existence not start any flame wars. I said I am n00b to both file systems. Nothing would make me happier than having Matt and FreeBSD kernel hackers edit my post. Your claim about RAM is pure BS. Please tell people/customer service of FreeNAS/TrueNAS that you are running FreeNAS/TrueNAS with 2GB of RAM and let me know what is their reaction.
 
ANOKNUSA said:
Care to elaborate? I choke on my coffee every time I see this stated. For some inexplicable reason, FOSS advocates like to declare a project "dead" if it goes more than 15 minutes without a code commit.
Click to expand...
This might be just my feeling. ZFS and Solaris of course were Sun children and I have personally associated (probably mistakenly) all development of ZFS to the employees and affiliates of that company. Oracle has closed sourced Solaris again. They charge money even if you think that you want to use Solaris. I have not seeing any talk of Solaris 12. It looks Solaris is dead and not marketed at all. ZFS is its native file system. Oracle closed source development. Native encryption for example was available for long time in Oracle version. My understanding is that Oracle implementation of ZFS is not compatible any more with the rest of the world and their ZFS pools can't be imported into other OSs.

If OpenZFS is a future what is their tier one development platform? illumos , FreeBSD, or God forbid Linux on which ZFS feels very awkward?
 
Since I would like to migrate my server from UFS to $something_else, I was actually looking for a ZFS vs.HAMMER comparison a while ago; I'm somewhat disappointed with FreeBSD as of late, and HAMMER seems like a very viable alternative.

However, I would love if you would add references to your points. Right now it's just a bunch of assertions, with very little extra info.

HAMMER has high focus on data integrity.
Click to expand...

So does ZFS.

Here are the slides of a talk on the subject I attended a while ago.
 
I don't really see how the code being CDDL is an "ugly" feature of ZFS when comparing file systems for use. What problem does that actually create for people who are looking for a file system to store their data on. Same for the point about Sun/Oracle. What issue does that actually cause for users of OpenZFS.

Do you have any examples of problems with AMD as well? A quick search didn't really turn anything obvious up, and both Intel and AMD processor use the same AMD64 version of FreeBSD, using the same code. So unless AMD processor have major bugs, I don't see why ZFS should have any specific problems on them.

I also don't really like the "ZFS needs ECC" arguments. ZFS needs ECC no more than any other file system. It's not like the developers specially made it rely on ECC features. With any file system, errors can happen in RAM which then get written to disk. Because one of the major ZFS "selling points" is that it has full data integrity, it's advised that users use ECC for production systems. Otherwise you run the risk of people thinking their data is 100% protected, when there is still actually a small chance of corruption. With other file systems no real guarantee of integrity is given in the first place. If HAMMER is supposed to provide great integrity, then I suspect ECC is advised there too.

Also I believe the OpenZFS project is fairly active. They've just had their developer summit and the original primary developer from Sun is still heavily involved. I think at the moment though the priority is more infrastructure work, building a solid API, making the code differences between ports smaller etc, stuff that doesn't really make any obvious changes for users.

I would like to see built in encryption, which does not exist at the moment. Can't see it happening any time soon though. The common way of doing it at the moment is with GELI, which is a lot more hassle than just using a dataset property like on Solaris.

It is also very complicated, there's not much chance of recovery if there is a problem, other than from backup. It does also seem to get slower over time as the data gets more and more fragmented. Makes scrubs slower as well as the scrub has to keep jumping around the disk to read the data in sequence. Not sure what the answer is for that other than rebuilding the pool every couple of years or so.

HAMMER seems intriguing but it still seems too new, hasn't had anywhere near the use and testing of ZFS so I'm not sure how much to trust it. It seems development has now moved onto v2, which has yet to appear, and has a very long list of desired features which worry me. Overall DragonFlyBSD just doesn't seem to have the user base or support for me to be happy using it for anything serious. And are there any sources to backup up the "phenomenal SQL performance" or it "beating the pants off everything else around"?
 
Your claim about RAM is pure BS. Please tell people/customer service of FreeNAS/TrueNAS that you are running FreeNAS/TrueNAS with 2GB of RAM and let me know what is their reaction.
Click to expand...

Pure BS? What's BS about him saying he runs ZFS systems with 2GB or RAM? I also have a backup system with 1.5TB of data and 2GB of RAM, runs fine. Is that BS as well? And what does it have to do with FreeNAS/TrueNAS devs? They will obviously prefer more RAM, especially in relation to their enterprise TrueNAS kit but that doesn't mean it doesn't work or anyone saying they run it with less is making it up.

You say you have little knowledge of either, and want to create a fair comparison but you seem quite biased towards HAMMER.

My main issue with HAMMER is that with file systems you can't just produce code with a huge range of features, including complex stuff like clustering and just say there you go, it's better than everything else. ZFS has been through 10 years of massive use and thousands of issues have been found and fixed. I don't see that HAMMER has been through that, or that it ever will.
 
usdmatt said:
You say you have little knowledge of either, and want to create a fair comparison but you seem quite biased towards HAMMER.
Click to expand...
Quite biased is little bit too heavy word for somebody who runs five large ZFS/FreeBSD file servers and have decommissioned the only DF based production machine at work. At the moment when I decommissioned our only DF production machine even LDAP authorization was not possible on DF so it was not realistically usable on the serious file server. In the mean time DF got infamous PAM modules so I was the first one to get LDAP actually work. Please check users@dragonflybsd.org

usdmatt said:
My main issue with HAMMER is that with file systems you can't just produce code with a huge range of features, including complex stuff like clustering and just say there you go, it's better than everything else. ZFS has been through 10 years of massive use and thousands of issues have been found and fixed. I don't see that HAMMER has been through that, or that it ever will.
Click to expand...
Please see ugly. HAMMER 1 is DEAD END. All your points are valid, no questions about it. HAMMER 2 for all I know might never be completed so I don't even want to go there. Once HAMMER 2 is finished (if ever), tested and DF code base stabilized we can try to compare ZFS and HAMMER 2 in enterprise environment. Until then I think your and my bank will continue to use ZFS.

However SOHO file server based on HAMMER might be viable option in particular if the cost is of matter of great concern. I will repeat this and you may call me bias: DF feels smoking fast. It has brand new incredibly fast network stack. They have their own supper fast implementation of NFSv3 (Matt even said that they are not even interested in NFSv4) and there are some other pieces which a serious hobbyist might appreciate.
 
I would be interested to try NFSv3 on Dragonfly. NFS really does feel a bit of a mess on FreeBSD to me at the moment. There's various versions of the daemon now, all with their own problems. You only have to look round this forum to find people who are struggling just to get the right version of NFS to run in the first place. I also think there's a few projects that outright dislike NFSv4? Didn't one of the OpenBSD devs say they don't want to touch it?

One of the big plus points for HAMMER to me was that it was designed from day one to be a lightweight contender for the OS's default file system. ZFS was designed to allow Sun to create NetApp style storage servers, with NetApp features and while it does work on the smaller end, it does seem much more suited to larger scale storage systems. HAMMER1 seemed to be a good default file system. Of course that's dead now, and with all the featured planned for v2, I wonder if that's going to end up with similar problems running on the low end?

Depending on the functionality, and how well it works, the clustering could be a big win for HAMMER2. I'd love a BSD supported file system where I can just throw some servers together and have clustered, resilient storage. At the moment we have to make do with stuff that's been hacked over from Linux.

We'll have to see how it develops. I've done very little research on DragonflyBSD, but last time I looked (probably 3+ months ago) I couldn't find much decent information about HAMMER2 at all other than a post by Matt about the planned features and a "I've started but it actually working is a long way off" message.

Edit, this is what OpenBSD said about NFSv4. Don't know if they've changed their mind since then.
Not everyone was happy with the new protocol. In 2010, OpenBSD's Theo de Raadt wrote: "NFSv4 is not on our roadmap. It is a ridiculous bloated protocol which they keep adding crap to."
Click to expand...
 
Many people report running ZFS with as little as 1G of RAM. FreeNAS is a canned package and they recommend what they feel works best with their system. And we're long past the days when 8G was an outlandish amount of system memory for running large RAID arrays. Consider also that ZFS takes over RAID controller functions, so memory is not an additional cost. Instead of expensive RAID controllers, you buy system memory and cheaper JBOD controllers.

So look at it from the other direction: what mature, modern filesystems are available that do not lock the user into a single operating system? ZFS, and, well, that's it.
 
Oko said:
Samples of code (how do you do for example native ZFS pool encryption) are especially appreciated.
Click to expand...
For people interested in native ZFS pool encryption see Pawel Jakub Dawidek's interview on BSDNow Episode 62, specifically at 14:15.

Oko said:
Your claim about RAM is pure BS. Please tell people/customer service of FreeNAS/TrueNAS that you are running FreeNAS/TrueNAS with 2GB of RAM and let me know what is their reaction.
Click to expand...
I have no idea what sort of defaults FreeNAS or TrueNAS use, but from my experience running ZFS on somewhat older desktops and laptops with less RAM (I've never gone below 3 GB) isn't a problem. I'm not talking about several TB drives with dedpuplication turned on and I do some tuning.

usdmatt said:
I also don't really like the "ZFS needs ECC" arguments. ZFS needs ECC no more than any other file system. It's not like the developers specially made it rely on ECC features. With any file system, errors can happen in RAM which then get written to disk. Because one of the major ZFS "selling points" is that it has full data integrity, it's advised that users use ECC for production systems. Otherwise you run the risk of people thinking their data is 100% protected, when there is still actually a small chance of corruption. With other file systems no real guarantee of integrity is given in the first place. If HAMMER is supposed to provide great integrity, then I suspect ECC is advised there too.
Click to expand...
This is basically what Matt Ahrens said to a group of us asking questions after his BSDCan 2014 talk.
 
  • Thanks
Reactions: Oko
jrm said:
I have no idea what sort of defaults FreeNAS or TrueNAS use, but from my experience running ZFS on somewhat older desktops and laptops with less RAM (I've never gone below 3 GB) isn't a problem. I'm not talking about several TB drives with dedpuplication turned on and I do some tuning.
Click to expand...

Agreed. The needs of a business are not always the same as the needs of a user running ZFS on a couple of servers at home. A home user often doesn't have the expectation or option of customer support and may not need or even desire all the features ZFS offers. As has been said already, ZFS can and does work fine with <8GB and even <4GB of RAM with some tuning when needed depending on the features needed/required according to the specific use case. It runs fine on my Laptop with 4GB of RAM with no tuning at all.

I am eager to check out HAMMER2 when it is finished though. :)
 
I missed few questions directed at me so I will try to answer them here.

usdmatt said:
If HAMMER is supposed to provide great integrity, then I suspect ECC is advised there too.
Click to expand...
Actually ECC RAM is advisable for HAMMER as well:) Matt has never been shy to say that HAMMER requires serious resources. It just requires less RAM than ZFS. On another hand I am surprised that people take this fact so hard. Look at this way. With ZFS you need $100 HBA controller with DF you need $600-$700 HWRaid card. Difference of $600 can buy more than enough RAM for ZFS.


usdmatt said:
Overall DragonFlyBSD just doesn't seem to have the user base or support for me to be happy using it for anything serious.
Click to expand...
Even though that Matt keeps repeating that he likes to keep it small this was actual main reason I removed DF server from production in my Lab. I just felt overwhelmed with a slew of trivial issues which could not have been addressed due to the tiny user/developer base.

usdmatt said:
And are there any sources to backup up the "phenomenal SQL performance" or it "beating the pants off everything else around"?
Click to expand...
Probably not of the quality you would expect from the serious project. I am looking forward to new benchmarks. The last serious one came 2012.
 
I'm definitely not a filesystem expert and know next to nothing about HAMMER.

ZFS on PCBSD as of 10.1 is tuned to use very little RAM (relatively) in the install: http://blog.pcbsd.org/2014/11/pc-bsd-10-1-release-now-available/

I'm not trying to say this is applicable to large arrays, but I have a couple laptops and desktops running PCBSD with low RAM usage. They run better now than before the tune.

Here is another link to ZFS running on low memory with some tuning: https://wiki.freebsd.org/ZFSTuningGuide
 
I was running OpenSXCE for a couple years, I beleive it was between 2004-2008. The machine had 4GB RAM and it used ZFS on a 72GB boot drive, a 120GB work drive and two 1TB drives running as a ZFS mirror. This computer ran 24/7 and had to be rebooted every couple of weeks because ZFS never released it's cache, it just filled up until everything was using swap.
If that issue is resolved than I will consider using it again.
 
Is there anyone out there using HAMMER in production then?

The thread seems to have become more about defending ZFS's honor than providing additional comparisons and use case examples from the real world.
 
  • Thanks
Reactions: Oko
Status
Not open for further replies.
Back
Top