Other Build a ZFS NAS (no, I don't want FreeNAS :-))

T

tryingagain

Good evening :)

To introduce myself in short: I was on the old forum some years ago. Being a FreeBSD noob, a great man helped me tremendously and patiently to make the move to FreeBSD. Then the forum changed, I couldn't log in to the forum any more and I had to create a new nick and become a newbie again (SunTzu says: become an newbie regularly:D). I never made it into FreeBSD, although I tried hard. It was always this program I couldn't get a suitable replacement for, that time lacking, that conversion (data: email & MS Office) not working, media not playing, this port fighting with that dependency, and then some more time lacking. I adore FreeBSD, I donated 3 times to The Foundation, yet I never had it live here.

Not to make you fall asleep with more of this boring stuff: the topic at hand:
  1. I am happily running Synology NAS-ses. Despite all the 'commercial crap' comments you can read in so many places, I am very satisfied with it. It works, support is good and committed, kind people, dedicated: no comments here at all.
  2. However, I am running out of space. Upgrading to the 12 bay Synology becomes expensive (I need more than 1).
  3. I tried FreeNAS, and, in the process for looking for replacements I discovered ZFS, and I think (I think...) I would want that.
  4. I managed to get FreeNAS to work (the system itself is easily installed, yet getting the packages to work is a disaster).
  5. But: I really dislike the whole atmosphere on that forum. What they do there is not the true BSD spirit; they bash, don't help, but wipe you away with 'RTFM' (which is bad IMHO) and 'go ask your questions on the FreeBSD forum, you need to learn BSD first' (I've taken notice of the thread in this forum that says: take questions about FreeBSD derivatives to their forum - and I fully agree). These comments come from arrogant users - but also from arrogant mods.
  6. So I went plan B: I asked for a quote from the Mother Company: perhaps there they are kind, perhaps their business formula is: you have to pay to not get bullied.
  7. That was a journey on its own: I'll spare the details (I wasn't bullied, which makes sense as I was a prospect, but there were some other annoyances), but a 16 bay NAS 'can be had for 20k USD'. Right... (not... You can buy a new car for that kind of money).
  8. So, then I said to myself: 'self, can't you do it yourself, with The Mother, FreeBSD, itself?'
  9. I have no clue how difficult this would be - as a complete noob in these matters -, but this aside, I have one question:
Can I get it to work to get a NAS, 16 disks, ZFS, based on Vanilla FreeBSD, with a comfortable GUI? Or will it all be CLI-work?

Just to explain, I am not against CLI work, I do some of it (very simple, grep'n |, chown/chmod/rm stuff) in Debian which I play with in a virtual box, but a GUI:
  1. Makes it all just a little bit more comfortable;
  2. Many times, two clicks in the GUI is the same as 100 commands on the CLI (because these 100 commands are programmed to the GUI-buttons :D).

So: is it doable, for a noob, to get a GUI for package install, system monitoring, well, basically: the kind of GUI FreeNAS has, but then without the whole FreeNAS experience (which really was horrible - I'm not alone, I discovered, by the way, the forum is famous on many other fora around the internets), or without the whole TrueNAS experience (with prices so far away from reality one can only start to try to imagine what their vision might be).

Thank you for comments :)

Bye,
 
My small NAS is just plain FreeBSD. It does not have all the integrated features of FreeNAS, but it does what I need, and I did not have to learn how to navigate the web UI. For me, the commands are easier. The operating system is on a small SSD, and the storage is a separate RAIDZ1 array.

It might be possible to turn a FreeNAS system into full FreeBSD, keeping the web UI for the NAS functions. Whether that's worth the effort depends on the features needed. Really, it depends on whether the FreeNAS GUI actually makes doing those jobs easier than the command line. A basic NAS is just a network fileserver. It is adding, changing, and administering the different types of fileserver systems (NFS, Samba, FTP) that make it more complex. I mostly use NFS and rsync, so this is easy for me.
 
For GUIs around ZFS storage servers, there are three options:
  1. FreeNAS
  2. NAS4Free
  3. ZFSguru
I've never used any of them, and don't really see a need for them, but that's me. I live at the command-line. :)

One of my co-workers has been playing with ZFSguru and loves it.

Personally, I'd recommend doing it by hand first, so that you really learn how ZFS works at the lowest levels. Once you understand how pools, vdevs, datasets, snapshots, clones, scrub/resilver, disk replacement, send/recv really work, then you move to a GUI to simplify things. Starting with a GUI, though, will make things harder to fix when they break, as you won't have the solid foundation to work with.

Also, it's *very* possible to do a custom-built storage server for under $20,000 US. The largest expense will be the harddrives (around $5,000-8,000 US for 24x 2TB drives, depending on price). The chassis, motherboard, CPU, RAM, and SATA controller(s) won't be very much at all (maybe $3-5000 US).

We have no issues with SuperMicro storage chassis (45-bay 4U JBOD rackmounts, 24-bay 4U rackmounts, 16-bay 3U rackmounts, and 20-bay 2U rackmounts for 2.5" drives), SuperMicro motherboards for AMD, Opteron CPUs, DDR3 ECC RAM, and LSI SATA controllers.

Beware: once you start building your own, you'll have a very difficult time even considering buying things from name-brand companies as their prices will be at least 10x if not 100x or 1000x more expensive, and their features will seem very limited in comparison. :D

(I get calls from the large storage vendors all the time asking about our setup, and they don't believe me when I tell them the hardware and software costs for the amount of storage we have. And "Can you beat free?" causes them to hiccup/choke.)
 
Whatever solution you decide to go with in the end and while ZFS provides excellent data protection through redundancy, make sure you have some kind of a real backup solution for your data. Preferably offsite.

Fifty plus terabytes of data would be a lot to loose should the something happen. There was a guy not too long ago on the FreeBSD forums who somehow lost access to his data pool and was trying to recover it.

I also second the comment about learning the underlying tools and then branching off to the GUI. FreeNAS is very convenient, and a lot of very large companies use it to secure their data. It is also rock solid. But building your own and experimenting with the commandline tools is a good way to learn how they really work under the hood.
 
Thank you all three for your kind replies :)

I guess you are right with your recommendations.

What actually is in the Synology DSM (= operating system, Busybox), is:
  1. A frontend for a couple of servers (CUPS, and if you install packages, Sabnzbd, Transmission, etc) which you access from your client via the web-GUI (http://NAS/.../.../index.html etc);
  2. Some own built functionality I don't use (like Cloud station, I use rsync);
  3. The userland CLI, accessible via SSH;
  4. The 'File station', a tool to browse directories and set permissions. That of course is sometimes easier than chmod/chown, because you keep an easy overview of directories, which in the CLI is sometimes hard.

So, nothing that can't be done without a GUI, but:
  1. It would really require you to dive indepth into FreeBSD. You gotta hive time for that (the motivation is not the problem);
  2. I'm a little bit scared :( I think my tests with FreeBSD go back to 2002. I was a member here, on Daemonforum. I did manage to get FreeBSD installed, with a GUI and all, but it didn't work well. Then on to DesktopBSD which died, then onto PC-BSD which wasn't very good, then back to FreeBSD, and so on and so forth. After 12 years of trying, I still don't have FreeBSD.

But, in the end: I think you all are right with all your arguments: learn it, so you know what's going on.

You wouldn't happen to know of a recommended tutorial that you all consider to be top of the bill? That would probably save me 6 weeks of googling around, dismissing all the incomplete, erroneous, etc writings on the web untill all I want is :beer::beer::beer::beer::beer::beer::beer::beer::beer::beer::beer::beer::beer:

:)D)

Thank you again :p
 
In the end though, if what you really want is a set it and leave it NAS with a GUI, FreeNAS is really good at what it does. It also makes using all of those extra services that you want pretty simple with jails. I am a very happy FreeNAS user and never hesitate to reccommend it to others. It has sane defaults. The amount of time that it saves, especially if you want your NAS to do more than just samba or NFS shares, is enormous.
 
Thank you very much for the links, Gofer :beer:

gofer_touch said:
There are quite a number of FreeBSD and ZFS tutorials on the web. Including a number which guide you through selecting and building a NAS from a selection of hardware.
Click to expand...

I know, but for every 1 good guide, there are a 1000 bad guides (all the pfSense tutorials on the internets are famous for that. One example: at least 10 guides telling you how to set up VPN without even mentioning NAT and firewall rules. Duh).

gofer_touch said:
In the end though, if what you really want is a set it and leave it NAS with a GUI, FreeNAS is really good at what it does. It also makes using all of those extra services that you want pretty simple with jails. I am a very happy FreeNAS user and never hesitate to reccommend it to others. It has sane defaults. The amount of time that it saves, especially if you want your NAS to do more than just samba or NFS shares, is enourmous.
Click to expand...

It may very well be very good, I don't argue that (yet). As a matter of fact, I have it running here on a box (not VM), and it runs. But the manual leaves so much to be desired, that you have no other choice but to ask in the forum. Where, as I shared my experience in the above, you get bullied and blasted away by arrogant mods and users, telling me to go ask here in this forum about a solution to a problem with their product. All in all: this is more the Linux experience where I've seen this so many times, yet never in the BSD fora where people are kind, and try to help.

Then the commercial solution I enquired about is the same problem with an attitude of the same kind (well, like I said, a little less insulting, probably because someone there realises he wants me to pay him 20K).

I have no problems paying for stuff: I do understand 'free' needs money too: I donate to the FreeBSD foundation, to pfSense, to XBMC, to Debian, and to some other things. Not because I have to, but because I want to and am grateful for what I am offered by kind developers without having to pay for it. So I want to do something back.

So, naturally, if this whole FreeNAS club would have been polite, and kind, it would have inspired trust, and I would have moved on on this road. And naturally then too, would have donated. But am I to put my digital assets on a system with such lousy treatment of people? So here's the pics and vids of my by now deceased family. On the FreeNAS. And then something happens. And I need support. And I am bullied away with a 'go ask on the FreeBSD forum'. Which, in turn, rightfully, says to don't do that here.

It's about trust, and I don't trust that club anymore.

The better way is to then go Vanilla BSD, find help in the nice BSD and Daemonforums, buy the fine members who helped you a coffee/tea/beer via Paypal, and donate some more to the FreeBSD foundation. Is my humble opinion :)

gofer_touch said:
Here is yet another tutorial for you:

Building a FreeBSD NAS, part III: ZFS
http://bastian.rieck.ru/blog/posts/2014/freebsd_nas_part_iii/
Click to expand...

Thank you again, Gofer :beer:

Edit: I'm not here to bully the freenas club, just trying to explain why I could never trust my digital assets (some of them, memories of my deceased family, more valuable than my hard assets) to such an ecosystem.
 
Wow, ok. I didn't realize that they had that kind of a reputation. It is a definite turnoff to be teased and mocked for asking for advice or tips. It is truly sad when that happens. If people don't have anything polite or meaningful to say, they shouldn't say anything at all. Its a waste of time for everyone. In all honesty I've never actually asked anything on the FreeNAS fora.

If you have the time, setting up a FreeBSD-based NAS is truly an enjoyable learning experience. It will be rock solid and you will wonder why anyone would use something else when such a good freely available solution exists.
 
FreeNAS is a great product but might not be suitable for all users. Users who have a very narrow set of requirement might find it easier to use vanilla FreeBSD without all that overhead. Users who need any kind a serious customization will also find vanilla FreeBSD easier to use since in FreeNAS they will be fighting GUI all the time. If you like Synology you will like FreeNAS.

I use FreeNAS, TrueOS and vanilla FreeBSD in production. Being away from almost 7-8 years form vanilla FreeBSD (I am predominantly OpenBSD user) I found easier at first to deal with FreeNAS. It has very sane defaults and helped me learned a lot. As my knowledge of vanilla FreeBSD got updated I found vanilla FreeBSD easier to use in many situations. I also realized that my knowledge of FreeNAS got serious bust from understanding how vanilla FreeBSD works. So learning FreeNAS helped me get idea about sane defaults for FreeBSD ZFS storage solutions and vice verse working more with vanilla FreeBSD helped me become more proficient FreeNAS user.

As of FreeNAS forum. There are several guys some of them forum moderators who have military background and some familiarity with Windows but little or no familiarity with UNIX. Their netiquette leaves much to be desired. Frankly I would just ignore them all together like a noise as they are clueless anyway.

FreeNAS developers frequent forums including Jordan Hubbard. If you have anything that resembles a bug they will be quick to answer your question and participate in the thread. Dru Lavigne wrote their documentation. It is stellar IMHO. She is answering users questions as well.
All in all I know exactly what annoyed you but my recommendation based on your original post is not to give up on FreeNAS just because of the noise on their forums (trust me I know that those 3-4 guys answer every thread and I still have to see a single intelligent thing coming out of them).
 
gofer_touch said:
If you have the time, setting up a FreeBSD-based NAS is truly an enjoyable learning experience. It will be rock solid and you will wonder why anyone would use something else when such a good freely available solution exists.
Click to expand...
I have a somewhat incomplete article about my RAIDzilla 2 design here. It doesn't go into the more recent changes like switching to 10 Gigabit Ethernet, but you may find it an interesting read as it goes into detail about hardware monitoring, replication and backup, and so forth.
 
Try ZFSguru at zfsguru.com. I am a Windows user and ZFSguru helped me learn FreeBSD and ZFS in 2011. Under the ZFSguru web interface there is a true vanilla FreeBSD. When you are tired with ZFSguru you can use the command line or the GNOME 2 graphical interface. Now, on my office server, I am using for 90% the command line interface via ssh, 5% GNOME for VirtualBox and 5% ZFSguru for disks, yes I still prefer the ZFSguru web interface for managing disks ;).
 
I would like to thank you all very, very, much :beer:

This now is the BSD community forum spirit that I know from this place, ever since back in the days, and that I know from the pfSense forum too: let's try to help each other, instead of the hostile environment over at FreeNAS.

So, having carefully considered I've decided to indeed forget about FreeNAS, that is a dead end for me. So that leaves:
  1. ZFSguru (but I thought that was 'rather old', and less 'mature and stable'? Have I read wrong?)
  2. NAS4Free (but that forum isn't very active I believe(?)).
  3. A NAS built on VanillaBSD.

More thinking to do :D

In the mean time, I decided I should be able, after 12 years of trying, to get FreeBSD 10.1 installed. I installed it in Virtualbox. The first part was a breeze. Then of course comes the GUI. That took a lot of googling, but I managed to install mate ( pkginstall xorg mate && change some config settings && reboot). However, that contains very little apps/programs by default :)oops:). I seriously dislike Gnome (been playing with that on my Debian box), so then it should have been KDE4 (which I also don't particularly like, but anything is better than Gnome to me), but that isn't available as a package (huh?)

What are you all running then, if I may ask? No fancy eyecandy GUI but the "90's" GUI such as XFCE? I'd love an OS-X GUI, running on FreeBSD :p

Once again: thank you very much for your comments; much appreciated (where is the 'kiss'-emoticon? :D).

Bye :beer:
 
Last edited by a moderator:
Oko said:
All in all I know exactly what annoyed you but my recommendation based on your original post is not to give up on FreeNAS just because of the noise on their forums (trust me I know that those 3-4 guys answer every thread and I still have to see a single intelligent thing coming out of them).
Click to expand...

It's a great explanation, Oko; thank you :)

The thing is:
  1. You obviously have way more skills in this area than I have --- you're a 'die hard BSD man' I read;
  2. You obviously have way less nerves problems than I have :D

As a rookie in these matters I am expecting to need some help along the road. The 'support people' over there are useless (the rude mods over there, I mean; and Dru is mostly replying to threads - at least those I read - with a 'have you fixed it already', so that is useless too). So I'd rather build it from scratch with VanillaBSD knowing there are kind people in here who can help me along the way, than install a point-and-click FreeNAS where, when the sh*t hits the fan the mods and Dru are doing their best not to help (:oops:). Imagine all my digital assets threathened - or even gone - due to a degraded pool (these assets are important life memories), and the answer being 'go ask over at the FreeBSD forum' or 'you need to learn FreeBSD, noob!' (answers I got in the past...).

So for a man with your wisdom FreeNAS and TrueNAS may be an option, since you know so much yourself. For me, as a rookie, I think it is not an option.

Which is a scary thought: an 'easy NAS for everybody' that is for ... not-rookies (grin).

Bye :beer:
 
There is one thing which is not trivial to replicate with vanilla FreeBSD when it comes to FreeNAS installation. People tend to forget that the base for FreeNAS is actually NanoBSD which is FreeBSD version for embedded devices. Customizing NanoBSD is not for a faint of heart. On another hand NanoBSD is the way to go when it comes solid state drives SSDs. Namely even the expensive modern SSDs use NAND-based flash memory.
NAND or for that matter any flash memory can be written on only limited number of times typically around 10 000 before it dies. It has some other nasty properties. If the medium is full it will effectively die. As you rw more and more it will slow down and become effectively slower than regular HDD with movable plates. That is why manufacturers have deployed very clever algorithms to make number of writes evenly spread across SSD and even reserve 7% of the medium space to prevent overfill. That is why you have to use NanoBSD not vanilla FreeBSD when it comes to SSDs.

If you want to install FreeBSD on an embedded SSD (or SSD mirror) and have all data on HDD you will have to be very skilful in customizing NanoBSD to make sure OS is mounted with read-only option and all OS specific writes like log files and such are done in the RAM. The easiest way to avoid all that hassle is to use FreeNAS. FreeNAS now even uses ZFS for OS and has ability to mirror it. The installer really rocks.

I would really like people to correct me if I am wrong in this post as it is of great interest to me and I did some investigation on the topic but I am not sure I am getting things 100%.
 
Oko said:
The easiest way to avoid all that hassle is to use FreeNAS. FreeNAS now even uses ZFS for OS and has ability to mirror it. The installer really rocks.
Click to expand...

Ceteris paribus you are right, Oko, unfortunately, I remain at:

tryingagain said:
As a rookie in these matters I am expecting to need some help along the road.

The 'support people' over there are useless (the rude mods over there, I mean; and Dru is mostly replying to threads - at least those I read - with a 'have you fixed it already', so that is useless too).

--- when the sh*t hits the fan the mods and Dru are doing their best not to help. Imagine all my digital assets threathened - or even gone - due to a degraded pool (these assets are important life memories), and the answer being 'go ask over at the FreeBSD forum' or 'you need to learn FreeBSD, noob!'
Click to expand...

I don't think I need SSD; I don't have it in my Synologies, and I also didn't have it in the FreeNAS testbox. For SOHO it was fast enough.

It is interesting what you wrote about the SSD's; does this mean for my desktop I also shouldn't use an SSD? I currently have Crucial M4 in my W7 desktop, is the great FreeBSD less capable of handling that than the crappy W7? Or does it only apply for this specific NAS-situation?

Thank you again, Oko :D
 
tryingagain said:
It is interesting what you wrote about the SSD's; does this mean for my desktop I also shouldn't use an SSD? I currently have Crucial M4 in my W7 desktop, is the great FreeBSD less capable of handling that than the crappy W7? Or does it only apply for this specific NAS-situation?
Click to expand...
There are a number of different things at work here. Flash chip storage blocks can only be [re-]written a finite number of times. In order to deal with this and still have a usable product, SSDs use a fancy function in the controller called "wear leveling" which means that it keeps track of how many times the blocks have been written and tries to even this number out across the whole capacity of the drive. Unlike on a hard disk, where "LBA 0" always refers to the same spot on the media, on a SSD it will move around if you keep writing to it. That's just some of the magic that happens "behind the curtain". Some drives also have spare blocks which are not visible to the host system, only to the controller chip in the SSD. With these features, a SSD should be able to withstand many repeated write cycles. For example, my 250 GB Samsung Evo 850 is rated for 75 TB of writes.

Even so, that's a finite number and you don't want to be writing un-necessarily. On a Windows 7 box, for example, you shouldn't defragment your SSD (it won't improve performance anyway). On a FreeBSD box, you don't want to be writing scratch data to the SSD if you can avoid it. Use something like a md(4) device instead. Things like security logs are important and you still want to keep them on the SSD, even if you think you'll never need them.

There is a SSD function called "trim" which both Windows 7 and [recent] FreeBSD support. With a regular file deletion, all that the drive will be told is to clear some information from the directory and allocation bitmap (or equivalent metadata, depending on the filesystem in use). This means that the SSD will eventually "fill up" all of its storage blocks (due to the aforementioned wear leveling) and when it needs to write new data, it must first erase some storage blocks. That takes time, and it would be better to do in the background when the SSD isn't being heavily used. The trim function tells the drive that the operating system doesn't need certain blocks any more, and the drive can go ahead and erase them when it isn't busy doing something else.

That's the way SSDs work. But many people configure systems to boot from other types of flash storage - USB memory sticks, Compact Flash cards, etc. These don't have all of the fancy wear-leveling features that SSDs do (after all, they're a lot less expensive - one eBay seller I know of throws an 8 GB flash drive in for free on every order). So you can easily wear out one of these with repeated writes. A lot of the "be very careful" type warnings come from people who have had bad experiences with rapid wear on USB memory sticks or other types of flash cards, and the warnings are not that critical with modern SSDs. Of course, you still want to avoid unnecessary writes to the SSD when possible.

I hope this has been informative.
 
You must log in or register to reply here.
Back
Top