Pooled storage

AngusBuchanan

AngusBuchanan

Hello,

I'm interested in putting together a system for hosting a large number of jails. It's for a non-profit, so budget is tight, and i'm probably going to be using 2nd hand or reconditioned kit from a variety of sources. The idea is to net boot FreeBSD on nodes with reasonable amount of memory, and then mount storage either via NFS or perhaps iSCSI.

It's the storage bit I'm interested in particularly. This may be impossible, but my dream would be to have a pool of storage which spans machines. So say I started with a 2 U box with 8 320G disks in it. If someone donated another 1U box with 4 x 160G disks, I'd like to be able to add that storage to the pool, and start to take advantage of more spindles and more space.

Is this even possible? Or is it just a crazy dream :)

I know GlusterFS does something similar, but I'm not sure if that will work on FreeBSD. And I wonder if there's something lower level? I had a look at HAST but that seems to be a different use case altogether.
 
Hi,

yes you can do exactly that with ZFS, have a read up on it and come back if you have any questions or need a bit of help:

http://en.wikipedia.org/wiki/ZFS

http://wiki.freebsd.org/ZFS

On FreeBSD iSCSI seems to suffer from poor performance when used in conjunction with ZFS (using istgt port) that don't seem to be present on Solaris based systems. You should be ok with NFS thought, again performance may not be stellar but it should be ok (depending on what your requirements are!),

thanks Andy.
 
PS sorry I just reread your original question. ZFS allows you to pool all your storage when it is connected to one server, if you add another server with direct attached disks then this cannot be pooled (with ZFS) to any other server. You need to share via NFS or something...
Sorry for any confusion!
 
Yes, this is possible, using a network connection to combine multiple "storage nodes" to a master "head node". But it requires 2 machines to start with.

Configure your "storage nodes" as per normal, using either hardware or software RAID to combine your disks into one (or more) array. Then export that array via iSCSI. (Alternatively, you can export each disk individually.)

On the "head node", use the iSCSI "disks" to create mirror or raidz vdevs inside of a single ZFS pool. Create your filesystems as per normal, and the share those out via NFS.

Be sure to use two separate networks, one that only connects the storage node to the head node. And a completely, physically separate network for the NFS shares. You don't want NFS traffic interfering with iSCSI traffic.

When you get another storage node, just create the iSCSI exports, and add them as another vdev in the ZFS storage pool.
 
Another option, if you have access to long SATA cables (I think 3m is the max?), and your initial storage box have a lot of PCI-X/PCIe slots is to create a massive storage box using ZFS.

Start with the onboard SATA slots and the harddrive in the server.

Then, when you get another box, just run SATA cables from the initial box to the harddrives in the other box. You don't even need a motherboard in the box, just a power supply and harddrives. :)
 
Thanks both. Andy - I was aware of ZFS pooling on one machine, but wasn't sure if it was possible to span machines. Phoenix - the first method sounds like exactly what I had in mind - I am sure I read about this on Solaris using COMSTAR. The FreeBSD wiki suggests that iSCSI isn't implemented yet. I assume this means natively - as there's the NetBSD iSCSI target in ports.

Has anyone here had a go at doing as Phoenix suggests (with ZFS) - any idea whether this is likely to incur big performance penalties?

Your second idea intrigues me. I've already got rack space allocated (a whole rack) so I don't see why I couldn't buy the cheapest possible case and fill it with SATA disks, and cable them up to a box with a stack of SATA controllers. I guess bus contention might hit me - so the head node would need reasonable CPU, and of course the cabling could get messy, and swapping and identifying faulty disks could be interesting, but as a way to get maximum value out of minimum spend it's a really interesting idea.

Any thoughts / feedback?
 
Hi,

as mentioned, I have iSCSI (using istgt) on top of ZFS implemented. My performance is very poor, but sufficient for my requirements. From researching the subject I've come to the conclusion that istgt is not well optimised for ZFS. That is, I have read of other cases of poor performance of istgt with ZFS, and not of any success stories. Solaris has the iSCSI target builtin and it would seem is (better)optimised to work with ZFS,

thanks Andy.
 
Why is iSCSI performance poor? Do you run iSCSI on a seperate vlan with jumbo frames? What speed do you get?
 
AngusBuchanan said:
Your second idea intrigues me. I've already got rack space allocated (a whole rack) so I don't see why I couldn't buy the cheapest possible case and fill it with SATA disks, and cable them up to a box with a stack of SATA controllers. I guess bus contention might hit me - so the head node would need reasonable CPU, and of course the cabling could get messy, and swapping and identifying faulty disks could be interesting, but as a way to get maximum value out of minimum spend it's a really interesting idea.

Any thoughts / feedback?
Click to expand...
I'd definitely look into getting a case which allows you to view the drives, at least enough to slap a label on them. The most likely way that you're going to end up losing data is if you pull the wrong drive.

You might take a look at this if you decide to go the DIY route. It's a lot more storage and expense than you're looking for, but it should give some insight into how to go about it. Petabytes on a budget:
How to build cheap cloud storage
 
You must log in or register to reply here.
Back
Top