SAN (FreeBSD + ZFS + HAST + CARP + iSCSI) Question

D

DungeonMaster3000

Hi Everyone,

I'm in the process of building a SAN (still very much in the testing / tinkering stage) based on FreeBSD + ZFS + HAST + CARP. I'm relatively new to FreeBSD (and this is my first post) but have been busy reading and building test after test after test in a virtual environment so I have at least a decent idea of what I'm doing before I touch real hardware.

I have read through many message board posts and Google searches but still have a few questions I'd like to ask here.

I'd like to build two boxes running on consumer hardware (primary node and secondary node); one that utilises SSD's for Cache and Log devices and the remaining drives are standard consumer HDD's. The primary box would be have the extra hardware (more RAM + better CPU + SSD's Cache & Log devices) and the secondary / fallback box would be just consumer HDD's only to be used temporarily while the primary is down; getting fixed etc.

After successfully building the initial version of the two boxes (again, virtually) and testing with HAST and CARP I was hoping someone could answer the below questions.

- If both boxes have multiple NICs (crossover between nodes [for HAST replication] and CARP on the other interfaces to the switch), how can I tell HAST which IP/interface to use?

- If the backup node doesn't contain Cache & Log devices; will I still be able to import the zpool on failover?

- Is HAST what I want to use? Am I better off using snapshots and an SSH tunnel between the two nodes?

- If someone has done similar, if you don't mind sharing any pointers, gotchas etc; I'd love to hear them.

Many Thanks,
Tarran
 
DungeonMaster3000 said:
- If both boxes have multiple NICs (crossover between nodes [for HAST replication] and CARP on the other interfaces to the switch), how can I tell HAST which IP/interface to use?
Click to expand...

The /etc/hast.conf includes the hostnames (or IPs) of each box. You make sure that your CARP interfaces are on one subnet, and your HAST interfaces are on a different subnet (preferably, a separate physical network).

Code: 
[slave] --*-- [master]
   |             |
   \---[switch---/
          |
      [clients]

Where * is the HAST subnet, either direct cross-over cables, or another switch (or vlan, or however you want to split it off), and the interfaces connected to [switch] are configured via CARP, so that [clients] always see/use the same IP to connect to the SAN.

- If the backup node doesn't contain Cache & Log devices; will I still be able to import the zpool on failover?
Click to expand...

No.

You will need to have at the very least the same log device configuration on each box.

You can import a pool with a missing cache device, since the data on the cache is volatile (doesn't survive reboots, it's just a read cache).

But you will have issues trying to import a pool with a missing log device, and you can lose data that way (data that's written to the log device but not yet to the pool). There are ways to import a pool with a missing log device (if using ZFSv19 or newer), but it's not worth the risk of data loss.

You will want to use HAST on the log device as well.

- Is HAST what I want to use? Am I better off using snapshots and an SSH tunnel between the two nodes?
Click to expand...

For a master/slave fail-over storage setup, HAST is what you want.

- If someone has done similar, if you don't mind sharing any pointers, gotchas etc; I'd love to hear them.
Click to expand...

I haven't done a true SAN setup (or even really a NAS setup), but HAST does work quite nicely for our current uses. 2 boxes with identical storage setup (24x harddrives for pool, 2x SSDs for OS, swap, cache) but different motherboard, CPU, RAM. "Master" box has the better hardware, and does all the backups services via rsync. HAST replicates that to the "slave" box. If CARP detects the network is down, HAST fails-over to the "slave" box.

You will want to use 8-STABLE (after April 11, 2011), as that includes a bunch of fixes for issues with HAST and multiple harddrives, stability, and throughput issues.
 
Hi,

DungeonMaster3000 said:
- Is HAST what I want to use? Am I better off using snapshots and an SSH tunnel between the two nodes?
Click to expand...

It depends on your requirements. For example if you use the snapshot route you don't need to have identical hardware, so you can get away without having SSD log disks on the standby server. The main advantage of using a HAST setup would be that it allows synchronous and nearly synchronous modes of operation. Although if you care about performance enough to want to use SSD disks for the ZIL then you are probably going to have to plump for the memsync HAST mode or you will kill your performance, so you will have an asynchronous config. Without getting into that, if its really important to you that the data is totally up to date on both systems then you really need to use HAST. If you go the snapshot route then at best the data on the slave is going to be a few minutes behind your master.

The up sides to the snapshot route would be, I think the master server performance will be less impacted than if using HAST. You don't have to worry about ZFS needing to recover from any semi committed writes or anything like that, so probably you will have less issues when failing over. You are relying on only one technology, so theoretically it should be more reliable.

cheers Andy.
 
Thanks Andy for your reply, I appreciate it. I think I'm going to have run a few more performance test but from your (and Freddies) reply, I believe HAST (with similar hardware) is the route I'm going to go down as need the data to be identical.

If you guys are interested, I'll keep you posted as I plan on documenting the whole setup with configuration and articles I read to get there.
 
One other thought. If you are serving 100% iSCSI targets, and you will use zvols then I believe you can replicate the zvols via HAST rather than the physical disks. Using this approach, again, you can avoid having identical hardware, ie you could skip the SSD drives on the standby. The implications to performance is another matter of course and will depend on many variables :P

If you get something working Im sure a few people would be interested to hear about it on the forum...

cheers Andy.
 
Hi Tarran,

did you get your setup working? I would like to build a similar infrastructure so your "howto" would be highly appreciated.

Thank you.

Petr
 
Sorry for the (very) late reply - I pretty much waited for 9 to be released before implementing after months of trying different solutions and scenarios… (Don’t judge, but even considered Windows Storage Server 2008) I ended up using HAST on zvol’s instead of disks so I could have the primary server as high performance machine and the failover as a good enough while I repaired the primary… I’ll probably be crucified for saying this but there is a performance gain on HAST hdd to hdd however for me at least it seems mainly academic compared to zvols for my environment (excluding SSD ZIL).

To implement what I have is after the combination of reading countless posts, ponderings & blogs, then sort of mixing them all together to what seemed to fit best for me. Unfortunately, I don’t have a blog or anything like, so if DutchDaemon doesn’t mind and others are interested I’ll post my setup on the end of this post.
 
You have my blessing, if that's what is needed. Don't forget to format ;) If this takes the form of a how-to or manual, posting it in the HowTo & FAQ forum may be more applicable.
 
You must log in or register to reply here.
Back
Top