ZFS 24 drive array suggestions wanted

Phishfry

Son of Beastie

Reaction score: 1,180
Messages: 3,417

I am back to my 24 drive ZFS project and I can't seem to find a scheme that pleases me yet.
RAID60 is where I am right now with 12 disks in striped 6xRAID-Z2 array. I was thinking of adding in my 3 NVMe as cache as I am messing with graid3 on them now for comparisons. That still leaves 12 drives to provision. Thinking of further nesting with raid600 or raid601.

RAID-Z3 is more protection than I need. Maybe I am wrong? Is using my 3 NVMe for ZFS cache the best use?
Maybe instead use 2 for cache and one for zil slog?

24 SAS2 disks of 450gb size was not the best arrangement, but I don't have much money into it.
I am just trying to learn how to work with disk cabinets and different storage arrangements.
I am using 3ea. SAS2-9207-8i controllers in a Chenbro 2U rack chassis.

Somewhat obsessively I bought a second 24 bay cabinet for 60 bucks. SuperMicro/Chenbro OEM
It has a different backplane arrangement for 8087 cabling with 3X8 drives versus my current 2X12 drive backplane.
I like the logical grouping of 3x8 as it jives better with the controllers being 8 ports.

Currently 24 drives are too much for a home lab but it does offer alot of flexibility.
Thinking maybe mirror these 2 storage boxes with 12 drives each for testing clustering filesystems like ceph and gluster.
I have a few dual port drives and would like to experiment with those. Especially on Supermicro EL2 backplanes.
That is a future project with SAS3 SSD's.
 

phoenix

Administrator
Staff member
Administrator
Moderator

Reaction score: 1,215
Messages: 4,066

We use 6-disk raidz2 vdevs in all our storage boxes (24 bay 4U servers and 45 bay 4U JBOD chassis).

Our biggest servers have 2 JBOD attached for a total of 90 drives. These connect to a 2U server with SSDs for the OS, cache, and SLOG. We use LSI 9xxx-8e HBAs for these. The SSDs use the motherboard connectors.

The JBODs use backplanes with expanders to share 4 SATA lanes across 24 drives (front) and another 4 SATA lanes across 21 drives (back).

The 24 bay chassis use direct-attached backplanes using multi-lane (SFF-8087) connectors. That gives the best mix of dedicated connections for each drive and least amount of cables. Most are Supermicro, with a pair of Chenbro. These use LSI-92xx-8i HBAs.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,180
Messages: 3,417

No I am thinking about upping my count to this: 3 X 8 drive raid-Z2. Right now still messing around.
When my other 24bay case comes in may pull the 3X8 drive backplane out and use that instead.
Then I have a separate controller and separate backplane for 3 each -8 drive zdev's.
That seems like the best I can come up with.

I hate to say but reddit had some excellent conversations which were useful.

Still reading on ZIL and SLOG. Not sure where to go with my 3 NVMe's.
Seems a 512GB NVMe drive size is a waste for SLOG but you want separate devices for SLOG and ZIL
Not totally sure where L2ARC comes in yet. Still trying to comprehend all this.

https://forums.servethehome.com/index.php?threads/partition-same-nvme-for-both-slog-and-l2arc.19700/
https://www.ixsystems.com/blog/get-maxed-out-storage-performance-with-zfs-caching/
https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/
 

justinnoor

Member

Reaction score: 45
Messages: 81

Sounds fun! I’m not familiar with the 3x8 array. You might be interested in Lucas chapter 2, which has some great performance charts for just about every array. Calomel has also done some excellent work on this.

There is a selection of 128GB nmve drives on Amazon that may interest you. Here’s a couple, and there’s more on Amazon:

Patriot Scorch 128GB https://www.amazon.com/dp/B07951BDZH/ref=cm_sw_r_cp_api_i_ufflCbGZXHDXE

XPG SX8000 PCIe 128GB https://www.amazon.com/dp/B06Y2R3YXP/ref=cm_sw_r_cp_api_i_RgflCb8YXY0SH

Yeah I’m researching caching myself, but I have all SATA lll interfaces. Storage is so shockingly cheap these days I’ll probably go with a separate SSD for slog, l2arc, and what the hell, swap too hah. Not sure if I should mirror those, that might be overkill. We’ll see.

Happy building
 

phoenix

Administrator
Staff member
Administrator
Moderator

Reaction score: 1,215
Messages: 4,066

Note: every pool had a ZIL (ZFS Intent Log). By default, it's part of the existing vdevs in the pool. But, you can configure a separate vdev as a log device, which creates a Separate LOG (aka SLOG).

IOW, ZIL and SLOG refer to the same thing.

Also, if you go with 6-drive raidz2 vdevs then you can assign two drives from each controller to each vdev. That way, if a controller dies, the pool remains operational (each vdev runs in a degraded state with 2 missing drives). That allows you to do an ordered shutdown to replace the dead controller, without losing any writes, data in flight, or corrupting running daemons.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,180
Messages: 3,417

IOW, ZIL and SLOG refer to the same thing.
Thats what I figured. So SLOGS are created with zpol create log /dev/xxx and this would be considered "in-pool" ?
But, you can configure a separate vdev as a log device, which creates a Separate LOG (aka SLOG).
And this would be considered "out-of-pool"?
 

justinnoor

Member

Reaction score: 45
Messages: 81

EDITED

new
When using dedicated SSDs for a log vdev, and a cache vdev (for slog and l2arc), should they be mirrored? Therefore the total number of disks would be four.

old
When using separate, dedicated, disks for SLOG and L2ARC caching, would it be overkill to mirror those devices (raid1)? Therefore the total disks for both caches would be four.
 

phoenix

Administrator
Staff member
Administrator
Moderator

Reaction score: 1,215
Messages: 4,066

Thats what I figured. So SLOGS are created with zpol create log /dev/xxx and this would be considered "in-pool" ?
No, that creates a separate log vdev, aka SLOG, and uses that for the ZIL.

When you create a pool, a ZIL is created automatically using the existing vdevs(s) of the pool.

All ZILs are in-pool. You can't write data for one pool using devices on another pool. Each pool is self-contained.
 

phoenix

Administrator
Staff member
Administrator
Moderator

Reaction score: 1,215
Messages: 4,066

When using separate, dedicated, disks for SLOG and L2ARC caching, would it be overkill to mirror those devices (raid1)? Therefore the total disks for both caches would be four.
Log vdevs should be mirrored to provide extra protection as they contain data that may not be written to the main data area of the pool yet.

zpool add poolname log mirror daXp2 daYp2

Cache vdevs hold copies of data that's already in the pool so they don't need to be mirrored. But you can create a cache vdev using multiple drives striped together for more throughout.

zpool add poolname cache daXp3 daYp3
 

justinnoor

Member

Reaction score: 45
Messages: 81

Great, thanks. Sorry I worded the previous post incorrectly, as if log vdevs and cache vdevs are the same. Thank you both for the awesome thread. Phishfry good luck on your build.
 
Top