Solved Question re ZFS properties setting for a movie server

M

mefizto

Greetings all,

I am assembling a server that will be used strictly for storing and serving (to one or two users), movie files. That is, no other file types will be stored therein and, apart for deleting a movie from time to time, the content will be immutable.

The search results appear to discuss mainly two settings compression and recordsize, some throwing in ashift, probably just to confuse the already inconclusive, or sometimes outright contradictory, conclusions.

So, I was wondering if someone, preferably one who has already set-up and run such a server and thus has some experience, could advise me on setting the parameters appropriatelly.

Kindest regards,

M
 
I use the XigmaNAS app on FBSD 13.1 and ZFS2
The default compression method is no impact on performance.
The 1M recordsize is more efficient for large files such as video files.

Advice: build the absolute largest amount of storage you can possibly afford.
Mine is six 4tb WD Red NAS as ZFS2, which is 16tb usable, and 8tb as check drives.
I am out of space, and have to build another one for the overflow.

Six SATA is the max on my Supermicro board.
I boot from a thumb drive which works flawlessly.
 
Hi bgavin,

thank you for the reply.

It is spooky, I have exactly the same hardware setup: Supermicro server board with ECC, six 4TB hard-drives, ZFS2, OS on a thumb-drive. I estimate to have about 10TB worth of movies, once I am reach 80-85%, hopefully the price of bigger hard drives will be more reasonable and I will rebuild the array.

If I understand you correctly, you recommend compression=lz4 and recordsize=1M, is that correct?

Kindest regards,

M
 
Compression setting likely won’t matter much; you’re storing movies which are already compressed; the compression will fail (at achieving compression, and abort, which is not an issue) so the records will be saved without (filesystem) compression, which is just like if you had turned it off. (So a slightly higher cpu usage to attempt compression during write, no change during read.) I would leave at (very fast) lz4 anyway. Again, only a very slight CPU penalty during writes to “try” compression.

ashift=12 to match 4k physical sectors.

Larger (1M) recordsize for streaming large sequential streams (movies) is good, and will give you a bit more usable space by cutting down metadata overhead.

I’m assuming RAIDZ2 (not ZFS2) — some of the terminology is confusing, but that also sounds reasonable.

All sounds good. Enjoy ZFS!
 
Hi yuripv79,

thank you for the reply.

yuripv79 said:
See also zstd in zfsprops(7).
Click to expand...
Well, from what I read about the compression was, that the movie files are for practical purposes not compressible. Consequently, the ZFS compression algorithm aborts if it cannot achieve a preset compression target.

Now with acknowledging that, most people still recommend to set compression=on, which invokes the default compression, which is currently lz4.

So, what would be gained to us the zsdt?

yuripv79 said:
What's that?
Click to expand...
that, was a mistake, I meant RAIDZ2 as Eric A. Borisch correctly observed.

​

Hi Eric A. Borisch,

thank you for your reply.

Eric A. Borisch said:
Compression setting likely won’t matter much; you’re storing movies which are already compressed; the compression will fail (at achieving compression, and abort, which is not an issue) so the records will be saved without (filesystem) compression, which is just like if you had turned it off. (So a slightly higher cpu usage to attempt compression during write, no change during read.) I would leave at (very fast) lz4 anyway. Again, only a very slight CPU penalty during writes to “try” compression.
Click to expand...
As noted in my reply to yuripv79, your recommendation is in line with most of what I read. Although I would agree that the slight increase in CPU usage upon writing is not an issue; therefore, I have no problem setting it on, if the content will only contain movie files, what is the motivation to use a compression?

Now, I realized that I was lying by saying "strictly", because there will be a dataset containing a database of the movies' metadata, so I can do a search. However, since the compression is a dataset property and not a pool property, the question remains.

Perhaps the answer is ease of maintenance having the different datasets with most common properties?

Kindest regards,

M

P.S. I will mark the thread as Solved, but please feel free to continue, if desired.

M
 
Have you measured the performance with ashift=12 and otherwise all default settings? If it is sufficient (with safety margin), my advice would be to not do anything.
 
Hi ralphbsz,

thank you for the reply.

I am at the stage where I assembled the hardware and installed the OS; thus, before setting the pool/dataset properties. Hence my question, so that I can set them optimally.

Kindest regards,

M
 
Hi chrcol,

thank you for the reply.

That is, what I have, since I found in a database that the hard-drives are formatted to.

Kindest regards,

M
 
I would suggest the recordsize=1M on the filesystems serving movies for a few reasons:
  • Minor improvement in storage efficiency. Fewer levels of indirect blocks (storage metadata rather than user data on disk) are required for the same size user data.
  • Minor improvement in metadata I/O / metadata cache efficiently. Natural consequence of the above.
  • Minor improvement (reduction) in CPU usage for both compression (if left on) and checksums. As these can be performed on larger records, there are fewer function calls to process the same data.
Unless you have a performance / latency-sensitive DB running as well, it is unlikely you will find any downside. (I don’t think the DB of metadata for your movies would hit the level of IOs that you would notice, note you shouldn’t set that DB’s recordsize to 1M.

Anecdotally, I used to run MythTV with multiple capturing tuners (high def streams) that would also serve out simultaneously (like when you pause and resume live TV) and had a noticeable improvement in “user experience” with 1M over 128k recordsize.
 
Hi Eric A. Borisch,

thank you for your reply and mentioning your exparience; that was what I was looking for.

I think the recordsize and the ashift issue has been settled. I was just wondering about the need for compression.

Yes, I understand that the recordsize for the database must be set for the size in which the particular database stores data.

Kindest regards,

M
 
OP, be sure to cap your system consumption at 80% and no higher.
ZFS doesn't like having less than 20% free space for its work.

I have to build a 2nd NAS, as mine (same as yours) is already at the 80% mark.
Newegg was running a firesale on WD Red 18tb disks for $239.99 which is $13.33 per TB.
This is far lower cost even below the WD Blue 8tb at $14.37 per TB.
However, if you want bullet proof storage, meaning RAIDZ2, the total cost of that storage will still be high.
 
Hi bgavin,

thank you, my understanding is that the recommendation of 80% is due to increased fragmentation when the COW algorithm is trying to find a contiguous space for writing. Since the content will be more-or-less static, the problem should be lessened, no?

In any event, the amount of space of the movies currently stored at another server is we below that, and by the time I will need more space, I hope that the hard-drive's cost will be lower, or I will win a lottery. In any event I will use RAIDZ2 as I shudder thinking about re-ripping the entire collection.

I am also planning to set a reservation, so that I accidentally do not lock-up the system.

Kindest regards,

M
 
For mostly static (and therefore read) content, you are correct that you likely won’t notice an issue at 80%.

Also always look for CMR (not shingled/SMR) drives for ZFS. Although this is again less of an issue with read-heavy workloads.

And I’m guessing you mean a quota and not a reservation. (Both words have specific — and very different — meanings in ZFS.)
 
Hi Eric A. Borisch,

thank you for bringing to my attention the CMR vs. SMR drives, I did not even know the difference and had to look it up.

My understanding of the quota is to be a limit on the amount of space a data-set, e.g., movies can use, while reservation is a guarantee that some amount of space is reserved for a data-set. So, my thinking was to reserve some space, e.g., 2GB for a hic sunt leones data-set and set the property to non-writable.

Am I thinking about it incorrectly?

Kindest regards,

M
 
mefizto said:
So, my thinking was to reserve some space, e.g., 2GB for a hic sunt leones data-set and set the property to non-writable.
Click to expand...
You’re thinking about it correctly; sorry for the noise.

Somewhat related, note that snapshots, if you use them, can only be destroyed completely — you can’t go back and remove just a file or directory tree within the snapshot histories. As such, try to build your tree such that every file within a dataset experiences a similar lifecycle / has the same “retention policy”. (Don’t put a temporary / scratch directory in the same dataset as archival directories.)

And plan to use snapshots, even if you think you don’t need them. They are a great way to prevent foot-shooting.
 
Hi Eric A. Borisch,

absolutely no need to apologize, I'd rather that someone catches my stupidity on time in order to prevent possibly irreversible problems.

Yes, I do know about snapshots, I have been using ZFS on file servers since it appeared in Open Indiana. Saying that, I am by no means an advanced user, so any information is appreciated.

Kindest regards,

M
 
You must log in or register to reply here.
Back
Top