ZFS 48TB ZFS, 64GB RAM, Ryzen 3900X for Plex 4k: What can be tuned/optimized?

jakobbg

jakobbg

Hi all!

I've got a new server which is going to work as a file server, web server and transcoding 4k content-oriented Plex server, all running on 4x 16TB Seagate X16 drives in RAID-Z.

I'm just wondering about tuning. Swap, ZFS, stuff like that. What can be wise to change from defaults? Running a locally compiled 12.1-p1 kernel as of now (just removed a lot of unused stuff), but what should/could I tweak?

All input highly appreciated! :)?

Code: 
~ uname -a
FreeBSD core24.grimstveit.no 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 8d3f226f4b8(releng/12.1) CORE24  amd64

~ sysctl hw.physmem
hw.physmem: 68609564672

~ freecolor -m
Physical  : [#..................................] 4%    (3091/63787)
Swap      : [###################################] 100%  (3851/3851)

~ cat /var/run/dmesg.boot | grep Ryzen | head -1
CPU: AMD Ryzen 9 3900X 12-Core Processor             (3792.95-MHz K8-class CPU)
 
Last edited by a moderator:
Beware swapping on a zvol, as it can lead to deadlock. Dedicated partitions for swap are probably the safest.

Be very sure you want RAIDZ, as it can't be upgraded in-situ to RAIDZ2 at a later date.

As far as tuning goes, I would look at operational behaviour before changing the defaults You have some fairly capable hardware. No point sweating until it does.

A write-intensive spinning disk ZFS server is likely to benefit from an SSD ZIL (I'd go NVMe), but do the homework (it must have a big latency advantage over the tank, and data loss protection on power failure is essential).
 
You can likely (on your bulk storage file systems) increase the recordsize since you will mainly have large sequential reads and writes. This can decrease the processing overhead as well as increase the on-disk efficiency. A SLOG will almost certainly be useless, and you likely want to set logbias=throughput as again these will be handling large transaction streaming rather than a maximum small IOPS (database) situation. (Assuming that my I/O expectations for your transcoding tool are valid.)

Additionally, if your are unlikely to re-read data (in a reasonable period of time) from the bulk storage filesystems, you might consider setting primarycache=metadata on them to keep your large reads and writes from wasting space in
the ARC.
 
gpw928 said:
Be very sure you want RAIDZ, as it can't be upgraded in-situ to RAIDZ2 at a later date.
Click to expand...

Thanks, will try to read and find out more about this.

gpw928 said:
As far as tuning goes, I would look at operational behaviour before changing the defaults You have some fairly capable hardware. No point sweating until it does.
Click to expand...

Probably a lot of truth in this :-). I hoped that to be the case, but also decided to ask you guys in case there was something I should do that "everyone" did :-).

gpw928 said:
A write-intensive spinning disk ZFS server is likely to benefit from an SSD ZIL (I'd go NVMe), but do the homework (it must have a big latency advantage over the tank, and data loss protection on power failure is essential).
Click to expand...

The root and OS is running on an Samsung 970 EVO Plus 500GB NVM SSD, so that might be a good choice. The server will probably be mostly read, with the occasional massive disk writes when downloading large chunks of media from various p2p sources at the same time as indexing second media and transcoding third media.
 
jakobbg said:
I'm just wondering about tuning. Swap, ZFS, stuff like that. What can be wise to change from defaults? Running a locally compiled 12.1-p1 kernel as of now (just removed a lot of unused stuff), but what should/could I tweak?
:)?
Click to expand...

Limit ZFS ARC to 50% of your memory (or some other number you feel comfortable with) via a setting in /etc/sysctl.conf:

vfs.zfs.arc_max = 34359738368

(That is 32GB). Otherwise it will try to use up most (and in theory it should give back memory if the system needs it but my experience on FreeBSD 11.3 so far has been that it doesn't always work out that way) of the available RAM.


Make sure the system uses 4K sectors when talking to the disks (you don't want it to think the disks are using 512 byte sectors and) via another setting i sysctl.conf (needs to be there before you create the ZFS filesystem - I'd consider mirroring instead of RAID-Z though):

vfs.zfs.min_auto_ashift=12
 
I realise, you've been at this since Jan, but I've got 4x same disks in a Raidz1 and wanted to share what I'd do.

I set up an NVME as a read cache. (zpool add poolname cache diskdevice)
I'd store primary metadata only in RAM for the large media files to prevent caching of large media that will only be read once i.e. set primarycache=metadata mediapoolname
Also atime=off (prevents updating read times of files - helps with reading speed of lots of smaller files)
Compression=on
Then I'd change record size =1M for the large media
Then see if you can match a record size to the Plex database which uses sqlite if I recall. Matching record sizes to databases can give a great performance enhancement.
The problem is that Plex isn't a folder structure entirely made up of databases, so you probably need to make a second dataset for just that folder with the recordsize
Also, point your transcode cache somewhere else if you have only consumer SSD's, get a second hand cheap Enterprise SSD as transcode cache will reduce the life of a consumer ssd very quickly.
Generally formatting all devices including SSD's with ashift=12 is a good idea. A lot of people say it's ashift=13, I'd strongly suggest reading up for your specific SSD to understand which is the correct one (be aware that some provide emulated settings and you want the native value) - Samsung is generally notorious for emulating.

Hope that helps someone.
 
Some of the tricks I have done include .. (I found one of the biggest pita's was plex and its media scans)

#1
run plex in a jail with vnets .. I did up a kb here with some example settings .. adjust to taste jailed_plex

in my case I have multiple zpools.. my main storage pool is a pile of rust (zraid3) and one of the other pools is a raid 1+0 striped mirror of 4 SSD's..

although my KB was simple, you could get more creative and create a zvol on the mirror+striped pool .. then mount that as your database in the plex jail.

I believe the default was /usr/local/plexdata/Plex Media Server/

you could also change the block size and or do further performance tuning on that zvol specifically for db .. I would try the defaults first tho..

#2
if you have multiple nics do up your vnet on its own nic.. (assuming your storage server does other things, ie in my case one nic is attached to my internal network and the other goes to a separate media network)

#3
disable hyperthreading in /boot/loader.conf
machdep.hyperthreading_allowed="0"

this may or may not help, set it and reboot .. or go into your bios and disable it ..

#4
there are also transcode settings in plex .. I believe one of them is "make my cpu bleed". sounds like something I would use.

#5
plex has some settings along the lines of "automatically scan new content". .. I personally disable that as everytime a file is touched, removed or added it will trigger a rescan process .. imo i found it better to just crontab a scan before and after my viewing hours


lastly..

it sounds like as long as your hardware is detected properly (ie network and nvidia drivers are good) ..your system itself is probably doing fine .. my guess is its plex's behaviour and scanning policy that's causing issues..

another possibility to if your having issues like choppy playback over wifi ..
check out and make sure your wifi is not fighting for channel bandwidth, and that your on say 5g at minimum .. streaming 4k usually requires around 500-900mbs .. I believe the recommended is more like 2.4+mbs ..

im sure your getting enough performance your pool itself but you could check you iops and read/write etc using the standard zfs monitoring tools .
 
You must log in or register to reply here.
Back
Top