Hi thanks for the response. However, i am bit confused on usage of SLOG/ZIL device. Our use case is as follows:
Cache devices only improve read access (in certain cases).Have you thought about adding the NVMe SSD as a cache vdev for the pool? That would likely improve it.
Cache devices Devices can be added to a storage pool as "cache devices." These devices provide an additional layer of caching between main memory and disk. For read-heavy workloads, where the working set size is much larger than what can be cached in main memory, using cache devices allow much more of this working set to be served from low latency media. Using cache devices provides the greatest performance improvement for random read-workloads of mostly static content.
I like to think of the ZIL as a backup of the sync writes in RAM until their flushed to disk with the TXG. The SLOG is just a "Separate zfs intent LOG" stored on a faster (flash) device. It should never be faster than your pool under ideal conditions.
You could setup the NVMe as its own pool (pref. 2 in a mirror) and script the file move so that once its done it "flushes" to the disk. This seems a bit silly though as I would imagine your pool is faster than the cumulative upload of your users. If not, add more disks!
I think we're down to semantics. I think of it as a backup of the RAM for a crash or power loss as all the work is still done in RAM including when the TXG gets flushed to the pool. It's not "in-line" with the write path. As you pointed out it's only read after a crash because the writes not flushed must be completed. I do understand a sync write is not returned as completed until it's in the ZIL (pool or SLOG). Or perhaps I'm missing some nuance?This is just not correct. ZIL is the actual log on the physical medium and its purpose is to be the replay journal that gets played back in case of a crash. ZFS guarantees that the ZIL information hits the disc almost at the same time with the actual synchronous write, almost because it's not possible to guarantee full atomicity with storage mediums that do their own write buffering. The log can be stored on the main pool or on a separate SLOG device as you correctly noted.