None of the sensors, (CPU or disk) indicate any thermal problems (system temp 43C). The drive temp SMART status is always 30C which seems sketch, but all other SMART variables are incrementing as one would expect. Still no reported errors. Load is quite light (2% right now, up to maybe 15%). My "hands on" support is technically limited, but a touch test should be within scope. I'll ask for that.
ralphbsz, you're making me sad with your insight. I'm moderately confident of the power situation not just because it is dual ABB (CP-E 12/10.0) power supply on an industrial grade (TDK-Lambda) redundancy module (DRM40), but also because the same output drives the AT&T fiber interface and CPE terminal and those have zero errors. Protectli seems to have a pretty solid user base and the forums are not filled with these sorts of errors so I doubt it is systematic on that end. I may just have a bad SATA controller part (I doubt these are going through extensive burn-in testing and my initial uptime was a week, I'd have passed it myself), which would be a bummer for sure.
Reading the history of the Apacer firmware updates, there are some pretty clear red flags and system halts for a problem with the "VT table" implementation and for TRIM implementation are explicitly mentioned. I wrote both Fortassa and Apacer to see if I could get the SFPS928A firmware and Apacer was nice but not helpful. I have despaired of mitigating any possible firmware faults with host configurations as I've run out of any plausible suggestions on that end, but still hold out hope the protectli box's SATA controller is OK.
My next step will be to try to get the box out of the critical path and revert to the tried and true old blue x336 box, then see if, thus derisked, my remote support will indulge me in experimenting to isolate the cause. The amended plan is (assuming no firmware update is available):
Verify temperature is indeed "reasonable" (not ouchy)
Wiggle/reseat the existing drive to ensure it isn't an oxide-layer problem (faulty wires, doesn't seem too likely but would be nice if that easy)
Swap the Fortassa drive (pic attached for posterity) with a modern high-volume branded one (Kingston SUV500MS/120G should rule out the drive as the problem)
Cure the on-board SATA controller with thermite