(Apologies for correcting your spelling below ... just makes it easier to communicate.)
ACB stays the same 5 times and then I receiving something like retries exhausted.
It happens for all sata drivers.
Usually such errors appears for some random one driver and then for all other.
(You mean SATA drives: A drive is a piece of hardware, while a driver is a piece of software)
What this means is that the problem is consistent enough that retrying the operation doesn't fix it, so it's not just rare random occurrence, but systematic. On the other hand, it clearly can not occur for every SATA IO operation, otherwise the system wouldn't get this far. Weird.
It happens only during boot and system isn't really responsible later. Or init will fail or just can't login or something like that, so can't say what happening.
(You mean the system isn't responsive, meaning it does not respond. Not responsible would mean that the system can't be blamed for what happened.)
That makes sense: If a disk IO completely fails (see above), then some important OS operation (like starting init, or starting getty, or ...) will not work, and the system will not work.
If I add something like "find / -name something" in zfs rc script after mounting, it boots fine but such "workaround" causes huge delays in boot and a little bit ugly)
Even weirder: If the disk drive has a heavy workload (caused by find), it has *fewer* errors, and therefore the problems don't occur? Usually intense workload makes for *more* errors. But assuming your trick actually works, you could try the following as an emergency measure to keep the system going: Instead of "find ..." (which causes a big delay), put the find command in the background, and let it run a little slower: "nice find ... &". Maybe that trick is a compromise that works a little better.
Totally agree that this issue is really weird. Maybe something wrong on HW layer and causing such issues. Will try to get suitable power module to check ...
Checking the hardware is a good idea (it is called a power supply, not power module). On the other hand, you said above that the problem started after the upgrade to FreeBSD 11.1, so perhaps the problem really is caused by the operating system. Perhaps your hardware is a little unhealthy, and in some fashion 11.1 is more unforgiving. Personally, I find a problem in the south bridge chip unlikely, while a problem with the power supply is possible (although still a weird coincidence that it would fail right when you do the OS upgrade).
As an alternative attempt to debug this: How difficult would it be to boot your system from an older FreeBSD install CD (some old version that worked correctly for you), and then see whether access to the disks works correctly.
Good luck!
