About the poudriere crash issue: My educated guess is that this is not a kernel or software problem, but a hardware problem. For example, the RAM in the machine might be marginal, and it is not ECC. Under normal light usage, it works fine. Under extreme load, it overheats, or it uses too much power and voltages drop, or errors that are normally rare start piling up. Eventually, memory errors corrupt kernel data structures, and the kernel can't do anything other than crash. Why are kernel data structures such a good target? Because all free memory is usually used for file system buffer cache, and the data structures that describe that buffer cache are the biggest target for memory corruption.
In the mid 90s, lots of people built cheap i386 and i486 systems and ran Linux of them. There was a whole cottage industry of white-box computer assemblers. Many of these systems were cheaply built, with little quality control. Most ran just fine under Windows 3.1 or DOS, but had problems under the more intense workloads that Linux could put on them. Memory tests typically didn't find the problem (they were too simple minded, and didn't stress the rest of the system like disks, which use power too), so the best stress test for the system was doing Linux kernel compiles. I used to run them overnight on my system, and if it survived from midnight to 7am without a kernel crash, it was good enough to use in production.
This is one of the reasons I swear by ECC memory ... except this is a case of "do as I say, not as I do": My little server at home does not have ECC. Shame on me.
In the mid 90s, lots of people built cheap i386 and i486 systems and ran Linux of them. There was a whole cottage industry of white-box computer assemblers. Many of these systems were cheaply built, with little quality control. Most ran just fine under Windows 3.1 or DOS, but had problems under the more intense workloads that Linux could put on them. Memory tests typically didn't find the problem (they were too simple minded, and didn't stress the rest of the system like disks, which use power too), so the best stress test for the system was doing Linux kernel compiles. I used to run them overnight on my system, and if it survived from midnight to 7am without a kernel crash, it was good enough to use in production.
This is one of the reasons I swear by ECC memory ... except this is a case of "do as I say, not as I do": My little server at home does not have ECC. Shame on me.