Since upgrade to 13.2-R poudriere bulk jobs lead to services being killed on host

sko · Jan 2, 2024

My buildhost at home (dual-socket Xeon E5-2660 (14c/28t); 128GB RAM) was upgraded from 12.4-RELEASE to 13.2-RELEASE a few weeks ago.
This host runs weekly poudriere bulk jobs to build packages for my desktops and servers on mondays and tuesdays via cronjobs.

Ever since the host was upgraded, I get *a lot* of log entries about conftest coredumping in some of the build jails (roughly one for every 20 packages built):

Code:

[...]
Dec 26 18:25:15 srv1 kernel: pid 24277 (conftest), jid 286, uid 0: exited on signal 11 (core dumped)
Dec 26 19:24:31 srv1 kernel: pid 94037 (conftest), jid 265, uid 0: exited on signal 11 (core dumped)
Dec 26 19:24:46 srv1 kernel: pid 13835 (conftest), jid 265, uid 0: exited on signal 11 (core dumped)
Dec 26 19:53:41 srv1 kernel: pid 16391 (conftest), jid 295, uid 0: exited on signal 11 (core dumped)
Dec 26 20:19:43 srv1 kernel: pid 94255 (conftest), jid 295, uid 0: exited on signal 11 (core dumped)
Dec 26 20:24:42 srv1 kernel: pid 76512 (conftest), jid 311, uid 0: exited on signal 11 (core dumped)
Dec 26 21:09:43 srv1 kernel: pid 92256 (conftest), jid 277, uid 0: exited on signal 11 (core dumped)
Dec 26 21:15:38 srv1 kernel: pid 81787 (conftest), jid 283, uid 0: exited on signal 11 (core dumped)
Dec 26 21:44:41 srv1 kernel: pid 9474 (conftest), jid 289, uid 0: exited on signal 11 (core dumped)
Dec 26 21:47:26 srv1 kernel: pid 58772 (conftest), jid 289, uid 0: exited on signal 11 (core dumped)
Dec 26 21:48:19 srv1 kernel: pid 16115 (conftest), jid 267, uid 0: exited on signal 11 (core dumped)
Dec 26 21:50:01 srv1 kernel: pid 84796 (conftest), jid 317, uid 0: exited on signal 11 (core dumped)
Dec 26 23:30:38 srv1 kernel: pid 86793 (conftest), jid 281, uid 0: exited on signal 11 (core dumped)
Dec 26 23:31:48 srv1 kernel: pid 23731 (conftest), jid 281, uid 0: exited on signal 11 (core dumped)
[...]

Additionally, and more importantly, services on the host system are silently being killed, e.g. cron, sshd, (new)syslog etc...
This host has 128GB of RAM and 64GB of swap space and never had issues with OOM events (I suppose this is what's happening?) when running 12.4-RELEASE, now every poudriere bulk run wreaks havoc... (and FreeBSD is missing some eqivalent to OpenBSDs "rcctl ls failed", making it even more tedious to find killed services)

rust is on TMPFS_BLACKLIST, but even if any other package would now would use ridiculous amounts of memory during build, this should only cause that single build to fail, not the host OOM-killer running around and shooting critical services in the head...

FTR:

Code:

# freebsd-version -urk
13.2-RELEASE-p8
13.2-RELEASE-p8
13.2-RELEASE-p8
# poudriere version
poudriere-git-3.3.99.20220831

And my poudriere.conf: https://termbin.com/n9vv

I already tried reducing PARALLEL_BUILDS to 15 to no avail, yet again: running too much builds in parallel should *never* cause services on the host being killed.

I run another build host @work running 13.2-RELEASE (originally set up with 13.1-RELEASE), which is not showing such behaviour - but this host has triple the amount of RAM (384GB) for 2x20C/40T and PARALLEL_BUILDS=30, so it might simply never triger OOM events during builds.

Any advice on what's going on here and how to prevent services on the host being killed instead of the build jobs?

SirDice · Jan 2, 2024

sko said:
Dec 26 18:25:15 srv1 kernel: pid 24277 (conftest), jid 286, uid 0: exited on signal 11 (core dumped)

These are configuration tests, they're supposed to crash.

Code:

Sep 23 09:13:50 molly kernel: pid 2787 (conftest), jid 18, uid 0: exited on signal 11 (core dumped)
Sep 23 10:07:03 molly kernel: pid 8421 (conftest), jid 18, uid 0: exited on signal 11 (core dumped)
Sep 23 11:08:38 molly kernel: pid 80659 (conftest), jid 18, uid 0: exited on signal 11 (core dumped)
Sep 23 11:16:54 molly kernel: pid 11026 (conftest), jid 21, uid 0: exited on signal 11 (core dumped)
Sep 23 11:28:26 molly kernel: pid 50845 (conftest), jid 18, uid 0: exited on signal 11 (core dumped)
Sep 23 18:10:00 molly kernel: pid 39469 (conftest), jid 48, uid 0: exited on signal 11 (core dumped)
Sep 23 19:08:12 molly kernel: pid 62162 (conftest), jid 48, uid 0: exited on signal 11 (core dumped)
Sep 23 20:09:34 molly kernel: pid 34965 (conftest), jid 44, uid 0: exited on signal 11 (core dumped)
Sep 23 20:18:58 molly kernel: pid 64947 (conftest), jid 48, uid 0: exited on signal 11 (core dumped)
Sep 23 20:28:47 molly kernel: pid 153 (conftest), jid 42, uid 0: exited on signal 11 (core dumped)
Sep 24 02:42:04 molly kernel: pid 82122 (conftest), jid 70, uid 0: exited on signal 11 (core dumped)
Sep 24 10:25:06 molly kernel: pid 29419 (conftest), jid 95, uid 0: exited on signal 11 (core dumped)
Sep 24 10:32:34 molly kernel: pid 24332 (conftest), jid 94, uid 0: exited on signal 11 (core dumped)
Sep 24 10:36:06 molly kernel: pid 36951 (conftest), jid 94, uid 0: exited on signal 11 (core dumped)
{snip}
Oct 14 12:02:33 molly kernel: pid 4807 (conftest), jid 486, uid 0: exited on signal 11 (core dumped)
Oct 14 12:05:35 molly kernel: pid 54110 (conftest), jid 490, uid 0: exited on signal 11 (core dumped)
Oct 14 17:22:45 molly kernel: pid 11166 (conftest), jid 486, uid 0: exited on signal 11 (core dumped)
Oct 14 18:12:29 molly kernel: pid 98904 (conftest), jid 486, uid 0: exited on signal 11 (core dumped)
Oct 15 08:29:44 molly kernel: pid 89028 (conftest), jid 508, uid 0: exited on signal 11 (core dumped)
Oct 15 08:43:23 molly kernel: pid 89965 (conftest), jid 508, uid 0: exited on signal 11 (core dumped)
Oct 15 09:41:00 molly kernel: pid 32777 (conftest), jid 506, uid 0: exited on signal 11 (core dumped)
Oct 17 22:05:32 molly kernel: pid 61339 (conftest), jid 520, uid 0: exited on signal 11 (core dumped)
Oct 18 06:25:11 molly kernel: pid 16644 (conftest), jid 538, uid 0: exited on signal 11 (core dumped)
Oct 18 07:13:34 molly kernel: pid 66065 (conftest), jid 542, uid 0: exited on signal 11 (core dumped)
Oct 18 07:35:48 molly kernel: pid 21363 (conftest), jid 536, uid 0: exited on signal 11 (core dumped)
Oct 18 08:22:43 molly kernel: pid 41652 (conftest), jid 536, uid 0: exited on signal 11 (core dumped)
Oct 18 09:32:57 molly kernel: pid 51502 (conftest), jid 548, uid 0: exited on signal 11 (core dumped)
Oct 18 17:34:58 molly kernel: pid 70912 (conftest), jid 564, uid 0: exited on signal 11 (core dumped)
Oct 18 18:19:36 molly kernel: pid 3309 (conftest), jid 569, uid 0: exited on signal 11 (core dumped)
Oct 18 18:45:19 molly kernel: pid 84874 (conftest), jid 568, uid 0: exited on signal 11 (core dumped)
Oct 18 19:27:31 molly kernel: pid 71385 (conftest), jid 568, uid 0: exited on signal 11 (core dumped)
Oct 18 20:30:56 molly kernel: pid 32925 (conftest), jid 576, uid 0: exited on signal 11 (core dumped)
Oct 19 05:10:37 molly kernel: pid 97230 (conftest), jid 593, uid 0: exited on signal 11 (core dumped)
Oct 19 06:38:36 molly kernel: pid 96332 (conftest), jid 606, uid 0: exited on signal 11 (core dumped)
Oct 19 14:57:57 molly kernel: pid 53443 (conftest), jid 621, uid 0: exited on signal 11 (core dumped)
Oct 19 17:48:21 molly kernel: pid 68576 (conftest), jid 622, uid 0: exited on signal 11 (core dumped)
Oct 19 18:33:40 molly kernel: pid 77174 (conftest), jid 621, uid 0: exited on signal 11 (core dumped)
Oct 19 18:40:38 molly kernel: pid 73751 (conftest), jid 621, uid 0: exited on signal 11 (core dumped)
Oct 19 23:49:12 molly kernel: pid 38407 (conftest), jid 622, uid 0: exited on signal 11 (core dumped)
Oct 20 00:38:36 molly kernel: pid 98426 (conftest), jid 622, uid 0: exited on signal 11 (core dumped)
Oct 20 00:40:49 molly kernel: pid 31309 (conftest), jid 622, uid 0: exited on signal 11 (core dumped)
Oct 20 02:20:41 molly kernel: pid 58457 (conftest), jid 621, uid 0: exited on signal 11 (core dumped)
Oct 20 02:58:40 molly kernel: pid 64231 (conftest), jid 621, uid 0: exited on signal 11 (core dumped)
Oct 20 03:18:53 molly kernel: pid 33701 (conftest), jid 626, uid 0: exited on signal 11 (core dumped)
Oct 20 11:17:21 molly kernel: pid 44609 (conftest), jid 632, uid 0: exited on signal 11 (core dumped)
Oct 20 19:18:59 molly kernel: pid 57348 (conftest), jid 648, uid 0: exited on signal 11 (core dumped)
Oct 20 19:22:50 molly kernel: pid 95388 (conftest), jid 651, uid 0: exited on signal 11 (core dumped)
Oct 20 21:18:36 molly kernel: pid 77692 (conftest), jid 652, uid 0: exited on signal 11 (core dumped)
Oct 20 22:30:24 molly kernel: pid 25773 (conftest), jid 652, uid 0: exited on signal 11 (core dumped)
Oct 20 23:27:18 molly kernel: pid 11389 (conftest), jid 650, uid 0: exited on signal 11 (core dumped)
Oct 21 00:47:57 molly kernel: pid 45943 (conftest), jid 651, uid 0: exited on signal 11 (core dumped)
Oct 21 00:52:57 molly kernel: pid 83490 (conftest), jid 651, uid 0: exited on signal 11 (core dumped)
Oct 21 01:03:50 molly kernel: pid 38756 (conftest), jid 650, uid 0: exited on signal 11 (core dumped)
Oct 21 05:58:21 molly kernel: pid 41703 (conftest), jid 650, uid 0: exited on signal 11 (core dumped)
Oct 21 05:58:47 molly kernel: pid 56451 (conftest), jid 651, uid 0: exited on signal 11 (core dumped)
Oct 21 20:34:26 molly kernel: pid 1969 (conftest), jid 5, uid 0: exited on signal 11 (core dumped)
Oct 22 13:41:12 molly kernel: pid 82360 (conftest), jid 23, uid 0: exited on signal 11 (core dumped)
Oct 22 16:03:38 molly kernel: pid 31029 (conftest), jid 22, uid 0: exited on signal 11 (core dumped)
{snip}
Nov 19 20:19:07 molly kernel: pid 90053 (conftest), jid 84, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 89221 (conftest), jid 107, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 81726 (conftest), jid 108, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 11603 (conftest), jid 108, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 30489 (conftest), jid 106, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 11474 (conftest), jid 106, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 69875 (conftest), jid 124, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 89810 (conftest), jid 134, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 43632 (conftest), jid 142, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 91411 (conftest), jid 142, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 38478 (conftest), jid 146, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 99114 (conftest), jid 140, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 26074 (conftest), jid 146, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 38581 (conftest), jid 141, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 23838 (conftest), jid 146, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 71166 (conftest), jid 141, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 38291 (conftest), jid 142, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 51000 (conftest), jid 142, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 32395 (conftest), jid 140, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 67834 (conftest), jid 146, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 95384 (conftest), jid 140, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 35152 (conftest), jid 146, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 66938 (conftest), jid 160, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 85818 (conftest), jid 160, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 34621 (conftest), jid 200, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 25832 (conftest), jid 195, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 17815 (conftest), jid 196, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 84156 (conftest), jid 214, uid 0: exited on signal 11 (core dumped)
Nov 19 20:19:07 molly kernel: pid 85515 (conftest), jid 212, uid 0: exited on signal 11 (core dumped)
Nov 20 04:36:03 molly kernel: pid 12308 (conftest), jid 12, uid 0: exited on signal 11 (core dumped)
Nov 20 05:52:18 molly kernel: pid 12645 (conftest), jid 16, uid 0: exited on signal 11 (core dumped)
Nov 20 07:21:03 molly kernel: pid 17311 (conftest), jid 12, uid 0: exited on signal 11 (core dumped)
Nov 20 12:01:30 molly kernel: pid 2847 (conftest), jid 32, uid 0: exited on signal 11 (core dumped)
Nov 20 17:00:45 molly kernel: pid 29581 (conftest), jid 42, uid 0: exited on signal 11 (core dumped)
Nov 20 20:43:17 molly kernel: pid 96307 (conftest), jid 48, uid 0: exited on signal 11 (core dumped)
Nov 20 20:50:38 molly kernel: pid 87773 (conftest), jid 54, uid 0: exited on signal 11 (core dumped)
Nov 20 22:07:19 molly kernel: pid 51142 (conftest), jid 50, uid 0: exited on signal 11 (core dumped)
Nov 20 23:21:20 molly kernel: pid 13300 (conftest), jid 49, uid 0: exited on signal 11 (core dumped)
Nov 21 00:27:55 molly kernel: pid 47738 (conftest), jid 48, uid 0: exited on signal 11 (core dumped)
Nov 21 02:22:12 molly kernel: pid 8059 (conftest), jid 48, uid 0: exited on signal 11 (core dumped)
Nov 21 02:54:11 molly kernel: pid 16625 (conftest), jid 48, uid 0: exited on signal 11 (core dumped)
Nov 21 02:57:34 molly kernel: pid 34944 (conftest), jid 54, uid 0: exited on signal 11 (core dumped)
Nov 21 03:12:48 molly kernel: pid 68267 (conftest), jid 54, uid 0: exited on signal 11 (core dumped)
Nov 21 03:12:54 molly kernel: pid 76323 (conftest), jid 54, uid 0: exited on signal 11 (core dumped)
Nov 21 04:01:19 molly kernel: pid 14888 (conftest), jid 54, uid 0: exited on signal 11 (core dumped)
Nov 21 04:07:17 molly kernel: pid 61405 (conftest), jid 54, uid 0: exited on signal 11 (core dumped)
Nov 21 04:11:37 molly kernel: pid 678 (conftest), jid 48, uid 0: exited on signal 11 (core dumped)
Nov 21 05:48:50 molly kernel: pid 3806 (conftest), jid 49, uid 0: exited on signal 11 (core dumped)
Nov 21 06:21:10 molly kernel: pid 3729 (conftest), jid 50, uid 0: exited on signal 11 (core dumped)
Nov 21 15:29:05 molly kernel: pid 86625 (conftest), jid 84, uid 0: exited on signal 11 (core dumped)
Nov 21 16:37:38 molly kernel: pid 52711 (conftest), jid 86, uid 0: exited on signal 11 (core dumped)
Nov 21 16:43:39 molly kernel: pid 4630 (conftest), jid 90, uid 0: exited on signal 11 (core dumped)
Nov 21 16:46:31 molly kernel: pid 16225 (conftest), jid 90, uid 0: exited on signal 11 (core dumped)
Nov 21 16:52:33 molly kernel: pid 99851 (conftest), jid 84, uid 0: exited on signal 11 (core dumped)
Nov 22 01:45:55 molly kernel: pid 32229 (conftest), jid 87, uid 0: exited on signal 11 (core dumped)
Nov 22 04:39:00 molly kernel: pid 76971 (conftest), jid 86, uid 0: exited on signal 11 (core dumped)
Nov 22 07:09:36 molly kernel: pid 14970 (conftest), jid 84, uid 0: exited on signal 11 (core dumped)
Nov 22 18:50:21 molly kernel: pid 73958 (conftest), jid 106, uid 0: exited on signal 11 (core dumped)
{snip}
Dec 10 14:08:50 molly kernel: pid 15530 (conftest), jid 50, uid 0: exited on signal 11 (core dumped)
Dec 10 15:55:15 molly kernel: pid 28094 (conftest), jid 52, uid 0: exited on signal 11 (core dumped)
Dec 10 16:14:10 molly kernel: pid 65310 (conftest), jid 52, uid 0: exited on signal 11 (core dumped)
Dec 10 16:31:11 molly kernel: pid 66569 (conftest), jid 50, uid 0: exited on signal 11 (core dumped)
Dec 10 16:53:33 molly kernel: pid 63844 (conftest), jid 50, uid 0: exited on signal 11 (core dumped)
Dec 10 17:13:52 molly kernel: pid 5548 (conftest), jid 50, uid 0: exited on signal 11 (core dumped)
Dec 10 17:36:36 molly kernel: pid 47503 (conftest), jid 56, uid 0: exited on signal 11 (core dumped)
Dec 10 17:42:30 molly kernel: pid 87694 (conftest), jid 54, uid 0: exited on signal 11 (core dumped)
Dec 10 17:43:37 molly kernel: pid 99741 (conftest), jid 54, uid 0: exited on signal 11 (core dumped)
Dec 10 19:28:01 molly kernel: pid 85518 (conftest), jid 56, uid 0: exited on signal 11 (core dumped)
Dec 10 19:48:44 molly kernel: pid 75863 (conftest), jid 56, uid 0: exited on signal 11 (core dumped)
Dec 10 20:13:59 molly kernel: pid 7482 (conftest), jid 54, uid 0: exited on signal 11 (core dumped)
Dec 10 20:23:52 molly kernel: pid 53030 (conftest), jid 56, uid 0: exited on signal 11 (core dumped)
Dec 11 00:44:31 molly kernel: pid 31799 (conftest), jid 72, uid 0: exited on signal 11 (core dumped)
Dec 11 05:45:50 molly kernel: pid 70363 (conftest), jid 78, uid 0: exited on signal 11 (core dumped)
Dec 11 09:28:10 molly kernel: pid 32630 (conftest), jid 90, uid 0: exited on signal 11 (core dumped)
Dec 11 09:34:01 molly kernel: pid 25584 (conftest), jid 86, uid 0: exited on signal 11 (core dumped)
Dec 11 10:54:59 molly kernel: pid 5085 (conftest), jid 88, uid 0: exited on signal 11 (core dumped)
Dec 11 12:30:38 molly kernel: pid 20191 (conftest), jid 89, uid 0: exited on signal 11 (core dumped)
Dec 11 13:23:36 molly kernel: pid 13906 (conftest), jid 89, uid 0: exited on signal 11 (core dumped)
Dec 11 14:59:34 molly kernel: pid 29459 (conftest), jid 89, uid 0: exited on signal 11 (core dumped)
Dec 11 15:29:19 molly kernel: pid 26579 (conftest), jid 88, uid 0: exited on signal 11 (core dumped)
Dec 11 15:55:27 molly kernel: pid 95767 (conftest), jid 88, uid 0: exited on signal 11 (core dumped)
Dec 11 15:59:17 molly kernel: pid 31517 (conftest), jid 90, uid 0: exited on signal 11 (core dumped)
Dec 11 15:59:23 molly kernel: pid 38688 (conftest), jid 90, uid 0: exited on signal 11 (core dumped)
Dec 11 16:47:05 molly kernel: pid 70533 (conftest), jid 90, uid 0: exited on signal 11 (core dumped)
Dec 11 16:55:16 molly kernel: pid 26631 (conftest), jid 89, uid 0: exited on signal 11 (core dumped)
Dec 11 17:00:31 molly kernel: pid 66234 (conftest), jid 89, uid 0: exited on signal 11 (core dumped)
Dec 11 18:30:34 molly kernel: pid 54489 (conftest), jid 88, uid 0: exited on signal 11 (core dumped)
Dec 11 18:43:25 molly kernel: pid 94716 (conftest), jid 89, uid 0: exited on signal 11 (core dumped)
Dec 12 00:38:06 molly kernel: pid 67637 (conftest), jid 108, uid 0: exited on signal 11 (core dumped)
Dec 12 05:39:02 molly kernel: pid 7305 (conftest), jid 118, uid 0: exited on signal 11 (core dumped)
Dec 12 09:49:49 molly kernel: pid 96329 (conftest), jid 122, uid 0: exited on signal 11 (core dumped)
Dec 12 10:27:57 molly kernel: pid 15277 (conftest), jid 122, uid 0: exited on signal 11 (core dumped)
Dec 12 10:59:45 molly kernel: pid 96395 (conftest), jid 128, uid 0: exited on signal 11 (core dumped)
Dec 12 11:52:31 molly kernel: pid 89422 (conftest), jid 124, uid 0: exited on signal 11 (core dumped)
Dec 12 12:00:43 molly kernel: pid 77952 (conftest), jid 128, uid 0: exited on signal 11 (core dumped)
Dec 12 23:05:20 molly kernel: pid 81971 (conftest), jid 142, uid 0: exited on signal 11 (core dumped)
Dec 12 23:57:30 molly kernel: pid 44007 (conftest), jid 146, uid 0: exited on signal 11 (core dumped)
Dec 13 00:03:37 molly kernel: pid 43596 (conftest), jid 146, uid 0: exited on signal 11 (core dumped)
Dec 13 00:27:28 molly kernel: pid 24587 (conftest), jid 142, uid 0: exited on signal 11 (core dumped)
Dec 13 02:05:41 molly kernel: pid 68185 (conftest), jid 147, uid 0: exited on signal 11 (core dumped)
Dec 13 05:11:39 molly kernel: pid 8074 (conftest), jid 146, uid 0: exited on signal 11 (core dumped)

sko said:
Additionally, and more importantly, services on the host system are silently being killed, e.g. cron, sshd, (new)syslog etc...

This has nothing to do with conftests, you're simply running out of resources; memory and swap.

sko said:
never had issues with OOM events (I suppose this is what's happening?)

Yep. That's exactly what's happening.

My builds sometimes end up building LLVM 15, GCC 12, Rust and perhaps another GCC or LLVM version concurently. That'll bring my server to its knees. I ended up building LLVM, GCC and Rust separately, before building the rest of the package list. That was the only sure way of not running into resource starvation.

Code:

# cat up_all.sh
#!/bin/sh

SERVER_REPOS="14-stable 140-release 132-release"
DESKTOP_REPOS="140-release 14-stable"

POUDRIERE=/usr/local/bin/poudriere

BASEDIR=/usr/local/etc/poudriere.d

${POUDRIERE} ports -u -p desktop
${POUDRIERE} ports -u -p server

for j in ${SERVER_REPOS}; do
  ${POUDRIERE} bulk -j ${j} -p server -f ${BASEDIR}/rust.lst
  ${POUDRIERE} bulk -j ${j} -p server -f ${BASEDIR}/gcc.lst
  ${POUDRIERE} bulk -j ${j} -p server -f ${BASEDIR}/llvm.lst
  ${POUDRIERE} bulk -j ${j} -p server -f ${BASEDIR}/${j}-server-package.lst
done

for j in ${DESKTOP_REPOS}; do
  ${POUDRIERE} bulk -j ${j} -p desktop -f ${BASEDIR}/rust.lst
  ${POUDRIERE} bulk -j ${j} -p desktop -f ${BASEDIR}/gcc.lst
  ${POUDRIERE} bulk -j ${j} -p desktop -f ${BASEDIR}/llvm.lst
  ${POUDRIERE} bulk -j ${j} -p desktop -f ${BASEDIR}/${j}-desktop-package.lst
done

#poudriere logclean -y 14

sko · Jan 2, 2024

SirDice said:
These are configuration tests, they're supposed to crash.

Thanks for that info; never recognized them while running that host with 12.4-R though...

I'm currently running another build job and monitor memory usage and some of the services on the host. Thanks to syslog being one of the services often being killed, I don't have meaningful logs from the last build (I suspect syslog was the first or a very early victim).

SirDice · Jan 2, 2024

sko said:
Thanks to syslog being one of the services often being killed,

I've had sshd(8) getting killed one to many times. Quite annoying. Luckily that server just sits behind behind me but getting the console is just a pain.

sko · Jan 2, 2024

SirDice said:
I've had sshd(8) getting killed one to many times. Quite annoying. Luckily that server just sits behind behind me but getting the console is just a pain.

The IPMI web interface has a HTML5 console I can use to access the host and re-start services, but still its quite annoying and I don't believe this is supposed to be normal - especially because this never happened in the ~1.5 years this host was running poudriere on 12.4-RELEASE.
It's even more annoying as one has to manually check all enabled services - something like service -l failed would be *extremely* helpful (haven't looked at how OpenBSD does rcctl ls failed but 'how hard can it be?'™)

PS: I'll be shamelessly stealing some parts of your build script ?

SirDice · Jan 2, 2024

Don't exactly know when this started happening but at some point I constantly had an LLVM, GCC and Rust trying to build concurrently with pretty much every bulk build. That was a little too much for my old 2 core i5 with only 16GB of memory. Nine times out of ten all three builds would fail but not before it killed sshd(8), Samba and a small MySQL database that was also running on the server.

sko said:
I'll be shamelessly stealing some parts of your build script

Take whatever you need. For what it's worth, I have these settings in poudriere.conf:

Code:

USE_TMPFS=yes

ALLOW_MAKE_JOBS_PACKAGES="pkg llvm* gcc* binutils rust firefox node* mame webkit* ghc mongodb*"

PRIORITY_BOOST="gcc*"

It all runs on an i5-3470, 16 GB memory and 16 GB swap. After splitting up LLVM, GCC and Rust I didn't have any memory issues any more. A typical build run (building 4-5 repositories) usually takes 1 or 2 days, not fast but it'll get there in the end.

sko · Jan 2, 2024

SirDice said:
Take whatever you need. For what it's worth, I have these settings in poudriere.conf:

Those look pretty much the same as in my poudriere.conf.

Given I have a slightly different logic behind my builds (using -z and distinct <portstree>-<set>-make.conf files and ...-options directories for different configurations between servers/desktops (and once even jails) I came up with this abomination:

Code:

#!/bin/sh

JAILS="FreeBSD:13:amd64"
PORTS="quarterly"
SETS="servers"

BUILDFIRST="lang/rust lang/gcc devel/llvm www/node"

POUDRIERE="/usr/local/bin/poudriere"
GIT="/usr/local/bin/git"
BASEDIR="/usr/local/etc/poudriere.d"
PORTSDIR="/usr/local/poudriere/ports"

# first update ports trees; if something fails here we can exit early
for p in ${PORTS}; do
    # poudriere runs 'git pull --rebase' and messes up our manual rebases, so we use git directly to update the ports tree
    ${GIT} -C ${PORTSDIR}/${p} pull || exit "error while updating ports tree ${p}"
done

# build loops - not my proudest work...
for j in ${JAILS}; do
    # update jail
    ${POUDRIERE} jail -u -j ${j} || break "error while updating jail ${j}"
    for p in ${PORTS}; do
        for s in ${SETS}; do
            # build ports with insanely high ressource requirements first
            for b in ${BUILDFIRST}; do
                ${POUDRIERE} bulk -j ${j} -p ${p} -z ${s} ${b}
            done
            ${POUDRIERE} bulk -j ${j} -p ${p} -z ${s} -f ${BASEDIR}/pkglist-${s}
        done
    done
done

I wish posix/bourne shell would have something similar to arrays e.g. in perl to make looping over multiple lists less ugly... It would also enable one to use tuples e.g. to only build 'servers quarterly' and 'desktop latest'.

I don't know if it there is any way to check dependencies of all ports in pkglist-* and filter for the ports in BUILDFIRST to see if there are specific version dependencies (e.g. gcc12 and gcc48). I couldn't find anything useful in poudriere manpages, so I'll just specify the versions in BUILDFIRST if something else than the current default version is needed...

SirDice · Jan 2, 2024

Yes, that would be nice. So far it hasn't been a problem, if LLVM, GCC or Rust doesn't need rebuilding the build starts, figures out it has nothing to do and just exits. This only adds a couple of build runs with 0 built packages in the logs, I'm ok with that.

sko · Jan 8, 2024

Another week and another round of builds and despite rust, llvm, gcc and node being build upfront, the host again has half of its services killed and none of the bulk jobs has finished...

So since upgrading that host to 13.2-R back in november, not a *single* weekly bulk job finished successfully and the host needs a lot of manual intervention afterwards.
syslogd is always one of the first jobs being killed, so I have no idea what was going on...

Is this really considered "normal behaviour" for poudriere on FreeBSD 13+??

Cath O'Deray · Jan 17, 2024

sko said:
Is this really considered "normal behaviour" for poudriere on FreeBSD 13+??

I would not have treated it as normal with poudriere-devel on CURRENT.

cracauer@ · Jan 18, 2024

Hmmm, I also had sshd exited on a machine that did single-step port builds in poudriere, but 4000 of them. Nothing parallel except when told to by the port itself. The machine has 64 GB and there is no hint in syslog of a OOM condition.

sko · Jan 22, 2024

Today the poudriere process itself was killed during builds, leaving behind a few orphaned jails...

Also no OOM events in /etc/messages - usually there were no logs at all because syslogd was also killed, but this time it is still running. I suspect this is because I have added "<service>_oomprotect=YES" to rc.conf for *all* services on that host, so at least this prevents random services being killed, but poudriere is still practically useless on that host if I have to invoke a bulk job manually several times until it finally succeeds...

sko · Jan 22, 2024

I did a bit more digging today and stumbled over something I maybe should have double-checked more thoroughly:
My host @home was running poudriere-devel version 3.4.99
the host and 14.0 VM @work were still on poudriere-devel version 3.3.99

I downgraded the host @home to 3.3.99 and upgraded the host @work to 3.4.99.
I'll run a bulk job on both tomorrow and see if @home will be fine and @work starts to fail...

sko · Jan 29, 2024

Today the host @home I downgraded to 3.3.99 performed scheduled bulk jobs and lo and behold:

The other build job only had 14 packages rebuild, so nothing serious that might have triggered all the process killing.
This build with 1045 ports definitely would have gone sideways before - so there *might* be something wrong with poudriere-devel 3.4.99 at least for my host @home.
The host @work I upgraded to 3.4.99 to counter-check only built a few dozen packages on all 3 bulk jobs (all on quarterly), so I'm not sure if this accounts as a valid data point; will see if there are enough packages in next weeks bulk job to get some more meaningful result.

@home I now went for the 3.4 non-devel verison - I only had devel installed for PACKAGE_FETCH (which never worked anyways...), so I'll simply stay on the stable version for now.

Since upgrade to 13.2-R poudriere bulk jobs lead to services being killed on host

sko

SirDice

Administrator

sko

SirDice

Administrator

sko

SirDice

Administrator

sko

SirDice

Administrator

sko

Cath O'Deray

cracauer@

sko

sko

sko