Solved Poudriere Bulk Hangs sporadically - sleepq_wait_signal

Amzo

Active Member

Reaction score: 41
Messages: 124

When trying to build packages with poudriere, the builds hang every few minutes for a few minutes with the follow:

Code:
load: 1.43  cmd: sh 66112 [nanslp] 432.47r 0.13u 0.34s 0% 3728k
mi_switch+0xc1 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x199 kern_clock_nanosleep+0x1e1 sys_nano
sleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8

I currently have FreeBSD 13 with the latest patches installed and running poudriere with zfs.

I have tried running poudriere with:

Code:
PARALLEL_JOBS=1

As well as setting make_jobs to 1. The issue still persists.

I am using a Ryzen 2700x with 32Gb of memory and an SSD.
 

Alain De Vos

Daemon

Reaction score: 608
Messages: 2,056

Related to you CPU ?.
I've got,
Code:
CPUTYPE?=core-avx-i
Not one problem to note.
make.conf
Code:
MAKE_JOBS_NUMBER=20
Poudriere.conf
Code:
PARALLEL_JOBS=9
PREPARE_PARALLEL_JOBS=40
 
OP
Amzo

Amzo

Active Member

Reaction score: 41
Messages: 124

Yeah, I came across that post, they were causing IO issues with their custom logging. My issue is system wide, even using 12.2, 11.4, 13.0 and 14-current all had the same issue on my machine.
 
OP
Amzo

Amzo

Active Member

Reaction score: 41
Messages: 124

Related to you CPU ?.

I have a Ryzen 2700x and people have reported issues with Ryzen on FreeBSD. The system performs fine on Windows and Linux.

Only other thing I can think of is ZFS issue. Though I have an SSD and 32GB of DDR4 memory.
 

Alain De Vos

Daemon

Reaction score: 608
Messages: 2,056

I do it in 8GB memory,
Code:
zpool status -v
 
OP
Amzo

Amzo

Active Member

Reaction score: 41
Messages: 124

I'm running a fresh install on UFS, since the install was out of the box install. I don't mind trying UFS to see if the same performance issue persists, but it narrows it down from ZFS.
 
OP
Amzo

Amzo

Active Member

Reaction score: 41
Messages: 124

Well with UFS when starting the builtd poudriere it eventually returns

Code:
mkdir failed: device not configured
and running any command will return:

Code:
/bin/sh: vi: Input/Output error


S.M.A.R.T returns status passed, but will do a thorough hdd check, looks like it could be the culprit.
 
OP
Amzo

Amzo

Active Member

Reaction score: 41
Messages: 124

Seems that it was a hard drive issue. The problem got progressively worse, while S.M.A.R.T still reports everything is okay.

I switched to a new samsung SSD, and installed FreeBSD with UFS on that. Then used a secondary 1TB drive with ZFS for package building and storing packages with poudriere. This was to avoid heavy disk I/O on the drive with the OS while building and packaging. Everything seems to be working fine so far, but will monitor it to see if the problem creeps back in.
 
Top