What I think I understand (please point out if any of this is wrong):
- FreeBSD has 5 priority groups (but only 3 for user processes): =https://papers.freebsd.org/2020/BSDcan/mckusick-Scheduling_in_the_FreeBSD_Kernel.files/mckusick-Scheduling_in_the_FreeBSD_Kernel.pdf
- "realtime" takes precedence over "normal" (default)
- "normal" (default) takes precedence over "idle"
- "Normal" process priority can be set programmatically from -20 to 20 (-20 highest) via the setpriority(2) call
- "Normal" process nice level can be adjusted at runtime from -20 to 20 (-20 highest) via the renice(8) command
- "Realtime" OR "idle" process OR thread priority can be set programmatically from 0 to 31 (0 highest) via the rtprio(2) call or adjusted at runtime via the rtprio(1) command
- Using "realtime" or "idle" requires root permissions.
- Thread priority can be set via the pthread_setschedparam(3).
- Highest value is actually the lowest priority (setpriority
- Default for "Normal" process is
SCHED_OTHER
withsched_priority
0. - Attempting to change
SCHED_RR
orSCHEDULE_FIFO
as non-root will get anEPERM
error.- Presumably these are actually set the thread to "realtime" or "idle" priority groups, per https://lists.freebsd.org/pipermail/freebsd-current/2014-May/050455.html
- Limits can be seen via sched_get_priority_min(2) and sched_get_priority_max(2)
- For
SCHED_RR
, these seem to be 0-31 - For
SCHEDULE_FIFO
, these seem to be 0-31 - For
SCHED_OTHER
, these seem to be 0-103
- For
- Is there any way / documentation to confirm definitively that values for this call also follow the pattern (highest priority) = (lowest numerical value), like most of the other FreeBSD priority features?
- When I set
sched_priority
to an out-of-range value (-1000000 or 1000000):- return value is 0 (expected -1)
- pthread_setschedparam(3) shows the
sched_priority
actually set to the out of range value (expected to either remain as last valid value, or snap to min/max)
- After starting 100 threads with different
sched_priority
values, attempting to view individual thread priority viaps -H -O lwp,cpu,pri,rtprio,upr
shows all threads with the same priority values: PRI=20, RTPRIO=normal:0,UPR=20 (expected to see something here that matched the thread priority set)
We are adapting most of a large existing code base to work on FreeBSD. It was formerly written to target a real-time operating system, with quite a bit of fine-tuning for thread priority, scheduling, and processor affinity. That operating system seems to have treated larger priority values as higher priorities, unlike what FreeBSD appears to do.
We are encountering an issue with one processes (having about 30 threads) slowing down over several days, also increasing in CPU usage (from <10% to 200% of 4 cores; <2.5% to 50% of each core). Initial gdb analysis found most threads were waiting on a pthread_mutex_lock(3). Enabling the
We noticed that over time when the issue occured,
We currently suspect some sort of priority inversion, but are having trouble interpreting the documentation and the results of the ps() command
We are encountering an issue with one processes (having about 30 threads) slowing down over several days, also increasing in CPU usage (from <10% to 200% of 4 cores; <2.5% to 50% of each core). Initial gdb analysis found most threads were waiting on a pthread_mutex_lock(3). Enabling the
PTHREAD_MUTEX_ADAPTIVE_NP
spin loop behavior for some mutexes per libthr(3) seemed handle the standalone pthread_mutex_lock(3) case, but gdb then showed threads waiting on another case where the mutex is used with pthread_cond_wait() and pthread_cond_signal()We noticed that over time when the issue occured,
ps -H -O lwp,cpu,pri,rtprio,upr
showed some threads were "niced" to a rather high value (RTPRIO=normal:69). As root, we tried setting the process priority to realtime via rtprio(1). This seemed to reset all all threads to the same priority (RTPRIO=real:31), but the slowdown and high CPU usage remained).We currently suspect some sort of priority inversion, but are having trouble interpreting the documentation and the results of the ps() command