C++ Correct way to set thread priority and scheduling (pthread_setschedparam)?

[Tim Rau]

What I think I understand (please point out if any of this is wrong):

• FreeBSD has 5 priority groups (but only 3 for user processes): =https://papers.freebsd.org/2020/BSDcan/mckusick-Scheduling_in_the_FreeBSD_Kernel.files/mckusick-Scheduling_in_the_FreeBSD_Kernel.pdf
  • "realtime" takes precedence over "normal" (default)
  • "normal" (default) takes precedence over "idle"
• "Normal" process priority can be set programmatically from -20 to 20 (-20 highest) via the setpriority(2) call
  • "Normal" process nice level can be adjusted at runtime from -20 to 20 (-20 highest) via the renice(8) command
• "Realtime" OR "idle" process OR thread priority can be set programmatically from 0 to 31 (0 highest) via the rtprio(2) call or adjusted at runtime via the rtprio(1) command
  • Using "realtime" or "idle" requires root permissions.
• Thread priority can be set via the pthread_setschedparam(3).
  • Highest value is actually the lowest priority (setpriority
  • Default for "Normal" process is SCHED_OTHER with sched_priority 0.
  • Attempting to change SCHED_RRor SCHEDULE_FIFO as non-root will get an EPERM error.
    • Presumably these are actually set the thread to "realtime" or "idle" priority groups, per https://lists.freebsd.org/pipermail/freebsd-current/2014-May/050455.html
  • Limits can be seen via sched_get_priority_min(2) and sched_get_priority_max(2)
    • For SCHED_RR, these seem to be 0-31
    • For SCHEDULE_FIFO, these seem to be 0-31
    • For SCHED_OTHER, these seem to be 0-103

What I do not understand regarding pthread_setschedparam(3) (testing with FreeBSD 12.3):

1. Is there any way / documentation to confirm definitively that values for this call also follow the pattern (highest priority) = (lowest numerical value), like most of the other FreeBSD priority features?
2. When I set sched_priority to an out-of-range value (-1000000 or 1000000):
   1. return value is 0 (expected -1)
   2. pthread_setschedparam(3) shows the sched_priority actually set to the out of range value (expected to either remain as last valid value, or snap to min/max)
3. After starting 100 threads with different sched_priority values, attempting to view individual thread priority via ps -H -O lwp,cpu,pri,rtprio,upr shows all threads with the same priority values: PRI=20, RTPRIO=normal:0,UPR=20 (expected to see something here that matched the thread priority set)

Perhaps I am not using these features correctly, or I am missing some prerequisite step not immediately apparent from the documentation.

Spoiler: Background / impetus for question

We are adapting most of a large existing code base to work on FreeBSD. It was formerly written to target a real-time operating system, with quite a bit of fine-tuning for thread priority, scheduling, and processor affinity. That operating system seems to have treated larger priority values as higher priorities, unlike what FreeBSD appears to do.

We are encountering an issue with one processes (having about 30 threads) slowing down over several days, also increasing in CPU usage (from <10% to 200% of 4 cores; <2.5% to 50% of each core). Initial gdb analysis found most threads were waiting on a pthread_mutex_lock(3). Enabling the PTHREAD_MUTEX_ADAPTIVE_NP spin loop behavior for some mutexes per libthr(3) seemed handle the standalone pthread_mutex_lock(3) case, but gdb then showed threads waiting on another case where the mutex is used with pthread_cond_wait() and pthread_cond_signal()

We noticed that over time when the issue occured, ps -H -O lwp,cpu,pri,rtprio,upr showed some threads were "niced" to a rather high value (RTPRIO=normal:69). As root, we tried setting the process priority to realtime via rtprio(1). This seemed to reset all all threads to the same priority (RTPRIO=real:31), but the slowdown and high CPU usage remained).

We currently suspect some sort of priority inversion, but are having trouble interpreting the documentation and the results of the ps() command