C++ Correct way to set thread priority and scheduling (pthread_setschedparam)?

What I think I understand (please point out if any of this is wrong):
What I do not understand regarding pthread_setschedparam(3) (testing with FreeBSD 12.3):
  1. Is there any way / documentation to confirm definitively that values for this call also follow the pattern (highest priority) = (lowest numerical value), like most of the other FreeBSD priority features?
  2. When I set sched_priority to an out-of-range value (-1000000 or 1000000):
    1. return value is 0 (expected -1)
    2. pthread_setschedparam(3) shows the sched_priority actually set to the out of range value (expected to either remain as last valid value, or snap to min/max)
  3. After starting 100 threads with different sched_priority values, attempting to view individual thread priority via ps -H -O lwp,cpu,pri,rtprio,upr shows all threads with the same priority values: PRI=20, RTPRIO=normal:0,UPR=20 (expected to see something here that matched the thread priority set)
Perhaps I am not using these features correctly, or I am missing some prerequisite step not immediately apparent from the documentation.

We are adapting most of a large existing code base to work on FreeBSD. It was formerly written to target a real-time operating system, with quite a bit of fine-tuning for thread priority, scheduling, and processor affinity. That operating system seems to have treated larger priority values as higher priorities, unlike what FreeBSD appears to do.

We are encountering an issue with one processes (having about 30 threads) slowing down over several days, also increasing in CPU usage (from <10% to 200% of 4 cores; <2.5% to 50% of each core). Initial gdb analysis found most threads were waiting on a pthread_mutex_lock(3). Enabling the PTHREAD_MUTEX_ADAPTIVE_NP spin loop behavior for some mutexes per libthr(3) seemed handle the standalone pthread_mutex_lock(3) case, but gdb then showed threads waiting on another case where the mutex is used with pthread_cond_wait() and pthread_cond_signal()

We noticed that over time when the issue occured, ps -H -O lwp,cpu,pri,rtprio,upr showed some threads were "niced" to a rather high value (RTPRIO=normal:69). As root, we tried setting the process priority to realtime via rtprio(1). This seemed to reset all all threads to the same priority (RTPRIO=real:31), but the slowdown and high CPU usage remained).

We currently suspect some sort of priority inversion, but are having trouble interpreting the documentation and the results of the ps() command
  • Like
Reactions: jbo