kern.sched.preemp_tresh

Hi!

I have long time in /etc/sysctl.conf kern.sched.preemp_thresh=224 but on 15-REELEASE-p1 doesn't works anymore.

Code:
 sysctl kern.sched.preemp_thresh
sysctl: unknown oid 'kern.sched.preemp_thresh'

Is it something with my system or it is removed.

Thank you.
 
Since 15.1 you can choose between the ULE and 4BSD scheduler, so that sysctl MIB was renamed accordingly; now it's kern.sched.ule.preempt_thresh or kern.sched.4bsd.preempt_thresh, depending on which scheduler you prefer.

EDIT. Corrected mispelling and wrong info.
 
I used this for years as well

Code:
kern.sched.preempt_thresh=224

And noticed it disappeared in Freebsd 15

fernandel and Manix missed the t from preemp_thresh

it should be

Code:
kern.sched.preempt_thresh=224

and not

Code:
kern.sched.preemp_thresh=224

also this is wrong

Code:
sysctl kern.sched.ule.preemp_thresh

it should be

Code:
sysctl kern.sched.ule.preempt_thresh

which on my system outputs

Code:
kern.sched.ule.preempt_thresh: 40

And this doesnt exist for me

Code:
kern.sched.4bsd.preempt_thresh

This is the full output kern.sched

Code:
[i] Yes Master ? sysctl kern.sched
kern.sched.ule.always_steal: 0
kern.sched.ule.trysteal_limit: 2
kern.sched.ule.steal_thresh: 2
kern.sched.ule.steal_idle: 1
kern.sched.ule.balance_interval: 127
kern.sched.ule.balance: 1
kern.sched.ule.affinity: 1
kern.sched.ule.idlespinthresh: 157
kern.sched.ule.idlespins: 10000
kern.sched.ule.static_boost: 115
kern.sched.ule.preempt_thresh: 40
kern.sched.ule.interact: 30
kern.sched.ule.slice: 12
kern.sched.ule.quantum: 94488
kern.sched.topology_spec: <groups>
 <group level="1" cache-level="3">
  <cpu count="12" mask="fff,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11</cpu>
  <children>
   <group level="2" cache-level="2">
    <cpu count="2" mask="3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">2, 3</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="30,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">4, 5</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">6, 7</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="300,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">8, 9</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c00,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">10, 11</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
  </children>
 </group>
</groups>

kern.sched.available: 4BSD,ULE
kern.sched.name: ULE
kern.sched.4bsd.runq_fuzz: 1
kern.sched.4bsd.ipiwakeup.useloop: 0
kern.sched.4bsd.ipiwakeup.usemask: 1
kern.sched.4bsd.ipiwakeup.delivered: 0
kern.sched.4bsd.ipiwakeup.requested: 0
kern.sched.4bsd.ipiwakeup.enabled: 1
kern.sched.4bsd.slice: 12
kern.sched.4bsd.quantum: 94488
kern.sched.preemption: 1
kern.sched.cpusetsizemin: 2
kern.sched.cpusetsize: 128

So after some digging around it looks

this is the new code you should use

Code:
kern.sched.ule.preempt_thresh=224

as a replacement for the old code

Code:
kern.sched.preempt_thresh=224

which should be added to

Code:
 /etc/sysctl.conf

some one correct me if im wrong
 
I used this for years as well

Code:
kern.sched.preempt_thresh=224

And noticed it disappeared in Freebsd 15

fernandel and Manix missed the t from preemp_thresh

it should be

Code:
kern.sched.preempt_thresh=224

and not

Code:
kern.sched.preemp_thresh=224

also this is wrong

Code:
sysctl kern.sched.ule.preemp_thresh

it should be

Code:
sysctl kern.sched.ule.preempt_thresh

which on my system outputs

Code:
kern.sched.ule.preempt_thresh: 40

And this doesnt exist for me

Code:
kern.sched.4bsd.preempt_thresh

This is the full output kern.sched

Code:
[i] Yes Master ? sysctl kern.sched
kern.sched.ule.always_steal: 0
kern.sched.ule.trysteal_limit: 2
kern.sched.ule.steal_thresh: 2
kern.sched.ule.steal_idle: 1
kern.sched.ule.balance_interval: 127
kern.sched.ule.balance: 1
kern.sched.ule.affinity: 1
kern.sched.ule.idlespinthresh: 157
kern.sched.ule.idlespins: 10000
kern.sched.ule.static_boost: 115
kern.sched.ule.preempt_thresh: 40
kern.sched.ule.interact: 30
kern.sched.ule.slice: 12
kern.sched.ule.quantum: 94488
kern.sched.topology_spec: <groups>
 <group level="1" cache-level="3">
  <cpu count="12" mask="fff,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11</cpu>
  <children>
   <group level="2" cache-level="2">
    <cpu count="2" mask="3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">2, 3</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="30,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">4, 5</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">6, 7</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="300,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">8, 9</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c00,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">10, 11</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
  </children>
 </group>
</groups>

kern.sched.available: 4BSD,ULE
kern.sched.name: ULE
kern.sched.4bsd.runq_fuzz: 1
kern.sched.4bsd.ipiwakeup.useloop: 0
kern.sched.4bsd.ipiwakeup.usemask: 1
kern.sched.4bsd.ipiwakeup.delivered: 0
kern.sched.4bsd.ipiwakeup.requested: 0
kern.sched.4bsd.ipiwakeup.enabled: 1
kern.sched.4bsd.slice: 12
kern.sched.4bsd.quantum: 94488
kern.sched.preemption: 1
kern.sched.cpusetsizemin: 2
kern.sched.cpusetsize: 128

So after some digging around it looks

this is the new code you should use

Code:
kern.sched.ule.preempt_thresh=224

as a replacement for the old code

Code:
kern.sched.preempt_thresh=224

which should be added to

Code:
 /etc/sysctl.conf

some one correct me if im wrong
D'oh! That's what happens when carelessly using copy/paste! 😅
Thank you for the correction. 🙂
 
I used this for years as well

Code:
kern.sched.preempt_thresh=224

And noticed it disappeared in Freebsd 15

fernandel and Manix missed the t from preemp_thresh

it should be

Code:
kern.sched.preempt_thresh=224

and not

Code:
kern.sched.preemp_thresh=224

also this is wrong

Code:
sysctl kern.sched.ule.preemp_thresh

it should be

Code:
sysctl kern.sched.ule.preempt_thresh

which on my system outputs

Code:
kern.sched.ule.preempt_thresh: 40

And this doesnt exist for me

Code:
kern.sched.4bsd.preempt_thresh

This is the full output kern.sched

Code:
[i] Yes Master ? sysctl kern.sched
kern.sched.ule.always_steal: 0
kern.sched.ule.trysteal_limit: 2
kern.sched.ule.steal_thresh: 2
kern.sched.ule.steal_idle: 1
kern.sched.ule.balance_interval: 127
kern.sched.ule.balance: 1
kern.sched.ule.affinity: 1
kern.sched.ule.idlespinthresh: 157
kern.sched.ule.idlespins: 10000
kern.sched.ule.static_boost: 115
kern.sched.ule.preempt_thresh: 40
kern.sched.ule.interact: 30
kern.sched.ule.slice: 12
kern.sched.ule.quantum: 94488
kern.sched.topology_spec: <groups>
 <group level="1" cache-level="3">
  <cpu count="12" mask="fff,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11</cpu>
  <children>
   <group level="2" cache-level="2">
    <cpu count="2" mask="3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">2, 3</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="30,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">4, 5</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">6, 7</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="300,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">8, 9</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="2">
    <cpu count="2" mask="c00,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">10, 11</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
  </children>
 </group>
</groups>

kern.sched.available: 4BSD,ULE
kern.sched.name: ULE
kern.sched.4bsd.runq_fuzz: 1
kern.sched.4bsd.ipiwakeup.useloop: 0
kern.sched.4bsd.ipiwakeup.usemask: 1
kern.sched.4bsd.ipiwakeup.delivered: 0
kern.sched.4bsd.ipiwakeup.requested: 0
kern.sched.4bsd.ipiwakeup.enabled: 1
kern.sched.4bsd.slice: 12
kern.sched.4bsd.quantum: 94488
kern.sched.preemption: 1
kern.sched.cpusetsizemin: 2
kern.sched.cpusetsize: 128

So after some digging around it looks

this is the new code you should use

Code:
kern.sched.ule.preempt_thresh=224

as a replacement for the old code

Code:
kern.sched.preempt_thresh=224

which should be added to

Code:
 /etc/sysctl.conf

some one correct me if im wrong
It was copy/paste...sorry
 
No problem mate happens to all of us

So to recap it looks like this is the new code

Code:
kern.sched.ule.preempt_thresh=224

for the old code

Code:
kern.sched.preempt_thresh=224

and this doesnt seem to exist

Code:
kern.sched.4bsd.preempt_thresh

I think we need SirDice to confirm that
or if he's not about Sir David Beckham instead
 
No problem mate happens to all of us

So to recap it looks like this is the new code

Code:
kern.sched.ule.preempt_thresh=224

for the old code

Code:
kern.sched.preempt_thresh=224

and this doesnt seem to exist

Code:
kern.sched.4bsd.preempt_thresh

I think we need SirDice to confirm that
or if he's not about Sir David Beckham instead
:beer:

Yes, it works.
 
In /boot/loader.conf, if you set:

Code:
kern.sched.name=4BSD

you can use the 4BSD scheduler instead of the ULE default. I do have a zfs BE for that. There is no corresponding preempt_thresh sysctl for that scheduler.

One of the 4BSD scheduler values you can change if you want is kern.sched.4bsd.slice.

For what it's worth with more modern cpus, I don't think there's any real need anyway to change the default preempt_thresh value for desktop use.

EDIT: This is a related forum discussion thread:

What is sysctl kern.sched.preempt_thresh
 
For what it's worth with more modern cpus, I don't think there's any real need anyway to change the default preempt_thresh value for desktop use.

I did wonder if anyone noticed the effect of it not being set before they noticed the error message. I've never really been sure about the benefit of this - it's easy to convince oneself that a tweak has produced a benefit.
 
So do you folks want more or less preemption?
That is the question, isn't it.
The bigger picture is "how does preemption affect responsivness on a desktop"?
What does responsiveness on a desktop actually look like?
Hit a key, character appears.
Move the mouse, cursor moves.
Click the mouse, menu pops up.

What's the path for that? Roughly (very roughly):
Device input, device driver, kernel event queue, Xorg processing event, Xorg handing event off to application, application rendering character/popping up menu.
Lots of things in the chain.
So what gets affected by the preemption threshhold?
 
That is the question, isn't it.
The bigger picture is "how does preemption affect responsivness on a desktop"?
What does responsiveness on a desktop actually look like?
Hit a key, character appears.
Move the mouse, cursor moves.
Click the mouse, menu pops up.

What's the path for that? Roughly (very roughly):
Device input, device driver, kernel event queue, Xorg processing event, Xorg handing event off to application, application rendering character/popping up menu.
Lots of things in the chain.
So what gets affected by the preemption threshhold?

Presumably this only matters when there are competing workloads, for example a `make world` in the background and a desktop in the foreground.

The code is not easy enough to read for me to instantly see how the "pri" value that this sysctl is compared against gets computed. Obviously for desktop responsiveness you would want better priority for the "foreground" tasks. Whether there is a mechanism in sched_ule to ensure that is a different question. The manpage says "Interactivity heuristics that detect interactive applications and schedules them preferentially under high load.". That probably means looking at how much wait and non-wait time a process has. Not some direct value for "foreground".
 
Yep, I think in one of the threads here there is a link to a presentation on ULE and how some of the calculations/comparisons are done (but I'm way too lazy to try and find it) and honestly the question of how to calculate has been confounding schedulers since day one.

"I'm editing photos while doing a build world in the background". I want the photo changes to show up immediately but I want the build world to finish ASAP. Scheduler is supposed to balance all that but how?
Desktop reactiveness: what is the threshold for a user to detect a mouse movement/key type. 1ms, 10ms, 100ms? Quicker is better but what is quick enough? I think most interactive things say roughly 10ms or less.
Conversely, build world taking 5min 30sec vs 5 min 45 sec?

Foreground/background: X application, say a word processor, with the cursor sitting it. User is thinking about how to word the email, so thinking but not typing. Is that foreground simply because the cursor is in the window, even though the input/output of the application is idle for 9 secs out of 10?

Heck even if you start simply with "switch every 1ms" (timeslicing), it's always the computation of "who's next" that is the magic.

I don't have answers, I just point out these things so I can say "like compiler writers, scheduler writers are a lot smarter than me".
 
Desktop reactiveness: what is the threshold for a user to detect a mouse movement/key type. 1ms, 10ms, 100ms? Quicker is better but what is quick enough? I think most interactive things say roughly 10ms or less.
I'd likely have issue above 2ms :p

Would prefer input highest-priority above anything on desktop: if that's slow, then every interaction while on a desktop is slow.
 
I'd likely have issue above 2ms :p

Would prefer input highest-priority above anything on desktop: if that's slow, then every interaction while on a desktop is slow.
Interesting. That implies you can detect the difference between a 2ms and a 10ms delay on say hitting a key and having it appear. Don't take that the wrong way; it's an analogy with speech and telephony. Look at everything related to voice on the POTS network and you wind up with 8KHz bandwidth with sampling at a rate lower than you would think.

Or audiophiles stating that vacuum tube amps and analog inputs are light years ahead of any digital system. I truly believe some people can tell the difference, most can't but the actual input makes a difference in what can be heard. I'm saying that as someone that can't carry a tune if you put it in a bucket.

To me the relationship to desktops is most desktop applications are sitting waiting for input. A key press, a mouse movement, reading from disk (video/audio playback) or network (streaming video/audio). Then as I posited, the stack from actual input to output display (key press to render in word processor); lots of layers. So some may be able to detect the difference but an average user maybe not.
 
Or audiophiles stating that vacuum tube amps and analog inputs are light years ahead of any digital system. I truly believe some people can tell the difference, most can't but the actual input makes a difference in what can be heard. I'm saying that as someone that can't carry a tune if you put it in a bucket.
Audiophiles shouldn't care about delay as long as it is constant.

Musicians on the other hand legitimately care about latency when they play and need to hear themselves. But there is a minimum delay you can't avoid due to sound being so slow between speakers and ear, so here again it is unreasonable to demand zero latency.
 
Audiophiles shouldn't care about delay as long as it is constant.

Musicians on the other hand legitimately care about latency when they play and need to hear themselves. But there is a minimum delay you can't avoid due to sound being so slow between speakers and ear, so here again it is unreasonable to demand zero latency.
Agree completely. I think we still come back to what is the acceptable latency above zero. And honestly, I think that starts to become personal.
If I type at 30 char/sec I latency is different than someone typing 75char/sec.

That's kind of what I've been trying to say (badly): for the average user, there is a latency that below they don't notice, above they do and how does the preemption threshhold affect that latency.
 
The paper on sched_ule: https://www.usenix.org/legacy/event/bsdcon03/tech/full_papers/roberson/roberson.pdf

Worth reading before turning knobs.

See also sysctl -d kern.sched.ule, and, of course, the source.

sched_ule(4) tries to dynamically detect "interactive" programs as those that spend more time voluntarily sleeping (in sleep() or by waiting on a conditional variable — assumed to be waiting for user interaction) than running (burning CPU on a intensive processing task). This is a balancing act, but programs marked interactive get put in the queue to be run before tasks marked non-interactive (a compiler is the canonical example; after some moderate I/O at launch, it typically burns CPU without ever calling sleep() or blocking/waiting for something else).

The main tunable to adjust how aggressively tasks are labeled interactive is kern.sched.ule.interact [0..100; default 30]. 100=everything is marked interactive; 0 = nothing is. (Note setting to 0 or 100 largely "turns off" any impact from interactivity-scoring.)

Using idprio(1) or cpuset(1) directly on high (CPU) intensity tasks might be a more direct approach to try.

Other runnable tasks are scheduled before ones in the idle-queue, and idle-queue tasks are always preempted (provided they aren't in a syscall) when a non-idle-queue task becomes runnable. This might be a better choice than trying to tune a global system scheduling parameter.

Note if the "interfering" task is an I/O or memory bandwidth hog, some effect is unavoidable. FreeBSD doesn't have an ionice command.

On modern multi-core systems, you hopefully don't need to adjust these at all, but if you do, you need to pin down what your goal is and under what workload.
 
The paper on sched_ule: https://www.usenix.org/legacy/event/bsdcon03/tech/full_papers/roberson/roberson.pdf

Worth reading before turning knobs.

See also sysctl -d kern.sched.ule, and, of course, the source.

sched_ule(4) tries to dynamically detect "interactive" programs as those that spend more time voluntarily sleeping (in sleep() or by waiting on a conditional variable — assumed to be waiting for user interaction) than running (burning CPU on a intensive processing task). This is a balancing act, but programs marked interactive get put in the queue to be run before tasks marked non-interactive (a compiler is the canonical example; after some moderate I/O at launch, it typically burns CPU without ever calling sleep() or blocking/waiting for something else).

The main tunable to adjust how aggressively tasks are labeled interactive is kern.sched.ule.interact [0..100; default 30]. 100=everything is marked interactive; 0 = nothing is. (Note setting to 0 or 100 largely "turns off" any impact from interactivity-scoring.)

Using idprio(1) or cpuset(1) directly on high (CPU) intensity tasks might be a more direct approach to try.

Other runnable tasks are scheduled before ones in the idle-queue, and idle-queue tasks are always preempted (provided they aren't in a syscall) when a non-idle-queue task becomes runnable. This might be a better choice than trying to tune a global system scheduling parameter.

Note if the "interfering" task is an I/O or memory bandwidth hog, some effect is unavoidable. FreeBSD doesn't have an ionice command.

On modern multi-core systems, you hopefully don't need to adjust these at all, but if you do, you need to pin down what your goal is and under what workload.

It would be great if we had a way to inspect this "interactivity score" on a per-thread or per-process basis.

I have concerns that CPU-heavy interactive applications such as web browsers might not be covered.
 
It would be great if we had a way to inspect this "interactivity score" on a per-thread or per-process basis.

I have concerns that CPU-heavy interactive applications such as web browsers might not be covered.

You should be able to observe the effect of it in top; low interactivity / cpu-heavy tasks should have their priority value go up.

The priority values are deli counter rules; lower values are served first.

It might be possible to use dtrace to monitor the values, but I haven’t tried.

Web browsers as a difficult case are specifically discussed in the paper. Part of sched_ule is that each process (thread, I think; would need to look at the code) has a history (of limited length) to allow the scheduling and interactivity score to fluctuate over the course of a process’s lifetime.
 
The paper on sched_ule: https://www.usenix.org/legacy/event/bsdcon03/tech/full_papers/roberson/roberson.pdf

Worth reading before turning knobs.

See also sysctl -d kern.sched.ule, and, of course, the source.

sched_ule(4) tries to dynamically detect "interactive" programs as those that spend more time voluntarily sleeping (in sleep() or by waiting on a conditional variable — assumed to be waiting for user interaction) than running (burning CPU on a intensive processing task). This is a balancing act, but programs marked interactive get put in the queue to be run before tasks marked non-interactive (a compiler is the canonical example; after some moderate I/O at launch, it typically burns CPU without ever calling sleep() or blocking/waiting for something else).

The main tunable to adjust how aggressively tasks are labeled interactive is kern.sched.ule.interact [0..100; default 30]. 100=everything is marked interactive; 0 = nothing is. (Note setting to 0 or 100 largely "turns off" any impact from interactivity-scoring.)
I don't follow this. The last time I looked, interactivity was a percentage which was added to the nice value to get the priority; the priorities were spit between interactive and batch. Interactive levels were assigned to dedicated queues (one per priority) with the batch levels handled by a ring of queues that guaranteed batch tasks all got some time. So being interactive wasn't a purely boolean distinction because there were many levels handled in priority order.

So what does kern.sched.ule.interact actually do? And why does setting it to 0 or 100 largely turn off any impact from interactivity-scoring ?
 
Back
Top