Upgrade from 8 to 10 has increased timer variability

[Jaunty]

We have many systems running on 8.x. We are starting to plan upgrading these systems to 10.1. These systems run in-house software applications that mostly spend their time waiting on a call to select(2) with a timeout configured. One of our QA tests checks how close the expected time the process resumes execution on a timeout condition.

These boxes really aren't doing much else and under 8.x we consistently measured variances of at most 1 ms with 99% of tests being less than 1 ms, with a 1000 Hz system tick rate configured. With 10.1 in an "out-of-the-box" configuration we routinely measure 20 to 30 ms of delay before the process resumes from select. We changed kern.eventtimer.periodic to 1 to undo the effect of the tickless event timer introduced in 9. This change helped substantially so that 98% of tests now are within 1 ms of the expected time, but we still get a very long tail in the distribution of delay measurements, with delays of many ms up to 50 ms on occasion. Even running a process at priority 0 of real-time does not change the results. Nor does disabling ACPI throttling.

We stripped down the test to a simple process that just does select(2) with no file descriptors and just a timeout configured. We still measure the long tail in the distribution of timing test results.

These are the sysctl kern settings for the system under 8.x:

kern.timecounter.tick: 1
kern.timecounter.choice: TSC(-100) ACPI-fast(1000) i8254(0) dummy(-1000000)
kern.timecounter.hardware: ACPI-fast
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.i8254.counter: 14990
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.ACPI-fast.counter: 1110085
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.quality: 1000
kern.timecounter.tc.TSC.mask: 4294967295
kern.timecounter.tc.TSC.counter: 2973346708
kern.timecounter.tc.TSC.frequency: 1668953390
kern.timecounter.tc.TSC.quality: -100
kern.timecounter.smp_tsc: 0
kern.timecounter.invariant_tsc: 1

And these are the sysctl kern settings for the system under 10.1:

kern.hz: 1000
kern.eventtimer.et.LAPIC.flags: 15
kern.eventtimer.et.LAPIC.frequency: 83116504
kern.eventtimer.et.LAPIC.quality: 400
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.choice: LAPIC(400) i8254(100) RTC(0)
kern.eventtimer.singlemul: 2
kern.eventtimer.idletick: 0
kern.eventtimer.timer: LAPIC
kern.eventtimer.periodic: 1
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.i8254.counter: 49194
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.ACPI-fast.counter: 2602969
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.TSC.mask: 4294967295
kern.timecounter.tc.TSC.counter: 2129367852
kern.timecounter.tc.TSC.frequency: 1662329270
kern.timecounter.tc.TSC.quality: 1000
kern.timecounter.stepwarnings: 0
kern.timecounter.alloweddeviation: 5
kern.timecounter.hardware: TSC
kern.timecounter.choice: TSC(1000) ACPI-fast(900) i8254(0) dummy(-1000000)
kern.timecounter.tick: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.smp_tsc: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.tsc_shift: 1

Any suggestions on configuring 10.1 to get performance closer to what we were getting with 8.x would be most appreciated.

 

[getopt]

These are the sysctl kern settings for the system under 8.x:

kern.timecounter.tick: 1
kern.timecounter.choice: TSC(-100) ACPI-fast(1000) i8254(0) dummy(-1000000)
kern.timecounter.hardware: ACPI-fast

And these are the sysctl kern settings for the system under 10.1:

kern.timecounter.choice: TSC(1000) ACPI-fast(900) i8254(0) dummy(-1000000)
kern.timecounter.hardware: TSC

Any suggestions on configuring 10.1 to get performance closer to what we were getting with 8.x would be most appreciated.

Try to set the timer to ACPI-fast what you had under 8.x

 

[Jaunty]

Try to set the timer to ACPI-fast what you had under 8.x

Thanks. I have tried this as well

sysctl kern.timecounter.hardware=ACPI-fast

and the results are the same. I can add that I have tried this test on three other types of hardware that have different timing capabilities (have HPET and have LAPIC) and all of them behave this way.

For example, if I use

struct timeval t = { 1, 0 };
struct timespec s, e;
[MAN]clock_gettime[/MAN](CLOCK_REALTIME_PRECISE, &s);
[MAN]select[/MAN](0, NULL, NULL, NULL, &t);
[MAN]clock_gettime[/MAN](CLOCK_REALTIME_PRECISE, &s);

to measure the elapsed time for a select() call with a 1 second timeout and do that thousands of times, I can compute a histogram of results which looks something like this:

For our 8.x systems:

Delay (ms)  Events
0.00      0.0%
0.25      0.9%
0.50      2.7%
0.75      4.1%
1.00      3.7%
1.25     36.7%
1.50     51.9%
1.75      0.1%
Max delay = 1.75 ms
Average delay = 1.3 ms

And for our 10.1 systems:

Delay (ms)  Events
0.00      0.0%
0.25      0.1%
0.50      6.1%
0.75      2.9%
1.00     89.5%
1.25      0.1%
1.50      0.0%
1.75      0.0%
2.0+      1.3%
Max delay = 46 ms
Average delay = 1.16 ms

So it seems like the average performance is measurably better, but the variability is greater and the worst case is much, much worse (and remember this is with kern.eventtimer.periodic=1, the measurements are drastically more variable if it has the default value of 0).