Solved Interrupt storm when switching virtual terminals

I've got a Supermicro 1114S-WTRT and I've put in an AMD EPYC 7232P.

Everything seems fine, except when I switch virtual terminals using for example Alt-F1 and Alt-F2 - the system seems to think hard (with lots of interrupts) for a few seconds.

If I have a login screen on VT1 (or just a prompt) and then switch to VT2 (where I have top running, showing system processes) in top I see these two processes:

intr 199.93%
usb 199.77%

The CPU line on top shows something like:

CPU: 0.0% user, 0.0% nice, 12.9% system, 12.9% interrupt, 74.2% idle

If I wait a few seconds, the interrupts drop down to 0 and it's responsive.

There's nothing else running on the machine - it's a base install.

Typing away normally seems fine e.g. if I call up top and press ? repeatedly to toggle between the main display and the help page - fine.

As soon as I switch the VTs - it takes two or three seconds to switch between them as it processes a huge number of interrupts.

Both processes running on CPU 12 (if that makes a difference). The CPU is

CPU: AMD EPYC 7232P 8-Core Processor (3100.06-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 1 package(s) x 4 cache groups x 2 core(s) x 2 hardware threads

This isn't meant as a proper bug report or anything - just an initial "anybody else seen something like this?" and "got any ideas what I can try and tweak to fix?" I'll have a wander through the BIOS settings but might take a while to do that, so if anyone has seen this before and knows how to fix it, please let me know.
 
Run systat -vmstat and have a look what interrupt might be causing this.
 
Thanks - so if I run systat -vmstat and do nothing, the total interrupts are about 70 or so.

I have discovered toggling Caps Lock on/off/on/off/on/off triggers the same issue.

So with systat -vmstat running and I hammer Caps Lock the total interrupts go from ~70 to e.g. 6295

The bulk of this is spread over cpuX: timer interrupts e.g.

cpu0: timer 799
cpu1: timer 1021
...
cpu7: timer 1348
... cpu 8, 9, 10, 11 are 200-or-so ...
cpu12: timer 764
...
cpu15: timer 540
xhci1 156
bnxt0:rxq0 10

If I stop hammering Caps Lock then the total interrupts drops to ~70, each cpuX: timer line shows 10 or less (mostly 2 or 3).
 
I consistently get about 2K interrupts/sec on one of the xhci ports (it has a USB GbE dongle on it). Any hints on how to debug this?
 
Well that was easy - not exactly sure WHICH of these knobs made the difference but with all three changed to Disabled my issue goes away.

BIOS->Advanced->USB Configuration

Legacy USB Support: Disabled
XHCI Hand-off: Disabled
Port 60/64 Emulation: Disabled

All three were Enabled before I made my changes.

Now if I hammer Caps Lock under systat I can get to just over 100 total interrupts, most in xhci1.
 
But it's often the combination of Legacy USB support and that Port 60/64 emulation that causes issues.
Thanks - that's the ticket! I re-enabled Legacy USB Support and XHCI Hand-off and left Port 60/64 Emulation set to Disabled and the problem is still absent.

So for clarity and the archives: try disabling Port 60/64 Emulation.
 
Back
Top