High if_io_tqg_* load after few hours.

[tbyte]

Something very wrong is going on. I had a torrent client running which I stopped and I had a dummynet rule (one) which I removed After about a day I get this :

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 187 ki31 0B 128K CPU3 3 23.3H 99.22% idle{idle: cpu3}
11 root 187 ki31 0B 128K CPU1 1 23.4H 98.42% idle{idle: cpu1}
11 root 187 ki31 0B 128K CPU7 7 23.1H 93.29% idle{idle: cpu7}
11 root 187 ki31 0B 128K CPU5 5 23.2H 78.88% idle{idle: cpu5}
11 root 187 ki31 0B 128K CPU0 0 18.9H 70.56% idle{idle: cpu0}
11 root 187 ki31 0B 128K RUN 4 19.8H 59.63% idle{idle: cpu4}
11 root 187 ki31 0B 128K RUN 6 19.5H 55.49% idle{idle: cpu6}
0 root -60 - 0B 4192K CPU2 2 286:24 55.31% kernel{if_io_tqg_2}
2 root -60 - 0B 128K WAIT 1 135:59 43.55% clock{clock (0)}
11 root 187 ki31 0B 128K RUN 2 19.7H 42.47% idle{idle: cpu2}
0 root -60 - 0B 4192K CPU4 4 276:29 37.38% kernel{if_io_tqg_4}
0 root -60 - 0B 4192K - 6 291:41 32.63% kernel{if_io_tqg_6}
0 root -60 - 0B 4192K - 0 333:39 29.25% kernel{if_io_tqg_0}
5478 975 20 0 3205M 534M uwait 3 0:49 0.33% java{inform_stat-1}
58484 root 20 0 18M 5832K CPU6 6 0:00 0.24% top
5478 975 20 0 3205M 534M uwait 7 0:27 0.23% java{inform-4}
0 root -64 - 0B 4192K - 1 56:32 0.22% kernel{dummynet}

Those if_io_tqg_* threads are consuming insane amount of CPU but the traffic on the only card that is used is minuscule:

#:> netstat -w1 -I igb2
input igb2 output
packets errs idrops bytes packets errs bytes colls
157 0 0 26596 189 0 29204 0
172 0 0 20925 213 0 24107 0
172 0 0 21902 205 0 24046 0
151 0 0 26745 185 0 29283 0
152 0 0 36214 177 0 38217 0
192 0 0 47249 229 0 49835 0
^C

#:> vmstat -i
interrupt total rate
irq4: uart0 4 0
cpu0:timer 95672404 1037
cpu1:timer 13136101 142
cpu2:timer 28552867 309
cpu3:timer 14787428 160
cpu4:timer 28458940 308
cpu5:timer 15361293 166
cpu6:timer 29538539 320
cpu7:timer 14930294 162
irq32: arcmsr0 9262893 100
irq33: arcmsr0 4361 0
irq34: arcmsr0 5192 0
irq35: arcmsr0 18437 0
irq36: xhci0 116361 1
irq47: igb2:rxq0 81644748 885
irq48: igb2:rxq1 21104606 229
irq49: igb2:rxq2 20460502 222
irq50: igb2:rxq3 22357461 242
irq51: igb2:aq 5 0
Total 395412436 4285

It's FreeBSD 14.3-RELEASE-p3 GENERIC amd64

It feels like those threads are stuck and none of down/up on the interface, reloading the IPFW rules, stopping processes fixed it. Rebooting does fix it for few hours.
Any idea what is going on?

 

[tbyte]

I left the router running without the transmission running and it was all fine. But after a day of running the transmission demon I get to this again:

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 187 ki31 0B 128K RUN 1 150.1H 99.07% idle{idle: cpu1}
11 root 187 ki31 0B 128K RUN 3 151.5H 96.39% idle{idle: cpu3}
11 root 187 ki31 0B 128K CPU7 7 151.5H 95.56% idle{idle: cpu7}
11 root 187 ki31 0B 128K CPU5 5 151.8H 95.46% idle{idle: cpu5}
11 root 187 ki31 0B 128K RUN 2 125.3H 78.76% idle{idle: cpu2}
11 root 187 ki31 0B 128K RUN 0 126.8H 76.46% idle{idle: cpu0}
11 root 187 ki31 0B 128K RUN 4 125.1H 72.07% idle{idle: cpu4}
11 root 187 ki31 0B 128K RUN 6 124.6H 70.07% idle{idle: cpu6}
0 root -60 - 0B 4016K CPU6 6 29.8H 29.20% kernel{if_io_tqg_6}
0 root -60 - 0B 4016K CPU4 4 29.5H 27.20% kernel{if_io_tqg_4}
2 root -60 - 0B 128K CPU1 1 29.5H 25.59% clock{clock (0)}
0 root -60 - 0B 4016K CPU0 0 30.3H 24.56% kernel{if_io_tqg_0}
0 root -60 - 0B 4016K CPU2 2 29.2H 21.97% kernel{if_io_tqg_2}

Stopping the transmission demon did not fix it. For some reason those threads are stuck and it gets worse and worse with time. That is most definitely a kernel bug.