interrupt storm detected

balanga

Son of Beastie

Reaction score: 109
Messages: 2,720

I have now built FreeBSD 12.0-RELEASE for ARM and have my GoFlex Home booting up without getting kernel panics, but instead I get a msg every second or so saying:-

interrupt storm detected on "intr13:" throttling interrupt source

Has anyone ever seen anything like this or knows what it relates to
 

hukadan

Well-Known Member

Reaction score: 247
Messages: 391

Have you asked Google/Startpage/YouNameIt before asking here ? 2 minutes of searching gave me this :
 

T-Daemon

Active Member

Reaction score: 84
Messages: 249

This is what I got:

Interrupt storm - From Wikipedia
In operating systems, an interrupt storm is an event during which a processor receives an inordinate number of interrupts that consume the majority of the processor's time. Interrupt storms are typically caused by hardware devices that do not support interrupt rate limiting.
...
Interrupt mitigating
There are hardware-based and software-based approaches to the problem. For example, FreeBSD detects interrupt storms and masks problematic interrupts for some time in response.
intr13 - 3 references:

atpic.c - PIC driver for the 8259A Master and Slave PICs in PC/AT machines.
atpic_vector.S , atpic_vector.s - Interrupt entry points for external interrupts triggered by the 8259A master and slave interrupt controllers.

What does the 8259 PIC do?
The 8259 PIC controls the CPU's interrupt mechanism, by accepting several interrupt requests and feeding them to the processor in order. For instance, when a keyboard registers a keyhit, it sends a pulse along its interrupt line (IRQ 1) to the PIC chip, which then translates the IRQ into a system interrupt, and sends a message to interrupt the CPU from whatever it is doing. Part of the kernel's job is to either handle these IRQs and perform the necessary procedures (poll the keyboard for the scancode) or alert a userspace program to the interrupt (send a message to the keyboard driver).
I’m not sure where to look further to narrow down the cause responsible. Maybe dmesg -a | grep -i pic gives something relevant.

Startpage
Ah, I see you’re a man of culture as well :)
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 109
Messages: 2,720

Have you asked Google/Startpage/YouNameIt before asking here ? 2 minutes of searching gave me this :
I've actually spent numerous hours finding lots of links which turned out not to be related to the problem I'm seeing, ie 'intr13'.
 

hukadan

Well-Known Member

Reaction score: 247
Messages: 391

I've actually spent numerous hours finding lots of links
So you know by now this is materiel specific and yet you do not provide any information regarding your hardware. The second post of one of the thread I linked to is :
Perhaps if you told us what version of FreeBSD, what architecture and on what hardware is it running.
The second post of one of the thread I linked to is:
First try to rise the treshold, sysctl hw.intr_storm_threshold=4000
Have you tried this ? Did it work ?

The second message of the mailing list I linked to is:
Output of "vmstat -i"?
What is the output in your case ?
 

tommiie

Active Member

Reaction score: 40
Messages: 154

I believe balanga to be in way over his head on this project. It doesn't help not being able to ask good questions.
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 109
Messages: 2,720

This is what I got:

Interrupt storm - From Wikipedia


intr13 - 3 references:

atpic.c - PIC driver for the 8259A Master and Slave PICs in PC/AT machines.
atpic_vector.S , atpic_vector.s - Interrupt entry points for external interrupts triggered by the 8259A master and slave interrupt controllers.

What does the 8259 PIC do?


I’m not sure where to look further to narrow down the cause responsible. Maybe dmesg -a | grep -i pic gives something relevant.
It's virtually impossible to run any commands because of all the 'interrupt' msgs, but what I have managed to glean is that I may be able to stop these msgs by editing /boot/device.hints except that there was no such file in the installation.
This is an ARM build so I doubt whether any references to 8259 PIC will be applicable.

I did read somewhere that I should run vmstat -i and here's what it showed:
Code:
interrupt                          total       rate
irq1: timer0                       27224        201
irq12: mge0                         6037         45
irq13: mge0                        11003         81
irq19: ehci0                        5665         42
irq33: uart0                         247          2
Total                              50176        371
mge0 is the NIC on this device.

/var/run/dmesg.boot:-
Code:
---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-RELEASE DOCKSTAR arm
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
CPU: Feroceon 88FR131 rev 1 (**unknown 4** core)
  Little-endian DC enabled IC enabled WA disabled DC streaming enabled
  BTB disabled L2 enabled L2 prefetch enabled
  WB enabled LABT branch prediction enabled
  16KB/32B 4-way instruction cache
  16KB/32B 4-way write-back-locking-C data cache
real memory  = 0 (0 MB)
avail memory = 120799232 (115 MB)
arc4random: no preloaded entropy cache
random: entropy device external interface
ofwbus0: <Open Firmware Device Tree>
simplebus0: <Flattened device tree simple bus> on ofwbus0
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
localbus0: <Marvell device bus> on ofwbus0
ic0: <Marvell Integrated Interrupt Controller> mem 0x20200-0x2023b on simplebus0
timer0: <Marvell CPU Timer> mem 0x20300-0x2032f irq 1 on simplebus0
Event timer "CPUTimer0" frequency 200000000 Hz quality 1000
Timecounter "CPUTimer1" frequency 200000000 Hz quality 1000
gpio0: <Marvell Integrated GPIO Controller> mem 0x10100-0x1011f irq 35,36,37,38,39,40,41 on simplebus0
gpio0: ERROR: no pin-count or ngpios entry found!
device_attach: gpio0 attach returned 6
rtc0: <Marvell Integrated RTC> mem 0x10300-0x10307 on simplebus0
rtc0: registered as a time-of-day clock, resolution 1.000000s
twsi0: <Marvell Integrated I2C Bus Controller> mem 0x11000-0x1101f irq 43 on simplebus0
iicbus0: <OFW I2C bus> on twsi0
iic0: <I2C generic I/O> on iicbus0
mge0: <Marvell Gigabit Ethernet controller> mem 0x72000-0x73fff irq 12,13,14,11,46 on simplebus0
mge0: PHY0 attached, phy_sc points to mge0
mge0: Ethernet address: 00:10:75:28:cc:00
miibus0: <MII bus> on mge0
e1000phy0: <Marvell 88E1116R Gigabit PHY> PHY 0 on miibus0
e1000phy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
uart0: <16550 or compatible> mem 0x12000-0x1201f irq 33 on simplebus0
uart0: console (1056,n,8,1)
uart1: <16550 or compatible> mem 0x12100-0x1211f irq 34 on simplebus0
cesa0: <Marvell Cryptographic Engine and Security Accelerator> mem 0x30000-0x30fff,0x3d000-0x3dfff irq 22 on simplebus0
ehci0: <Marvell Integrated USB 2.0 controller> mem 0x50000-0x50fff irq 48,19 on simplebus0
usbus0: EHCI version 1.0
usbus0 on ehci0
cryptosoft0: <software crypto>
Timecounters tick every 10.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
DUMMYNET 0 with IPv6 initialized (100409)
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded
usbus0: 480Mbps High Speed USB v2.0
Trying to mount root from ufs:/dev/da0s2 [rw]...
Root mount waiting for: usbus0
ugen0.1: <Marvell EHCI root HUB> at usbus0
uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
uhub0: 1 port with 1 removable, self powered
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.2: <Kingston DataTraveler 3.0> at usbus0
umass0 on uhub0
umass0: <Kingston DataTraveler 3.0, class 0/0, rev 2.10/1.10, addr 2> on usbus0
mountroot: waiting for device /dev/da0s2...
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <Kingston DataTraveler 3.0 PMAP> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 94DE80724795B2809941CD07
da0: 40.000MB/s transfers
da0: 14784MB (30277632 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
random: unblocking device.
lo0: link state changed to UP
interrupt storm detected on "intr12:"; throttling interrupt source
interrupt storm detected on "intr13:"; throttling interrupt source
mge0: link state changed to UP
interrupt storm detected on "intr12:"; throttling interrupt source
interrupt storm detected on "intr13:"; throttling interrupt source
interrupt storm detected on "intr12:"; throttling interrupt source
...
...
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 109
Messages: 2,720

So you know by now this is materiel specific and yet you do not provide any information regarding your hardware.
Ermmm.... the first sentence in my post says:
I have now built FreeBSD 12.0-RELEASE for ARM and have my GoFlex Home
FreeBSD 12.0-RELEASE, ARM and GoFlex Home show exactly what I am using and where. Also this is in the Embedded forum.
 

hukadan

Well-Known Member

Reaction score: 247
Messages: 391

The emphasize in my post was on hardware. My point was that not everyone knows what a GoFlex Home is and what hardware is in it. But never mind.
 

T-Daemon

Active Member

Reaction score: 84
Messages: 249

hukadan, tommiie, please guys, there is no need to be getting upset, no need to use harsh language or negative criticism . Such behavior doesn't suit you. If someone is a nuisance to you, ignore the post, act like a gentleman. Don't make me take back the man of culture :)
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,268
Messages: 29,700

I've had a few interrupt storms over the years. In my case it was pretty much always that horrid re(4) driver and certain -STABLE revisions. They typically disappeared a couple of revisions later. But this was on AMD64 hardware.

As you're working on ARM (Tier 2) you probably want to go with 12-STABLE to get the latest bug fixes and updates.
 

Phishfry

Son of Beastie

Reaction score: 1,240
Messages: 3,598

There was a major rework of FreeBSD 12 network drivers for the iflib changes.
What I would do is first, build my kernel without mge driver just to see it work.

My opinion is you need to be using either 11.x release or 11-STABLE.
Remember the mailing list post with a working GoFlex. That was using what version of FreeBSD.
That is the last know instances of it working.
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 109
Messages: 2,720

I've had a few interrupt storms over the years. In my case it was pretty much always that horrid re(4) driver and certain -STABLE revisions. They typically disappeared a couple of revisions later. But this was on AMD64 hardware.

As you're working on ARM (Tier 2) you probably want to go with 12-STABLE to get the latest bug fixes and updates.
In my case, I'm using old hardware and I'm told not to expect any bugfixes for Arm5 devices...

One thing I'm uncertain about is whether I should expect a /boot/device.hints file included in an ARM build. I don't have one and wondered if there might be some setting which would stop this interrupt storm. This file is something I never look at, but wondered if I should study it, or is it mainly used for bespoke configuration for PCs?

Altenatively maybe the is something I could add to /etc/sysctl.conf ... I added sysctl.hw.intr_storm_threshold=4000 but it made no difference.
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 109
Messages: 2,720

There was a major rework of FreeBSD 12 network drivers for the iflib changes.
What I would do is first, build my kernel without mge driver just to see it work.
Not sure how to do that. Do I just comment out device mge
from /usr/src/sys/arm/conf/DOCKSTAR ?
My opinion is you need to be using either 11.x release or 11-STABLE.
Remember the mailing list post with a working GoFlex. That was using what version of FreeBSD.
That is the last know instances of it working.
I've tried numerous versions of FreeBSD. Using 11.2-RELEASE ended up with kernel panics as soon as I logged on. This does not happen with 12.0-RELEASE which simply spews out 'interrupt storm' msgs every second.

Is it possible to include the old mge0 driver?
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 109
Messages: 2,720

Building a kernel without the mge driver stopped the interrupt storm, but without the NIC the unit is useless :) ...

One thing I did notice in comparing boot logs was this.

Here's what I got under 11.2:-
Code:
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
CPU: Feroceon 88FR131 rev 1 (**unknown 4** core)
  Little-endian DC enabled IC disabled WA disabled DC streaming enabled
  BTB disabled L2 enabled L2 prefetch enabled
  WB enabled LABT branch prediction disabled
  16KB/32B 4-way instruction cache
  16KB/32B 4-way write-back-locking-C data cache
real memory  = 134213632 (127 MB)
avail memory = 121294848 (115 MB)
and with 12.0:-
Code:
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
CPU: Feroceon 88FR131 rev 1 (**unknown 4** core)
  Little-endian DC enabled IC enabled WA disabled DC streaming enabled
  BTB disabled L2 enabled L2 prefetch enabled
  WB enabled LABT branch prediction enabled
  16KB/32B 4-way instruction cache
  16KB/32B 4-way write-back-locking-C data cache
real memory  = 0 (0 MB)
avail memory = 120799232 (115 MB)
Could "IC (en/dis)abled" or "LABT branch prediction (en/dis)abled" be responsible for the interrupt storm?

I've no idea what these terms refer to...
 

Vull

Well-Known Member

Reaction score: 119
Messages: 285

For STABLE look here, or for 11-STABLE look here. When I used STABLE I usually tried the one with the most recent date first.
 
Top