FreeBSD SMP on ESXi triggering interrupt storm only when number of vcore > 1

B

boris_net

Hi all,

I would like some help understanding why I get an interrupt storm only when I set up more than one vcore on a FreeBSD VM.

Setup:
FreeBSD 9.1 on an ESXi 5.1 VM with 64GB + 4 PCIe adapters in Passthru (2 x LSI SAS2008 & 1 Dual core intel NIC which actually look like 1 PCIe adapter per port for ESXi - motherboard is a Supermicro X9SRL-F)

Here is the dmesg with two vcores configured on the VM:

Code: 
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4 09:23:10 UTC 2012
    root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz (2400.00-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206d7  Family = 6  Model = 2d  Stepping = 7
  Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0x9e982203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 68719476736 (65536 MB)
avail memory = 66241769472 (63173 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL 440BX> on motherboard
acpi0: Power Button (fixed)
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
pci0: <base peripheral> at device 7.7 (no driver attached)
vgapci0: <VGA-compatible display> port 0x10d0-0x10df mem 0xd8000000-0xdbffffff,0xd0800000-0xd0ffffff irq 16 at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem 0xd0040000-0xd005ffff,0xd0020000-0xd003ffff irq 17 at device 16.0 on pci0
mpt0: MPI Version=1.2.0.0
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci3: <ACPI PCI bus> on pcib3
mps0: <LSI SAS2008> port 0x4000-0x40ff mem 0xd2400000-0xd2403fff,0xd2440000-0xd247ffff irq 18 at device 0.0 on pci3
mps0: Firmware: 15.00.00.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci11: <ACPI PCI bus> on pcib11
mps1: <LSI SAS2008> port 0x5000-0x50ff mem 0xd2500000-0xd2503fff,0xd2540000-0xd257ffff irq 19 at device 0.0 on pci11
mps1: Firmware: 15.00.00.00, Driver: 14.00.00.01-fbsd
mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pci13: <ACPI PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pci14: <ACPI PCI bus> on pcib14
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pci15: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pci16: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pci17: <ACPI PCI bus> on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pci18: <ACPI PCI bus> on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci19: <ACPI PCI bus> on pcib19
em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0x6000-0x601f mem 0xd2620000-0xd263ffff,0xd2600000-0xd2603fff irq 16 at device 0.0 on pci19
em0: Memory Access and/or Bus Master bits were not set!
em0: MSIX: incorrect vectors, using MSI
em0: No MSI/MSIX using a Legacy IRQ
em0: Ethernet address: 00:25:90:a9:35:e4
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pci20: <ACPI PCI bus> on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pci21: <ACPI PCI bus> on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pci22: <ACPI PCI bus> on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pci23: <ACPI PCI bus> on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pci24: <ACPI PCI bus> on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pci25: <ACPI PCI bus> on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pci26: <ACPI PCI bus> on pcib26
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pci27: <ACPI PCI bus> on pcib27
em1: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0x7000-0x701f mem 0xd2720000-0xd273ffff,0xd2700000-0xd2703fff irq 17 at device 0.0 on pci27
em1: Memory Access and/or Bus Master bits were not set!
em1: MSIX: incorrect vectors, using MSI
em1: No MSI/MSIX using a Legacy IRQ
em1: Ethernet address: 00:25:90:a9:35:e5
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pci28: <ACPI PCI bus> on pcib28
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pci29: <ACPI PCI bus> on pcib29
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pci30: <ACPI PCI bus> on pcib30
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pci31: <ACPI PCI bus> on pcib31
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pci32: <ACPI PCI bus> on pcib32
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pci33: <ACPI PCI bus> on pcib33
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pci34: <ACPI PCI bus> on pcib34
acpi_acad0: <AC Adapter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse, device ID 3
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
ctl: CAM Target Layer loaded
acpi_throttle0: <ACPI CPU Throttling> on cpu0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
Timecounters tick every 10.000 msec
da0 at mpt0 bus 0 scbus2 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 102400MB (209715200 512 byte sectors: 255H 63S/T 13054C)
[remove HDD info to save on character count since I hit the 10k chars]
SMP: AP CPU #1 Launched!
[remove HDD info to save on character count since I hit the 10k chars]
Trying to mount root from ufs:/dev/da0p2 [rw]...
interrupt storm detected on "irq17:"; throttling interrupt source
interrupt storm detected on "irq17:"; throttling interrupt source

And vmstat output:

Code: 
vmstat -i
interrupt                          total       rate
irq1: atkbd0                           6          0
irq16: em0                          1304          1
irq17: em1 mpt0                  2131838       2722
irq18: mps0                          305          0
irq19: mps1                            7          0
cpu0:timer                         45345         57
cpu1:timer                         80975        103
Total                            2259780       2886


I did some research and tried the following with no success to stop the interrupt storm when I run this VM with two vcores:

  • Installed the VMware tools for FreeBSD - recognised by vsphere - no difference
  • Remove as many unused devices in the VM BIOS as possible - no difference
  • Set PassthruX.msiEnabled to false under VMware for the 4 adapters - no difference
  • Set kern.timecounter.hardware to ACPI-fast in /etc/sysctl.conf - no difference
  • Change the slot of the adapters on the motherboard, have not tried many combinations since it works with one vcore only - no difference

It works fine with one vcore only. Is there anything to do on FreeBSD to leverage more vcores while stopping the interrupt storm?

Thanks!
 
Thanks Jem, I think this is what I already have:

Code: 
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem 0xd0040000-0xd005ffff,0xd0020000-0xd003ffff irq 17 at device 16.0 on pci0
mpt0: MPI Version=1.2.0.0

From the man of mpt:
The mpt driver provides support for the LSI Logic Fusion-MPT family of SCSI, Fibre Channel and SAS controllers.
Click to expand...
 
Ok this is solved. I only kept the following in loader.conf:
Code: 
hw.pci.enable_msix="0" # Driver Interrupts problem SAS2008
and removed hw.pci.enable_msi set to the default of 1.

No more interrupt storm and I set up 4 vcores for this VM.
 
You must log in or register to reply here.
Back
Top