Initial setting of hw.intr_storm_threshold too low for new processors?

torqueturns

torqueturns

While upgrading FreeBSD and hardware, we've run into issues with 10Gig ethernet drivers and expected processing power until changing the value of hw.intr_storm_threshold from the default of 1000 to 40000.

Both "systat -v 1" and "vmstat -i" confirm that even the processor timer seems to be generating 2000 interrupts/second, so is the inital value hampering the processing power of the new systems?


This machine is pretty idle running FBSD 7.2-RELEASE-p3
Code: 
> vmstat -i
interrupt                          total       rate
irq1: atkbd0                           8          0
irq6: fdc0                            10          0
irq15: ata1                        13713          8
irq52: amr0                         3549          2
irq59: em1                           434          0
cpu0: timer                      3421359       1999
cpu3: timer                      3421368       1999
cpu1: timer                      3421368       1999
cpu2: timer                      3421325       1999
Total                           13703134       8008

This machine is a coreI7 940 machine with some ftp load.
7.2-RELEASE-p2 of the OS.

Code: 
> vmstat -i
interrupt                          total       rate
irq19: fwohci0+                  9923210         28
irq41: ixgb0                  1319628798       3805
cpu0: timer                    693204729       1999
irq257: re1                       291212          0
cpu6: timer                    693194426       1999
cpu1: timer                    693194695       1999
cpu2: timer                    693194565       1999
cpu5: timer                    693194407       1999
cpu4: timer                    693194409       1999
cpu7: timer                    693194425       1999
cpu3: timer                    693194567       1999
Total                         6875409443      19827

Same Corei7 940 only "systat -v 1" output

Code: 
   2 users    Load  3.17  2.82  2.66                  Aug 18 15:53

Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
        Tot   Share      Tot    Share    Free           in   out     in   out
Act   32260    5916   247400     8108 1380704  count
All  255640    9944  4705480    83488          pages
Proc:                                                            Interrupts
  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt  95346 cow   29328 total
  2         152      121k 116k 334k  13k  417 110k   5903 zfod        fwohci0+ 1
                                                          ozfod 13331 ixgb0 41
24.2%Sys   2.3%Intr  7.0%User  0.0%Nice 66.5%Idle        %ozfod  1999 cpu0: time
|    |    |    |    |    |    |    |    |    |    |       daefr     4 re1 irq257
============+>>>>                                   96292 prcfr  1999 cpu6: time
                                       302 dtbuf   125430 totfr  1999 cpu1: time
Namei     Name-cache   Dir-cache    100000 desvn          react  1999 cpu2: time
   Calls    hits   %    hits   %     78996 numvn          pdwak  1999 cpu5: time
   47607   41650  87                 24998 frevn          pdpgs  1999 cpu4: time
                                                       15 intrn  1999 cpu7: time
Disks   ad4   ad6   ad8                            277316 wire   2000 cpu3: time
KB/t   0.00  0.00  0.00                             26824 act
tps       0     0     0                           1320476 inact
MB/s   0.00  0.00  0.00                                   cache
%busy     0     0     0                           1379156 free
 
Remove recently added note about DEVICE_POLLING not working with SMP.
Remove warning from kern_poll.c to allow DEVICE_POLLING to be built with SMP.
Click to expand...
That was more than 4 years ago =)
I recommend to use it, and, if this servers is mainly router, you can set `sysctl kern.polling.idle_poll=1` to increase throughput. Just google some about tuning and you can really rock with this hardware :D

p.s. `hw.intr_storm_threshold` its not number of interrupts/sec, so values with your cpu clock is ok =)
 
Great!

I will try polling the interface..

The machine is being used to generate ftp traffic, which flows thru our RF router. We are trying to scale to 10,000 concurrent connections.
 
I have done that already using a variation of the information found at this url
Code: 
"http://rerepi.wordpress.com/2008/04/19/tuning-freebsd-sysoev-rit/"

I have even tuned the transmit buffer space in the ixgb driver. It seems that the last tweak was the sysctl variable hw.intr_storm_threshold, once I bumped it up, everything is running. No more issues with the 10Gig ethernet card.

I had just wondered about the relation of what that sysctl variable had on the amount of interrupts reported by vmstat -i or systat -v 1, for the newer processors.
 
You must log in or register to reply here.
Back
Top