Reducing interrupts on a genet interface?

`Orum

`Orum

I've been playing around with FreeBSD 13.0 on my Pi 4B, and when benchmarking the genet(4) Ethernet interface I found I could only get to a peak of ~92MB/s (736 mb/s). Looking at top, it appears this is limited by interrupt handling rate, as performance was showing 25% interrupt usage (hw.ncpu=4 for those unaware). I thought about trying polling(4) to get higher speeds, but that doesn't appear to be suported in the genet driver. Is there any sysctl I should be looking at tuning to get higher network performance, or am I just stuck with this as the maximum speed?
 
There is a sysctl hw.<interface>.max_interrupt_rate that defaults to 8000. Look at the output of systat -vmstat for example and see if you're hitting that ceiling. You can increase that rate limit in /boot/loader.conf.
 
SirDice said:
There is a sysctl hw.<interface>.max_interrupt_rate that defaults to 8000. Look at the output of systat -vmstat for example and see if you're hitting that ceiling. You can increase that rate limit in /boot/loader.conf.
Click to expand...
I'm not seeing it, at least for the genet interface:
Code: 
sysctl -a | grep max_interrupt
hw.ix.max_interrupt_rate: 31250
hw.em.max_interrupt_rate: 8000

I'm not even sure if it's bumping into a artificial interrupt limit; it might be simply a hardware one, though I'd have to do my own testing under Linux to confirm. It doesn't appear to be hitting the 8000 limit, as total system interrupts are < 7000. I'm also not sure exactly how to interpret all of the output of systat -vmstat, so here it is while running the benchmark (I used a 10-sec interval for systat):
Code: 
    2 users    Load  1.83  1.18  0.62                  Apr  9 10:33
   Mem usage:   5%Phy  2%Kmem                           VN PAGER   SWAP PAGER
Mem:      REAL           VIRTUAL                        in   out     in   out
       Tot   Share     Tot    Share     Free   count
Act 30620K       0    328M        0    1810M   pages
All 33548K   2828K    370M   35428K                       ioflt  Interrupts
Proc:                                                     cow    6339 total
  r   p   d    s   w   Csw  Trp  Sys  Int  Sof  Flt       zfod   2182 gic0,p11:-
  1           24        8K       48K   6K   13            ozfod       gic0,s33:
                                                         %ozfod       gic0,s85:
14.3%Sys  25.0%Intr  0.4%User  0.0%Nice 60.3%Idle         daefr       gic0,s121:
|    |    |    |    |    |    |    |    |    |    |       prcfr       gic0,s126:
=======+++++++++++++                                      totfr     2 gic0,s148:
                                         1 dtbuf          react   282 gic0,s157:
Namei     Name-cache   Dir-cache    111501 maxvn          pdwak     2 pcib0,0: b
   Calls    hits   %    hits   %       562 numvn        5 pdpgs       cpu0:ast
       3       3 100                    25 frevn          intrn       cpu1:ast
                                                   69764K wire        cpu3:ast
Disks mmcsd                                        19508K act     891 cpu0:preem
KB/t   0.00                                         3452K inact  2220 cpu1:preem
tps       0                                             0 laund       cpu2:preem
MB/s   0.00                                         1810M free    757 cpu3:preem
%busy     0                                        17657K buf         cpu0:rende
                                                                      cpu1:rende
                                                                      cpu2:rende
                                                                      cpu3:rende
                                                                    3 cpu0:hardc

covacat said:
can it use jumbo frames ?
Click to expand...
Maybe, I'm not really sure. While I love the theory behind jumbo frames, in practice I've always found it to be a disaster that typically greatly reduces performance. I avoid it whenever possible.
 
The rest of your infrastructure also needs to support jumbo frames to make use of them. It does no good if only one side of the connection has jumbo frames enabled while your switches and/or the target doesn't. Everything in the path must be able to support it. If everything lines up correctly then jumbo frames can definitely improve performance with certain loads.

I think (though not sure) the Pi's hardware might just be a limiting factor here.
 
SirDice said:
The rest of your infrastructure also needs to support jumbo frames to make use of them. It does no good if only one side of the connection has jumbo frames enabled while your switches and/or the target doesn't. Everything in the path must be able to support it. If everything lines up correctly then jumbo frames can definitely improve performance with certain loads.
Click to expand...
Yeah, I'm aware. The issue invariably becomes that even if one single device doesn't support it, it causes problems everywhere else. It's probably fine in some SAN settings where you can tightly control everything on the network, but I'm rarely in that scenario.

SirDice said:
I think (though not sure) the Pi's hardware might just be a limiting factor here.
Click to expand...
That was my fear. I'm going to order another microSD card (so I don't have to keep overwriting the one I have for the Pi) to put Linux on it, and test the Pi with that. If the results are better, it should at least answer that question.
 
While the Pi 4's gigabit interface is much improved over the Pi3 (that one was running on USB 2.0 and thus limited to USB2 max throughput) it's not perfect, so it's going to be difficult to sustain the full gigabit capability. It might be fine in short bursts but getting it to continuously push out 900Mbps is going to be difficult.
 
You must log in or register to reply here.
Back
Top