need help choosing a configuration border router

allan_sundry

allan_sundry

Hi all,

Help me choose a border router configuration with Intel dual head 10G NIC with SFP+ (for ISP). Need a configuration tested to work on high-load (PF Shaping and NAT, VLAN, BGP, OSPF).
 
SirDice said:
What does this have to do with FreeBSD?
Click to expand...

FreeBSD as a software platform to act as a border router. Supported devices to obtain maximum performance from a server running FreeBSD.
 
Define 'configuration'. Do you want some hardware advice, or advice on system/software installation/configuration, or something like that? The question and desired information is unclear. What do you have now, and what would you like to have later?
 
DutchDaemon said:
Define 'configuration'. Do you want some hardware advice, or advice on system/software installation/configuration, or something like that? The question and desired information is unclear. What do you have now, and what would you like to have later?
Click to expand...

Now I have:

- CPU Intel Core i5
- Motherboard Supermicro X8SIL (http://www.supermicro.com/products/motherboard/Xeon3000/3400/X8SIL.cfm?IPMI=N)
- RAM 4GB
- SSD 4GB Verbatim flash for nanobsd 8.2 (PF shaping and NAT + quagga)
- On-board two Intel ethernet cards (Intel PRO/100 in dmesg.boot / em(4) drivers)

As a result of:

- Using only two cores out of four
- Maximum load for stable operation 640MBit per second and 100kpps

I want to:

Replace the built-in network card to Intel 10G XF SR Dual Port Server Adapter, or something else that supports SFP+

But I'm having these issues:

- Better to use the drivers from Intel or the built-in FreeBSD?
- Is a uniform load of processor cores?
- How much traffic will be able to route a router? 10G?
- That should change in the configuration options for even more performance?
 
allan_sundry said:
- Using only two cores out of four
Click to expand...
Are you sure your i5 has 4 cores? Not all of them do. FreeBSD will use 4 cores if you have them.

- Better to use the drivers from Intel or the built-in FreeBSD?
Click to expand...
The FreeBSD Intel drivers are written by Intel.

- Is a uniform load of processor cores?
Click to expand...
That depends on the application.

- How much traffic will be able to route a router? 10G?
Click to expand...
That I don't know.

- That should change in the configuration options for even more performance?
Click to expand...
Perhaps you should figure out where your bottlenecks are. Measure twice, cut once.
 
SirDice said:
Are you sure your i5 has 4 cores? Not all of them do. FreeBSD will use 4 cores if you have them.
Click to expand...

Code: 
#cat /var/run/dmesg.boot | grep CPU
CPU: Intel(R) Core(TM) i5 CPU         680  @ 3.60GHz (3600.03-MHz 686-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
p4tcc2: <CPU Frequency Thermal Control> on cpu2
p4tcc3: <CPU Frequency Thermal Control> on cpu3
da0: 3820MB (7823360 512 byte sectors: 255H 63S/T 486C)SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!

Code: 
#top -PSI
last pid: 74238;  load averages:  0.01,  0.00,  0.00                                                                                                      up 41+14:56:51  09:06:02
96 processes:  5 running, 67 sleeping, 24 waiting
CPU 0:  0.0% user,  0.0% nice,  1.5% system,  0.0% interrupt, 98.5% idle
CPU 1:  0.0% user,  0.0% nice,  0.0% system, 38.0% interrupt, 62.0% idle
CPU 2:  0.0% user,  0.0% nice,  0.0% system, 40.6% interrupt, 59.4% idle
CPU 3:  0.0% user,  0.0% nice,  0.0% system,  3.4% interrupt, 96.6% idle
Mem: 30M Active, 26M Inact, 265M Wired, 996K Cache, 112M Buf, 1672M Free
Swap: 

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   10 terabit     4 171 ki31     0K    32K CPU0    0 3192.3 335.21% idle
   11 terabit    24 -68    -     0K   192K WAIT    3 746.2H 75.63% intr
    0 terabit    11 -68    0     0K    80K -       0 855:39  0.54% kernel

That depends on the application.
Click to expand...

Routing of traffic to be balanced on all cores?

Perhaps you should figure out where your bottlenecks are. Measure twice, cut once.
Click to expand...

Network cards support MSIX, but at the same loaded only two of the four cores.

Why?

Code: 
# cat /var/run/dmesg.boot | grep em0
em0: <Intel(R) PRO/1000 Network Connection 7.1.9> port 0xdc00-0xdc1f mem 0xfb5e0000-
0xfb5fffff,0xfb5dc000-0xfb5dffff irq 16 at device 0.0 on pci2
em0: Using MSIX interrupts with 3 vectors
em0: [ITHREAD]
em0: [ITHREAD]
em0: [ITHREAD]
em0: Ethernet address: 00:25:90:00:59:d6

# cat /var/run/dmesg.boot | grep em1
em1: <Intel(R) PRO/1000 Network Connection 7.1.9> port 0xec00-0xec1f mem 0xfb6e0000-
0xfb6fffff,0xfb6dc000-0xfb6dffff irq 17 at device 0.0 on pci3
em1: Using MSIX interrupts with 3 vectors
em1: [ITHREAD]
em1: [ITHREAD]
em1: [ITHREAD]
em1: Ethernet address: 00:25:90:00:59:d7
 
According to Intel, Core i5 680 has two cores and Hyper-Threading enabled.
It seems that your machine is fine, as long as it doesn't cause packet loss, but I'm unsure if your current platform will be able to support two 10 GBps connections, it's unusual to see a LGA1156 platform with too many PCI Express x8 slots. For high-bandwidth applications, a server (or workstation) class chipset should be used.
The way I see it, you need:
- a platform with at least two PCI Express x8 (or x16) slots, preferably a server board with IPMI or other remote management capabilities and platform status reporting. This should be fine: Intel® Workstation Board WX58BP.
- a CPU capable to work at high clock rates on at least two cores, a true quad-core should be fine, a hexa-core doesn't fit your actual usage. For the prevoius motherboard, a compatible processor is Intel Core i7 950, Intel Core i7 940 or Intel Xeon W3540.

Of course, a RAID controller should also be used and configured to send alerts on critical events.
 
ecazamir said:
According to Intel, Core i5 680 has two cores and Hyper-Threading enabled.
It seems that your machine is fine, as long as it doesn't cause packet loss, but I'm unsure if your current platform will be able to support two 10 GBps connections, it's unusual to see a LGA1156 platform with too many PCI Express x8 slots. For high-bandwidth applications, a server (or workstation) class chipset should be used.
The way I see it, you need:
- a platform with at least two PCI Express x8 (or x16) slots, preferably a server board with IPMI or other remote management capabilities and platform status reporting. This should be fine: Intel® Workstation Board WX58BP.
- a CPU capable to work at high clock rates on at least two cores, a true quad-core should be fine, a hexa-core doesn't fit your actual usage. For the prevoius motherboard, a compatible processor is Intel Core i7 950, Intel Core i7 940 or Intel Xeon W3540.

Of course, a RAID controller should also be used and configured to send alerts on critical events.
Click to expand...

thanks

what version of freebsd is better to use an i386 or amd64?
will give a performance compilation of systems using gcc4.5 instead of the standard?
what settings should be changed to increase network performance?
 
allan_sundry said:
what version of freebsd is better to use an i386 or amd64?
Click to expand...

I usually avoid using the 64 bit version if my application doesn't require huge amounts of memory. Typically, databases use such amounts of memory, requiring more than 2 GB per process. I don't see any reason why 3 GB RAM wouldn't be enough for your application.
 
I can't tell if pf or ipfw are better suited for molti-core processing. Look how is ipfw (~30 rules) using a quad-core system with ~ 150 MBps in and ~120 mbps out traffic (approx. 17kpps per direction, the numbers reffers only to the wan side) and no pf:
Code: 
last pid: 32841;  load averages:  0.08,  0.11,  0.08    up 3+15:27:33  12:59:00
233 processes: 8 running, 202 sleeping, 23 waiting
CPU 0:  1.1% user,  0.0% nice,  0.8% system, 35.0% interrupt, 63.2% idle
CPU 1:  1.9% user,  0.0% nice,  0.4% system,  5.6% interrupt, 92.1% idle
CPU 2:  1.9% user,  0.0% nice,  0.0% system,  4.5% interrupt, 93.6% idle
CPU 3:  0.8% user,  0.0% nice,  1.5% system,  6.0% interrupt, 91.7% idle
Mem: 399M Active, 639M Inact, 429M Wired, 315M Buf, 1499M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root       171 ki31     0K    64K CPU3    3  82.3H 97.36% {idle: cpu3}
   11 root       171 ki31     0K    64K CPU2    2  82.8H 96.00% {idle: cpu2}
   11 root       171 ki31     0K    64K RUN     1  81.8H 94.87% {idle: cpu1}
   11 root       171 ki31     0K    64K CPU0    0  65.0H 58.98% {idle: cpu0}
   12 root       -68    -     0K   416K WAIT    0 630:36 20.65% {irq257: bce0}
   12 root       -68    -     0K   416K WAIT    0 415:54 13.67% {irq26: bge1}
   12 root       -44    -     0K   416K WAIT    1 177:38  7.76% {swi1: netisr 1
   12 root       -44    -     0K   416K CPU2    2 195:18  6.49% {swi1: netisr 2
   12 root       -44    -     0K   416K CPU0    0 153:46  4.39% {swi1: netisr 0
   12 root       -44    -     0K   416K WAIT    3 206:44  4.30% {swi1: netisr 3
   12 root       -64    -     0K   416K RUN     0  85:15  1.66% {irq18: uhci2}
 2785 root        44    0 45700K 26580K select  3  39:26  0.00% snmpd
    9 root        44    -     0K    16K ipmire  3  21:50  0.00% ipmi0: kcs
   12 root       -32    -     0K   416K WAIT    0  20:04  0.00% {swi4: clock}
    0 root       -68    0     0K   112K -       2  19:44  0.00% {dummynet}
 2564 root        44    0 77920K 26864K select  2  15:04  0.00% {mpd5}
   22 root        44    -     0K    16K syncer  2  14:03  0.00% syncer
   13 root        44    -     0K    16K -       1   9:51  0.00% yarrow
 3012 mysql       44    0  1710M   310M select  1   6:22  0.00% {mysqld}
   12 root       -68    -     0K   416K WAIT    0   4:53  0.00% {irq24: bge0}
 3012 mysql       44    0  1710M   310M select  1   4:48  0.00% {mysqld}
 
You must log in or register to reply here.
Back
Top