FreeBSD as Loadbalancer - Bad performance

D

devnull82

Hello!

Im using linux with ucarp and haproxy as loadbalancer / ha since a while. Now i test FreeBSD and get bad performance values.

The FreeBSD Setup:

Interface carp0 (renamed intel e1000 NIC) has a public IP to get requests
Interface carp1 (renamed intel e1000 NIC) has a private IP to comunicate with the webservers.

I do the following tuning options:

/etc/sysctl.conf:
Code: 
net.inet.icmp.icmplim=20000
kern.ipc.maxsockbuf=4000000000
kern.ipc.somaxconn=32768
kern.maxfiles=1000000

/boot/loader.conf:
Code: 
kern.maxusers=38400

The FreeBSD machine (Who have 8 XEON 2GHz Cores) can manage arround 30MBytes per second. Then I/O-Errors begin to raise.

netstat -I carp0 1

Code: 
           input        (carp0)           output
   packets  errs      bytes    packets  errs      bytes colls
     10393     0    2264257       7089     0    5438558     0
     15622     0    3406790      10768     0    8724167     0
     20562   401    4528858      14125     0   11623182     0
     20227   557    4390213      14125     0   11496360     0
     20106   545    4226344      14413     0   12472746     0
     20051  1584    4268895      14227     0   12574244     0
     20136  1375    4233550      14162     0   13361860     0
     19749   850    4206581      14186     0   12779683     0
     19837  1056    4242046      14132     0   12901977     0
netstat -I carp1 1

Code: 
          input        (carp1)           output
   packets  errs      bytes    packets  errs      bytes colls
     20548     0   11832097      19819     0    4399361     0
     19864     0   10614310      19343     0    4418355     0
     20295     0   11123973      19810     0    4449668     0
     20153     0   10967781      19496     0    4459658     0
     19757     0   10629329      19059     0    4351931     0
     20554     0   11343990      19920     0    4532443     0
     20590     0   11495033      19800     0    4427276     0
     20118     0   10914201      19670     0    4422610     0
     20246     0   11046725      19705     0    4405509     0
     20316     0   10942016      19634     0    4492243     0
     20296     0   10929203      19665     0    4507805     0
     20717     0   11812626      19871     0    4503441     0
     20470     0   11419763      19723     0    4419329     0
     20802     0   11884091      20248     0    4478751     0
     19946     0   10584217      19612     0    4442248     0
     20126     0   10604589      19551     0    4473400     0
     20347     0   10826504      20049     0    4483137     0
     20014     0   10718654      19408     0    4479361     0

The average CPU utilisation (sys) is at ~80%

The same setup with Linux on a 4 CPU machine can handle 33MBytes / second.
 
What version of FreeBSD are you running? And for what architecture?
 
Why did you set maxusers so high? What happens if you remove those tuning options?
 
A buffer was to smal - so i increased some values like this.
I forgot to decrease it again. But the problems with the "low"-bandwidth happend befor i change it too.
 
See tuning(7) for tuning information. Normally you don't need to touch maxusers.

Are you running a custom kernel or GENERIC? If you're running GENERIC try building a custom kernel and add options DEVICE_POLLING. Turn on polling for the intel interfaces. See polling(4).
 
I already tested polling.
The interrupt of the em* cards already used all CPUs.
If i use polling - cpu.sys ~ 0%, cpi.irq ~ 85% avg
If i dontuse polling - cpu.sys ~ 85%, cpi.irq ~ 0% avg

The result is the same. So i use the default kernel again
 
Doesn't hurt to run a custom kernel though. Lean and mean ;)

It's interesting you only seem to get errors on the input of carp0. This leads me to believe the error may be caused by the other end of the cable.

Have you tried fixing the speed duplex settings?
 
Do you get the same low speed and errors if you remove CARP from the equation? ie, using em0/em1 directly, with only 1 box.

Are these PCI, PCI-X, or PCIe NICs?

If they are PCI-X/PCIe, have you enabled any of the msi/msi-x sysctls (they should be enabled by default, but double-check).
 
What tool/method are you using to create the load ? 30 MB/s (about 250 Mbps) seems extremely low for your hardware, which is 3 times as large as the one I run my 10 Gbps validations on. I used to get haproxy to support your load on a Celeron ULV 650 MHz without cache.

How many connections per second are you seeing, and how many concurrent connections ? You can find those numbers (current and max) in the haproxy stats page.

Could you also check that the support for kqueue is enabled in your haproxy build ? ("haproxy -vv"). Without it, poll() could be slower with many connections, causing larger latencies resulting in input buffer overruns.

Also, I second the question about PCI/PCIe/PCI-X. With legacy PCI 33 MHz NICs on the same bus, your forwarding performance is normally limited to about 300-350 Mbps, which approximately matches what you're observing. But according to the link you've posted, onboard NICs are PCIe so they should not be an issue if you're using them.
 
On my new workstation (AMD Phenom(tm) II X4 965 Processor (3400.16-MHz K8-class CPU) with RealTek Gigabit Ethernet NIC(NDIS 6.0) (RTL8168/8111) i have done over 60MB/s sustained, if that says something.
 
OK, the Realtek chip just has the ability to connect to gigabit network, but is unable to sustain real loads. That's why it's the cheapest on earth and the one installed on cheapest generic NICs and motherboards (although unfortunately Gigabyte also happens to cripple their motherboards with it). So that explains it for your workstation but not for your server. Do you have the information related to the other questions (version, build options, numbers...) ?
 
willy said:
Do you have the information related to the other questions (version, build options, numbers...) ?
Click to expand...

Sorry for the confusion Achix, I thought you were the original reporter. Of course you won't have his numbers :)
 
The load of 30Mbyte / sec (15Mbyte at the internal interface and 15MByte at the external interface) are real requests by site requests.
30MByte / Sec -> 30000 TCP Pakets / Sec. in my case.
 
devnull82 said:
I already tested polling.
The interrupt of the em* cards already used all CPUs.
If i use polling - cpu.sys ~ 0%, cpi.irq ~ 85% avg
If i dontuse polling - cpu.sys ~ 85%, cpi.irq ~ 0% avg

The result is the same. So i use the default kernel again
Click to expand...
With polling pps is same ? Have tried link0 ability? Fastforwarding? Try to touch sysctls net.inet.tcp.sendspace and net.inet.tcp.recvspace. And show ifconfig pls
 
I recently tested the NIC bandwidth of two identical machines (HP DL360 G4) with iperf (FreeBSD 8.0 AMD64) and achived a bandwidth of 910Mb/s. Some improvement compared to FreeBSD 7.2.
 
Alt said:
With polling pps is same ? Have tried link0 ability? Fastforwarding? Try to touch sysctls net.inet.tcp.sendspace and net.inet.tcp.recvspace. And show ifconfig pls
Click to expand...

Already wrote it:
If i use polling - cpu.sys ~ 0%, cpi.irq ~ 85% avg
If i dontuse polling - cpu.sys ~ 85%, cpi.irq ~ 0% avg

The result is the same. I dont test other things like link0 or fastforwarding. Whre i can find infos about it / what do it?
I have already played with all net-sysctl-values. Without usable results.

I recently tested the NIC bandwidth of two identical machines (HP DL360 G4) with iperf (FreeBSD 8.0 AMD64) and achived a bandwidth of 910Mb/s. Some improvement compared to FreeBSD 7.2.
Click to expand...

My problem is not the raw bandwidth. Can you test my senario with your server? 1 if with external ip, 1 if with internal ip, one or more webservers behind with internal ips and haproxy at the bsd machine. Then we get a real comparison.

At the moment i running linux again to handle the big traffic at this time.
 
Well at least your carp is working, I'm suffering from other issue under FreeBSD 8. We are using FreeBSD 7.2 and at pick pushing 60-80Mbps without any problem. I really wanted to use 8.0 but it is not working..
 
devnull82 said:
Already wrote it:
If i use polling - cpu.sys ~ 0%, cpi.irq ~ 85% avg
If i dontuse polling - cpu.sys ~ 85%, cpi.irq ~ 0% avg

The result is the same. I dont test other things like link0 or fastforwarding. Whre i can find infos about it / what do it?
I have already played with all net-sysctl-values. Without usable results.



My problem is not the raw bandwidth. Can you test my senario with your server? 1 if with external ip, 1 if with internal ip, one or more webservers behind with internal ips and haproxy at the bsd machine. Then we get a real comparison.

At the moment i running linux again to handle the big traffic at this time.
Click to expand...

Could you explain me more in detail how I can test your scenario?
Currently I have three HP DL360 G4 servers and one HP DL320 G5.
They have all FreeBSD 8.0 AMD64 installed and except for some GNU octave computations are not used.

By the way I was wrong about the numbers. Only the HP DL320 G5 gets close to the theoretical 1Gb/s. The others max raw bandwidth got around 750 Mb/s. This is understandable: the first one are NICs on PCI-E while the latter are NIC-s on a PCI-X bus.
 
You must log in or register to reply here.
Back
Top