PF How BSD pf performance depend on CPU frequency, L2/L3 cache size

Sergei_Shablovsky

Member

Reaction score: 2
Messages: 33

Hi, hardware gurus!

How exactly BSD pf performance (in matter of low latency, high PPS, etc) depend on bus frequency, main CPU frequency and L2/L3 cache size in multi-package (mean physical multi-CPU, like Intel E5500/5600, E5-2000 series) server systems that intend working as border firewall with a lot of double/quad PCIe 2.0+ NICs ?

Note, we not speak about using firewall in VMs, only bare metal + FreeBSD.

THANK You for attention and much detailed answers!
 

Kristof Provost

Active Member
Developer

Reaction score: 71
Messages: 120

I'm not aware of anyone having done a benchmarking series across different CPUs, so there's not really any information there.

It may be more useful for you to tell us what sort of performance your application needs.

To give you an idea, the latest version of FreeBSD pf can push around 18Mpps (in simple setups) on my particular hardware configuration.
 
OP
Sergei_Shablovsky

Sergei_Shablovsky

Member

Reaction score: 2
Messages: 33

I'm not aware of anyone having done a benchmarking series across different CPUs, so there's not really any information there.
Or may be because this is property of some IT company. ;)

It may be more useful for you to tell us what sort of performance your application needs.
On moment now, interesting
1. pfSense (lets name it “GUI for pf + some other stuff”)
2. HAProxy

Speeds 10-30G, copper Eth.
Intel CPUs

To give you an idea, the latest version of FreeBSD pf can push around 18Mpps (in simple setups) on my particular hardware configuration.
Please could You describe Your setup (of course just drop parts about security). Thanks!
 

Kristof Provost

Active Member
Developer

Reaction score: 71
Messages: 120

On moment now, interesting
1. pfSense (lets name it “GUI for pf + some other stuff”)
You'll want to talk to Netgate about that. They have more performance numbers than I do.

2. HAProxy

Speeds 10-30G, copper Eth.
Intel CPUs
In 64-byte packets? 1500 bytes? 9000 bytes?

Please could You describe Your setup (of course just drop parts about security). Thanks!
It's a development setup. It's entirely artificial.
 
OP
Sergei_Shablovsky

Sergei_Shablovsky

Member

Reaction score: 2
Messages: 33

You'll want to talk to Netgate about that. They have more performance numbers than I do.
Fellows from Netgate user forum not agree with that: most of them keep pfSense-based firewalls for small net (hotel, campus, 10-20 people’s office, etc...), fall in love to GUI, and not any motivation to goes down into bus, cpu, NIC, and FreeBSD network stack working...
So, best of them just redirect me on FreeBSD user forum or the same Calimet articles.

In 64-byte packets? 1500 bytes? 9000 bytes?
1500

It's a development setup. It's entirely artificial.
May be interesting anyway. ;)

Thank You!
 

Kristof Provost

Active Member
Developer

Reaction score: 71
Messages: 120

Fellows from Netgate user forum not agree with that: most of them keep pfSense-based firewalls for small net (hotel, campus, 10-20 people’s office, etc...), fall in love to GUI, and not any motivation to goes down into bus, cpu, NIC, and FreeBSD network stack working...
So, best of them just redirect me on FreeBSD user forum or the same Calimet articles.
They are mistaken. I contract for NetGate (but explicitly do not speak for them). They have a team dedicated to performance testing. Talk to them, they can help.
Many users use the free (community edition, CE) version, but Netgate has commercial offerings, including support, that are suitable for business use at larger scale than the examples you list here.

May be interesting anyway. ;)
That setup uses Dell T640s (Intel Xeon Silver 4210 2.2G, 10C, 32GB RAM) with Chelsio T62100 NICs. I stress that this is a development setup and not a production system with real traffic.
 
OP
Sergei_Shablovsky

Sergei_Shablovsky

Member

Reaction score: 2
Messages: 33


Some sub-question (not needed to create separate tread because question are too close to this tread):

How EXACTLY the technologies:

Intel SpeedStep
Multi-Threads
CPU Enhanced Halt (C1E)
C3/C6/C7 State Support
CPU EIST Function
AHCI
QPI Frequency
Memory Scrubs

impact on FreeBSD performance as a 24/7 Firewall (pf / ipfw) ?

P.S. For example, I definitely know that C3 on 1-thread application (like pf) cause going to sleep other cores and small speed up the 1-st core.
 

smdb01us

Member

Reaction score: 3
Messages: 21

IMHO, knowing how exactly a system will perform (compute + pf in this case) depends on some many factors that it would be impossible to guess. When companies like Netgate give you specs on how their appliances perform is because they have a discrete and consistent set of HW and variables to perform tests on and even then, you can only hope that their “real traffic” (that is short/long lived flows, udp traffic, tcp optimizations based on OSs, MTU, applications, etc) and setup (number of rules, options, clients behind it) matches your real traffic and setup. If your hardware deviates from what it has been tested, your best approach would be test it yourself and if you don’t have the hardware yet, look for approximate HW to other tests and extrapolate. Cheers.
 
OP
Sergei_Shablovsky

Sergei_Shablovsky

Member

Reaction score: 2
Messages: 33

IMHO, knowing how exactly a system will perform (compute + pf in this case) depends on some many factors that it would be impossible to guess. When companies like Netgate give you specs on how their appliances perform is because they have a discrete and consistent set of HW and variables to perform tests on and even then, you can only hope that their “real traffic” (that is short/long lived flows, udp traffic, tcp optimizations based on OSs, MTU, applications, etc) and setup (number of rules, options, clients behind it) matches your real traffic and setup. If your hardware deviates from what it has been tested, your best approach would be test it yourself and if you don’t have the hardware yet, look for approximate HW to other tests and extrapolate. Cheers.

In common, I agree with You and other here.

So, at the some point of discussions like this, each time we conclude to “play” on a real hardware in a lab environment.

Because we speak about only one use case - firewalling, the test we really need is to simulate “real traffics”, change the settings (BIOS/UEFI) and make measurement.
(I write «BIOS/UEFI” because initial message in this thread was about hardware technologies: power management (ACPI, etc) , memory management (fast scrub, patrol scrub, mirrored ram banks, etc), CPU proprietary functionality (SpeedStep, Multi-Threads, etc...)

What set of software You (and May be some others) suggest for this sort of testing?

iperf/iperf3, rdmon, ping, what else?
 

smdb01us

Member

Reaction score: 3
Messages: 21

iperf/iperf3, rdmon, ping, what else?
Can you invite the IT and marketing departments for a ping party? :D. Seriously tho, iperf, and similar tools are good to test throughput and latency for a small number of flows but to simulate a real-world deployment, you will need something that will generate hundreds of flows per client so it can stress the CPU, memory and NICs. I seem to remember reading in this forum that pf is not multi-threaded so I would also research if multiple cores and threads will make a difference or if just a low number of cores and high clock will do the trick.

I haven't tested any of these tools but it looks like there are several tools like Seagul (http://gull.sourceforge.net/index.html) that can help.
 

Jose

Daemon

Reaction score: 904
Messages: 1,110

...I seem to remember reading in this forum that pf is not multi-threaded so I would also research if multiple cores and threads will make a difference...
"This implementation is derived from OpenBSD 4.5. It has been heavily modified to be capable of running in multithreaded FreeBSD kernel and scale its performance on multiple CPUs."
 
Top