Optimal Router Hardware

I couldn't decide if this should go in this section or the networking one but I decided since it's only about the hardware side it best belonged here. Also, I know this has been asked before, but I couldn't find any posts that were recent and specific to my needs.

So, I have been looking at getting FTTH and am looking at using FreeBSD as a router. The speed I'm looking at is 2gbit so a conventional home router won't handle it and enterprise routers are stupidly expensive compared to their stated performance. So, I decided hey, let's build a FreeBSD server to act as a router.

Here's the plan so far as far as hardware goes: Fiber into a Mellanox ConnectX-3 10GbE SFP+ card. Passive copper from another Mellanox ConnectX-3 10GbE SFP+ card going to a Ubiquiti Networks US-16-XG (10G 16-port managed switch). From there it will branch out into a Linksys EA9500 for wireless (in bridge mode) and a Ubiquiti BULLET-M2 (also in bridge mode) as well as switches, etc.

The plan is to be truly dedicated to this task, nothing else will be on this hardware. It would perform the same functions as a typical home router. This includes routing, standard firewall (closed ports/port forwarding), DHCP, DNS, QoS (to some extent), etc. No databases, web servers, media servers, etc.

So, that brings me to my question. What would the optimal CPU be for this task? I don't want to skimp on it, but I also don't want to go way overkill. I want to reduce latency and maximize PPS. What matters most here? Single-core speed? The number of cores? How much and how many are needed? Intel vs AMD? ECC sounds like a good idea, so if Intel a Xeon would be needed. That might make AMD a better choice if tons of CPU isn't needed while the higher end Intels tend to perform better. Speaking of RAM, how much RAM would be ideal? I've used FreeBSD for a long time, but I've never used it for this before. If you are going to recommend a barebone system, I'm going for a 1U/2U form factor. Power consumption is not a top concern, performance is more important.

Any advice about anything I've said is appreciated. If you see anything in my setup that seems wrong or has a better choice available feel free to let me know. Also, if another flavor of BSD would be better-suited feel free to state the case for it, though I'm more comfortable using FreeBSD as I have not used the others at all.

Thanks.
 
Have you thought about using OpenBSD? Their PF is under active development and I remember reading from somewhere that with Mellanox hardware you don't have to deal with "big lock" (you'll get proper SMP performance) on OpenBSD. Also their PF is supposedly better performing (some claim ~4x better) than version/fork FreeBSD has.
I realize this is FreeBSD forum but I have no bias against neither BSD and you asked for an opinion.
 
I want to reduce latency and maximize PPS. What matters most here? Single-core speed? The number of cores?
I went the Chelsio T540 route and stuffed 4 in a X9SRL. This is single socket LGA2011 using 16GB RAM and mirrored 16GB SSD.
It is extreme overkill. I wanted to ensure all 4 cards had x8 lanes of PCIe 3.0.
I am using an E5-2618LV2. I downgraded from 2650LV2 as no need for all those cores for a router/softswitch.
In my opinion a Core I3 quad core chip is plenty. Skylake 1151 chip like E3-1240LV5 is ideal drawing only 25W TDP.
Just remember that LGA1151 only has x16 lanes of PCIe3. So MATX is fine. For a single ConectX3 you could use ITX.
Use an industrial grade board for 24x7 ops. I prefer SuperMicro but also own a server grade Gigabyte Skylake MATX board.
Single core speed is only a factor when using something like PPPoE whereas PPP is a single threaded application.
There are ways around these limitations as well by using the mpd5 daemon.

No need for more than 8GB RAM as FreeBSD networking&routing has real low overhead. pf really doesn't need much umph either.
You start adding monitoring tools like Suricata or Snort and you might need to step up the RAM & cores some.
1U case with a 90 degree riser should work fine.
I used hotswap Emacs power for versatility. They are loud but dependable.
Because my network cards were full heigth I went with a 3U chassis I had in storage. Built a heat shroud for the fan to maximize airflow over the Chelsios. They run really hot.
 
I really couldn't care if FreeBSD's pf is versions behind OpenBSD's version.
All it does is filter packets. No need for all the fancy features.
That is what is important to me. It is more likely another OS component that will be vulnerable.
Thus use the operating system you know the best as a bad setup is more likely than a pf vulnerability.
Do notice that OpenSSH is a frequent culprit on OpenBSD.

FreeBSD bugs are more varied in scope.
Unfortunately to undercut my argument FreeBSD did have a pf bug very recently.
 
Last edited:
I really couldn't care if FreeBSD is versions behind OpenBSD's version.
All it does is filter packets. No need for all the fancy features.
That is what is important to me.

I have a working OpenBSD installation on my IBM T43 but that's my Precious and never use it. I do use the same pf ruleset on FreeBSD and OpenBSD with the expection of changing one word in the egress rule on OpenBSD and both work fine. MIne is set to heavily block incoming traffic and I don't have any services running so rarely go beyond looking at the stats in the daily Security Mailings.
 
Do notice that OpenSSH is a frequent [vulnerability] culprit on OpenBSD.
I think that your reasons for choosing FreeBSD are sound, but might the reason that OpenBSD has lots of OpenSSH issues listed be because it is the development platform for OpenSSH?
 
Take a look at this one:


I have their older "Vigor 3200n" SOHO/Multi-WAN router that was given to me, many years ago, as a demo. But, I was never asked to return it. Although, they stopped updating firmware on this router, it still works OK, in my friend's small resort/ hotel/restaurant with 2x100mbps WAN connections and my tech/admin support ;) It has very nice and simple to navigate UI with tons of features.
 
FWIW, I have a 1 Gbps internet connection (technically it is FTTH, but it ends in a Genexis Platinum 6840 gateway, so my router / gateway / firewall only sees a Gigabit Ethernet interface). My gateway / router / firewall is a Shuttle XH61V, Celeron G540 cpu, 8 GB RAM, an SSD for storage, it has two gigabit ethernet interfaces. It runs FreeBSD, I use ipfw as firewall.
The machine never has any load;
Code:
root@kg-omni1# date;uptime
Tue Oct 22 20:25:04 CEST 2019
 8:25PM  up 62 days, 12:34, 1 user, load averages: 0.00, 0.00, 0.00
it looks like this all the time, no matter what kind of traffic I'm running through the internet connection.
 
I have CTTW from a wooden pole with a shitty 100Mbps down and 50Mpbs up, because I live in a digital area shit hole. My ISP's IP/GW router's firmware is too old. My own LAN to GW-IP/WAN router is an old laptop that runs *BSD firewall connected to another old laptop, over dumb switch, with FreeBSD server that is also protected by another FW. I have minimal understanding of how the automagic of telecomm switches, TCP/IP routers, gateways, bridges and servers with their service ports, work. I just play, pray and hope that the chips and software running the monster called Internet, with its TCP/IP services, has enough AI to take care of itself and me – hehe

Even if I had FTTW and those slick IP/GW routers, none would help me and my dumb-ass TCP/IP show. But, I think that an IP/GW router with customized embedded OS kernel, TCP/IP-FW and related utilities is more efficient than poorly developed or configured router, on PC with a generic OS kernel.
 
I used to use Draytek but have become a fan on Mikrotik for small office or home use now. Something like the well named RB4011iGS+5HacQ2HnD-IN would be a great router for a FTTH connection, isn't badly priced compared to "higher end" brands and it will happily route at more than 2Gb, even with 25+ firewall rules. They also have a cheaper desktop/rackmount version with a slightly less ridiculous name (RB4011iGS+RM) if you don't need built-in wifi.

Personally I tend to avoid homebrew routers. I couldn't build something to outperform the above Mikrotik for less money (unless I had a half decent machine lying around already), it would use more power and configuring/managing it, especially more advanced routing or IPsec configs, would be far more hassle.
 
So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?
 
Probably.

I use an OPNsense-powered HP ProDesk 400 G4 as my "router" on 300 Mbps Verizon FTTH, connected directly to the Verizon ONT (GPON to Ethernet media converter). I know its overkill when compared to many low-power solutions, but I had trouble optimizing various "Mini PCs" (both HP and AliExpress/QCY models) so I decided to use a real desktop. And when >1 Gbps broadband comes, I can just swap the NICs.
 
So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?
Nowadays, any half-ass modern PC hardware and older decent platfrom with SSD, including laptops, can be used as a IP/GW router for a simple LAN, or as advanced as you want, if you know howto PF, IPFW and trim your FreeBSD kernel to do just what you need in your router. The problem with consumer level PC(s) with their CPU(s) and generic kernels is that both waist tons of resource for things that you maybe or will do, with your CPU and OS. That's what developers and/or manufacturers of those slick and small footprint routers are charging for - custom hardware platfrom, ARM based CPU(s) and kernels to support all the fancy TCP/IP utilities ,with point and click UI, that you may or may not ever need :)

ATM, I'm experimenting with FreeBSD to learn howto make my laptop a better router, than it's now, with things like DOS protection and few extra futures. I also want to run authoritative and caching DNS servers in the router.
 
Nowadays, any half-ass modern PC hardware and older decent platfrom with SSD, including laptops, can be used as a IP/GW router for a simple LAN, or as advanced as you want, if you know howto PF, IPFW and trim your FreeBSD kernel to do just what you need in your router. The problem with consumer level PC(s) with their CPU(s) and generic kernels is that both waist tons of resource for things that you maybe or will do, with your CPU and OS. That's what developers and/or manufacturers of those slick and small footprint routers are charging for - custom hardware platfrom, ARM based CPU(s) and kernels to support all the fancy TCP/IP utilities ,with point and click UI, that you may or may not ever need :)

ATM, I'm experimenting with FreeBSD to learn howto make my laptop a better router, than it's now, with things like DOS protection and few extra futures. I also want to run authoritative and caching DNS servers in the router.

However, you probably can't if you have a Gigabit FTTH connection which uses PPPoE. Some telcos like Bell Canada, CenturyLink (US), and NTT (Japan) are like this. I heard of people having trouble with PPPoE on Bell and CenturyLink Gigabit connections with pfSense.

I have Verizon FiOS FTTH and they use DHCP, and I never had Gigabit broadband, very less PPPoE on one. I have 300 Mbps, and if it weren't for my Tor relay, I would only have 100 Mbps (the most I do is ISO downloads or YouTube otherwise). But if US broadband was as cheap as Europe or Asia, maybe I'll go Gigabit, who knows?

My HP ProDesk handles 300 Mbps without PPPoE just fine, and can probably handle a Gigabit as well. Heck, I could do multi-Gigabit when it comes with just a NIC swap. It probably can also do PPPoE Gigabit, but I don't know.
 
So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?
Whether you go turnkey or DIY, a lot of the small form factor PCs and Mikrotik appliances you see on places like Amazon have CPUs, like the Celernon J1900, that don't support "Intel AES New Instructions (AES-NI)". For a new investment in a firewall, AES-NI is a "must have" for future-proofing.
 
Whether you go turnkey or DIY, a lot of the small form factor PCs and Mikrotik appliances you see on places like Amazon have CPUs, like the Celernon J1900, that don't support "Intel AES New Instructions (AES-NI)". For a new investment in a firewall, AES-NI is a "must have" for future-proofing.
If you are just doing packet forwarding (like me), AES-NI isn't necessary. If you want to do VPN, it's a must-have.

Even if you are just doing packet forwarding, get something AES-NI compatible. Maybe you'll want VPN in the future, or if you ever resell your device your resale value is higher since the next owner can also do VPN.

I skip the Mini PCs and use a real desktop, tuning a bicycle is too hard so I'll just use a motorbike. But then, I have FTTH and run Tor relays on my connection, many of you may have fiber but will never run Tor and in that case a Mini PC works pretty well.

If you don't mind non-FreeBSD, a Ubiquiti box could also be a good buy. I never had one but heard great things about them.
 
However, you probably can't if you have a Gigabit FTTH connection which uses PPPoE. Some telcos like Bell Canada, CenturyLink (US), and NTT (Japan) are like this. I heard of people having trouble with PPPoE on Bell and CenturyLink Gigabit connections with pfSense.

Sorry for generalizing and characterizing my FreeBSD router. What I, maybe, meant was more like LAN/IP/FW router.

In my case, I operate on both Static LAN/IP's, and Static Public IPs from my ISP. So, my FreeBSD router handles my LAN or VLAN network traffic on one network interface and my WAN Public IP on another (USB to Ethernet conversion cable) network interface which get routed through my IPS's LAN-GW/IP pass-through router with a very basic LAN port forwarding and security options.

That's why I consider my laptop with FreeBSD and two network interfaces an IP/GW router, and my other laptop with FreeBSD, connected over dumb switch to my FreeBSD router, a server. All other computers, that connect over my dumb switch to my FreeBSD router, just sit there and wait to be updated, upgraded or hacked-up :)

Edit:
Almost forgot,
I also have old wire/wireless router, for my wireless devices, connected to my ISP's router, which I also consider an IP/GW router - hehe
 
The global switches connect to modems, modems connect to routers. Then, routers connect to more routers and switches. And, somewhere in-between are my IP/GW routers 😁
 
Get a i3 with good ethernet ports, 16gb ram and you should be fine.
Mine had an older i3 and rarely didn't see any cpu usage, hitting almost 1gbit from my 1gbit ISP. Think I was averaging on 987mbit or something.
 
If it performs well as a server, it generally will perform better as a router. You're only concerned up to ISO layer 4 (TCP/UDP ports) with the primary application of this box. With servers you're concerned about performance all the way up to layer 7. Firewalling, routing and VPN are not that big of a deal or CPU intensive in my experience.
 
I changed my firewall again, to a HP T730. I'm also moving, where my new ISP is Gigabit Wave G versus my old ISP, 300 Mbps Verizon FiOS. Interesting to see how the T730 will scale on a full Gigabit.

The T730 is a decent box, but has some issues with certain Intel NICs, some people had success with "genuine" (meaning non-counterfit) Intel T350 cards, I went with a Dell Broadcom 5720 unit. If you want a T730 and have the guts to go Broadcom, just do it.

I repurposed my previous firewall, a HP ProDesk 400 G4, as a desktop. I did a few upgrades to the CPU (Pentium->i7) and RAM (8GB->24GB) and it works pretty well. Not as powerful as my workplace's Dell Precision, but powerful enough for compiling Ports as a maintainer.
 
So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?
Yes.

At the risk of carbon dating myself, I've had dedicated FreeBSD firewalls and routers for years. Even a half duplex ISA NIC if you're running an old or slow broadband connection will do the job. 20 years ago, I had an old crap P54C-90 (socket 5) system, 32 MB RAM, i430VX chipset with an SMC 8013 16-bit ISA NIC (ed(4)) going to a cable modem connection at half duplex via a crossover cable. I forget what internal NIC I used; probably an Intel 8461 Pro/100+ management; fxp(4). The cable modem had a slower upload than download speed so half duplex wasn't a major issue. The CPU was pretty much yawning waiting for something to do. I was using ipfw(8) at the time before one of its rewrites. This was before pf(4) existed. Pf is probably what you should be using on your dedicated router.

An HP Proliant Microserver G7 could probably do the job as it has a PCIe 2.0 x16 slot. I would get a system that has support the same PCIe version or newer as your 10GbE NIC to get the maximum bus speed of the NIC. Packet filtering is not very CPU intensive especially if you have a dedicated box for it and a good NIC that does some of the L2 processing on the card.

What will hammer a router CPU are (distributed) DoS attacks such as with the Blaster worm. Construct your ACLs accordingly or it won't make a bit of difference how much bandwidth, CPU and memory you throw at your router. In fact, higher bandwidth can cause the CPU to go into clock speed arrest.

I ended up pulling the overtime CERT duty to deworm the WAN. It knocked out over 200 Cisco 2503 routers and a pair of 7513s. The ICMP traffic was causing a WAN wide distributed DoS attack. We kept the anti Blaster worm ACL to this day (or at least of as of when I retired on July 11, 2013).

I could also tell you the horror story of a real librarian of genius (*hums Budlight's Real Men of Genius theme*) took down a site's WAN connection (on the date payroll was due to be authorized) leeching via Napster by saturating their WAN connection. Bittorrent client users that do not throttle their bandwidth and go through a VPN are also known to do this. I had to hunt down one of those real men of genius as well. There I go again, carbon dating myself. :cool:

I don't know anything about your switching hardware. I'm a Cisco purist by trade. Your Linksys is supported by DD-WRT which makes it good in my book.
 
Last edited:
I just bought a used Lanner NCA-1010B that was a Untangle branded box.
It is the smallest router in my inventory.

Some of the shelf sized routers I have bought from ebay:
Checkpoint U-5
Jetway JCB375
Caswell CAD-205
Lanner FW-7535
Lanner FW-7525
Nexcom DNA-110
Sophos XG85
Sophos XG105
Sophos SG135
Astaro ASG110 rev4
PCEngines APU1,2,3
 
Have you looked at PfSense for a router? It uses FreeBSD as the operating system. I put one together several years ago with an ITX motherboard the interface is through a web page for maintenance.
 
Back
Top