Optimal Router Hardware

Ancyker

New Member


Messages: 2

I couldn't decide if this should go in this section or the networking one but I decided since it's only about the hardware side it best belonged here. Also, I know this has been asked before, but I couldn't find any posts that were recent and specific to my needs.

So, I have been looking at getting FTTH and am looking at using FreeBSD as a router. The speed I'm looking at is 2gbit so a conventional home router won't handle it and enterprise routers are stupidly expensive compared to their stated performance. So, I decided hey, let's build a FreeBSD server to act as a router.

Here's the plan so far as far as hardware goes: Fiber into a Mellanox ConnectX-3 10GbE SFP+ card. Passive copper from another Mellanox ConnectX-3 10GbE SFP+ card going to a Ubiquiti Networks US-16-XG (10G 16-port managed switch). From there it will branch out into a Linksys EA9500 for wireless (in bridge mode) and a Ubiquiti BULLET-M2 (also in bridge mode) as well as switches, etc.

The plan is to be truly dedicated to this task, nothing else will be on this hardware. It would perform the same functions as a typical home router. This includes routing, standard firewall (closed ports/port forwarding), DHCP, DNS, QoS (to some extent), etc. No databases, web servers, media servers, etc.

So, that brings me to my question. What would the optimal CPU be for this task? I don't want to skimp on it, but I also don't want to go way overkill. I want to reduce latency and maximize PPS. What matters most here? Single-core speed? The number of cores? How much and how many are needed? Intel vs AMD? ECC sounds like a good idea, so if Intel a Xeon would be needed. That might make AMD a better choice if tons of CPU isn't needed while the higher end Intels tend to perform better. Speaking of RAM, how much RAM would be ideal? I've used FreeBSD for a long time, but I've never used it for this before. If you are going to recommend a barebone system, I'm going for a 1U/2U form factor. Power consumption is not a top concern, performance is more important.

Any advice about anything I've said is appreciated. If you see anything in my setup that seems wrong or has a better choice available feel free to let me know. Also, if another flavor of BSD would be better-suited feel free to state the case for it, though I'm more comfortable using FreeBSD as I have not used the others at all.

Thanks.
 

aht0

Active Member

Reaction score: 56
Messages: 164

Have you thought about using OpenBSD? Their PF is under active development and I remember reading from somewhere that with Mellanox hardware you don't have to deal with "big lock" (you'll get proper SMP performance) on OpenBSD. Also their PF is supposedly better performing (some claim ~4x better) than version/fork FreeBSD has.
I realize this is FreeBSD forum but I have no bias against neither BSD and you asked for an opinion.
 

Phishfry

Son of Beastie

Reaction score: 1,530
Messages: 4,444

I want to reduce latency and maximize PPS. What matters most here? Single-core speed? The number of cores?
I went the Chelsio T540 route and stuffed 4 in a X9SRL. This is single socket LGA2011 using 16GB RAM and mirrored 16GB SSD.
It is extreme overkill. I wanted to ensure all 4 cards had x8 lanes of PCIe 3.0.
I am using an E5-2618LV2. I downgraded from 2650LV2 as no need for all those cores for a router/softswitch.
In my opinion a Core I3 quad core chip is plenty. Skylake 1151 chip like E3-1240LV5 is ideal drawing only 25W TDP.
Just remember that LGA1151 only has x16 lanes of PCIe3. So MATX is fine. For a single ConectX3 you could use ITX.
Use an industrial grade board for 24x7 ops. I prefer SuperMicro but also own a server grade Gigabyte Skylake MATX board.
Single core speed is only a factor when using something like PPPoE whereas PPP is a single threaded application.
There are ways around these limitations as well by using the mpd5 daemon.

No need for more than 8GB RAM as FreeBSD networking&routing has real low overhead. pf really doesn't need much umph either.
You start adding monitoring tools like Suricata or Snort and you might need to step up the RAM & cores some.
1U case with a 90 degree riser should work fine.
I used hotswap Emacs power for versatility. They are loud but dependable.
Because my network cards were full heigth I went with a 3U chassis I had in storage. Built a heat shroud for the fan to maximize airflow over the Chelsios. They run really hot.
 

Phishfry

Son of Beastie

Reaction score: 1,530
Messages: 4,444

I really couldn't care if FreeBSD is versions behind OpenBSD's version.
All it does is filter packets. No need for all the fancy features.
That is what is important to me. It is more likely another OS component that will be vulnerable.
Thus use the operating system you know the best as a bad setup is more likely than a pf vulnerability.
Do notice that OpenSSH is a frequent culprit on OpenBSD.

FreeBSD bugs are more varied in scope.
Unfortunately to undercut my argument FreeBSD did have a pf bug very recently.
 

Trihexagonal

Daemon

Reaction score: 1,027
Messages: 1,700

I really couldn't care if FreeBSD is versions behind OpenBSD's version.
All it does is filter packets. No need for all the fancy features.
That is what is important to me.
I have a working OpenBSD installation on my IBM T43 but that's my Precious and never use it. I do use the same pf ruleset on FreeBSD and OpenBSD with the expection of changing one word in the egress rule on OpenBSD and both work fine. MIne is set to heavily block incoming traffic and I don't have any services running so rarely go beyond looking at the stats in the daily Security Mailings.
 

gpw928

Active Member

Reaction score: 76
Messages: 224

Do notice that OpenSSH is a frequent [vulnerability] culprit on OpenBSD.
I think that your reasons for choosing FreeBSD are sound, but might the reason that OpenBSD has lots of OpenSSH issues listed be because it is the development platform for OpenSSH?
 

toorski

Active Member

Reaction score: 49
Messages: 157

Take a look at this one:


I have their older "Vigor 3200n" SOHO/Multi-WAN router that was given to me, many years ago, as a demo. But, I was never asked to return it. Although, they stopped updating firmware on this router, it still works OK, in my friend's small resort/ hotel/restaurant with 2x100mbps WAN connections and my tech/admin support ;) It has very nice and simple to navigate UI with tons of features.
 

tingo

Daemon

Reaction score: 432
Messages: 2,111

FWIW, I have a 1 Gbps internet connection (technically it is FTTH, but it ends in a Genexis Platinum 6840 gateway, so my router / gateway / firewall only sees a Gigabit Ethernet interface). My gateway / router / firewall is a Shuttle XH61V, Celeron G540 cpu, 8 GB RAM, an SSD for storage, it has two gigabit ethernet interfaces. It runs FreeBSD, I use ipfw as firewall.
The machine never has any load;
Code:
root@kg-omni1# date;uptime
Tue Oct 22 20:25:04 CEST 2019
 8:25PM  up 62 days, 12:34, 1 user, load averages: 0.00, 0.00, 0.00
it looks like this all the time, no matter what kind of traffic I'm running through the internet connection.
 

toorski

Active Member

Reaction score: 49
Messages: 157

I have CTTW from a wooden pole with a shitty 100Mbps down and 50Mpbs up, because I live in a digital area shit hole. My ISP's IP/GW router's firmware is too old. My own LAN to GW-IP/WAN router is an old laptop that runs *BSD firewall connected to another old laptop, over dumb switch, with FreeBSD server that is also protected by another FW. I have minimal understanding of how the automagic of telecomm switches, TCP/IP routers, gateways, bridges and servers with their service ports, work. I just play, pray and hope that the chips and software running the monster called Internet, with its TCP/IP services, has enough AI to take care of itself and me – hehe

Even if I had FTTW and those slick IP/GW routers, none would help me and my dumb-ass TCP/IP show. But, I think that an IP/GW router with customized embedded OS kernel, TCP/IP-FW and related utilities is more efficient than poorly developed or configured router, on PC with a generic OS kernel.
 

usdmatt

Daemon

Reaction score: 527
Messages: 1,418

I used to use Draytek but have become a fan on Mikrotik for small office or home use now. Something like the well named RB4011iGS+5HacQ2HnD-IN would be a great router for a FTTH connection, isn't badly priced compared to "higher end" brands and it will happily route at more than 2Gb, even with 25+ firewall rules. They also have a cheaper desktop/rackmount version with a slightly less ridiculous name (RB4011iGS+RM) if you don't need built-in wifi.

Personally I tend to avoid homebrew routers. I couldn't build something to outperform the above Mikrotik for less money (unless I had a half decent machine lying around already), it would use more power and configuring/managing it, especially more advanced routing or IPsec configs, would be far more hassle.
 
OP
OP
Ancyker

Ancyker

New Member


Messages: 2

So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?
 

neel

Member

Reaction score: 14
Messages: 60

Probably.

I use an OPNsense-powered HP ProDesk 400 G4 as my "router" on 300 Mbps Verizon FTTH, connected directly to the Verizon ONT (GPON to Ethernet media converter). I know its overkill when compared to many low-power solutions, but I had trouble optimizing various "Mini PCs" (both HP and AliExpress/QCY models) so I decided to use a real desktop. And when >1 Gbps broadband comes, I can just swap the NICs.
 

toorski

Active Member

Reaction score: 49
Messages: 157

So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?
Nowadays, any half-ass modern PC hardware and older decent platfrom with SSD, including laptops, can be used as a IP/GW router for a simple LAN, or as advanced as you want, if you know howto PF, IPFW and trim your FreeBSD kernel to do just what you need in your router. The problem with consumer level PC(s) with their CPU(s) and generic kernels is that both waist tons of resource for things that you maybe or will do, with your CPU and OS. That's what developers and/or manufacturers of those slick and small footprint routers are charging for - custom hardware platfrom, ARM based CPU(s) and kernels to support all the fancy TCP/IP utilities ,with point and click UI, that you may or may not ever need :)

ATM, I'm experimenting with FreeBSD to learn howto make my laptop a better router, than it's now, with things like DOS protection and few extra futures. I also want to run authoritative and caching DNS servers in the router.
 

neel

Member

Reaction score: 14
Messages: 60

Nowadays, any half-ass modern PC hardware and older decent platfrom with SSD, including laptops, can be used as a IP/GW router for a simple LAN, or as advanced as you want, if you know howto PF, IPFW and trim your FreeBSD kernel to do just what you need in your router. The problem with consumer level PC(s) with their CPU(s) and generic kernels is that both waist tons of resource for things that you maybe or will do, with your CPU and OS. That's what developers and/or manufacturers of those slick and small footprint routers are charging for - custom hardware platfrom, ARM based CPU(s) and kernels to support all the fancy TCP/IP utilities ,with point and click UI, that you may or may not ever need :)

ATM, I'm experimenting with FreeBSD to learn howto make my laptop a better router, than it's now, with things like DOS protection and few extra futures. I also want to run authoritative and caching DNS servers in the router.
However, you probably can't if you have a Gigabit FTTH connection which uses PPPoE. Some telcos like Bell Canada, CenturyLink (US), and NTT (Japan) are like this. I heard of people having trouble with PPPoE on Bell and CenturyLink Gigabit connections with pfSense.

I have Verizon FiOS FTTH and they use DHCP, and I never had Gigabit broadband, very less PPPoE on one. I have 300 Mbps, and if it weren't for my Tor relay, I would only have 100 Mbps (the most I do is ISO downloads or YouTube otherwise). But if US broadband was as cheap as Europe or Asia, maybe I'll go Gigabit, who knows?

My HP ProDesk handles 300 Mbps without PPPoE just fine, and can probably handle a Gigabit as well. Heck, I could do multi-Gigabit when it comes with just a NIC swap. It probably can also do PPPoE Gigabit, but I don't know.
 

gpw928

Active Member

Reaction score: 76
Messages: 224

So it sounds like I was overthinking things and pretty much any "decent" hardware will work fine?
Whether you go turnkey or DIY, a lot of the small form factor PCs and Mikrotik appliances you see on places like Amazon have CPUs, like the Celernon J1900, that don't support "Intel AES New Instructions (AES-NI)". For a new investment in a firewall, AES-NI is a "must have" for future-proofing.
 

neel

Member

Reaction score: 14
Messages: 60

Whether you go turnkey or DIY, a lot of the small form factor PCs and Mikrotik appliances you see on places like Amazon have CPUs, like the Celernon J1900, that don't support "Intel AES New Instructions (AES-NI)". For a new investment in a firewall, AES-NI is a "must have" for future-proofing.
If you are just doing packet forwarding (like me), AES-NI isn't necessary. If you want to do VPN, it's a must-have.

Even if you are just doing packet forwarding, get something AES-NI compatible. Maybe you'll want VPN in the future, or if you ever resell your device your resale value is higher since the next owner can also do VPN.

I skip the Mini PCs and use a real desktop, tuning a bicycle is too hard so I'll just use a motorbike. But then, I have FTTH and run Tor relays on my connection, many of you may have fiber but will never run Tor and in that case a Mini PC works pretty well.

If you don't mind non-FreeBSD, a Ubiquiti box could also be a good buy. I never had one but heard great things about them.
 

toorski

Active Member

Reaction score: 49
Messages: 157

However, you probably can't if you have a Gigabit FTTH connection which uses PPPoE. Some telcos like Bell Canada, CenturyLink (US), and NTT (Japan) are like this. I heard of people having trouble with PPPoE on Bell and CenturyLink Gigabit connections with pfSense.
Sorry for generalizing and characterizing my FreeBSD router. What I, maybe, meant was more like LAN/IP/FW router.

In my case, I operate on both Static LAN/IP's, and Static Public IPs from my ISP. So, my FreeBSD router handles my LAN or VLAN network traffic on one network interface and my WAN Public IP on another (USB to Ethernet conversion cable) network interface which get routed through my IPS's LAN-GW/IP pass-through router with a very basic LAN port forwarding and security options.

That's why I consider my laptop with FreeBSD and two network interfaces an IP/GW router, and my other laptop with FreeBSD, connected over dumb switch to my FreeBSD router, a server. All other computers, that connect over my dumb switch to my FreeBSD router, just sit there and wait to be updated, upgraded or hacked-up :)

Edit:
Almost forgot,
I also have old wire/wireless router, for my wireless devices, connected to my ISP's router, which I also consider an IP/GW router - hehe
 

toorski

Active Member

Reaction score: 49
Messages: 157

The global switches connect to modems, modems connect to routers. Then, routers connect to more routers and switches. And, somewhere in-between are my IP/GW routers 😁
 

Lars Skogstad

Member

Reaction score: 23
Messages: 74

Get a i3 with good ethernet ports, 16gb ram and you should be fine.
Mine had an older i3 and rarely didn't see any cpu usage, hitting almost 1gbit from my 1gbit ISP. Think I was averaging on 987mbit or something.
 
Top