Web Server supporting up to 4 WANs/Interfaces

J

jaysternet

I am in need of immediate help and while I rarely post on boards (I promise I'll start sharing more of my own wisdom in the future), this issue has me stumped like no other. First of all, I need multiple gateways. Yes, I understand there is no such thing as multiple "default" gateways and I have a solid grasp of networking in general...:). I also have a reasonable grasp of FreeBSD, though I am no where near the expert of many. I actually come from the world of programming and I've only used Linux and Windows in the past.

I am currently working on a big project that involves many sites and a custom appliance with multiple applications designed in-house. In fact, I might be the ONLY person in snowy Michigan hiring right now...:). After an exhaustive investigation, I chose FreeBSD over all other OSs. It was not a light choice, but I've been very pleased despite having zero knowledge at the start of it. The main reasons were flexibility, licensing and most important, I'm still shocked how open the community is to answering questions in posts, instead of putting people down for having less knowledge than the rude poster. I have found an answer to the most obscure questions through very quick google searches...until now. Kudos to those who take the time...FreeBSD far outnumbers other OS communities. Please help me avoid this issue being the big letdown, possibly forcing us to leave FreeBSD. I like the OS way too much already.

Now the issue. Without too much detail, my device has 4 GigE ports on it. Each will be attached to a routed network. There is NO routing required between networks inside the box (not a router or firewall) and in fact, it CANNOT be allowed to happen because of security. Instead, each WAN port needs access to this box, but nothing beyond. The access consists of a Web Server, though several other Ports are required, such as SNMP Traps, Syslog, etc. Getting to the box is easy, routers do all the work. The issue is getting traffic back through the same interface it came in on and through the same router gateway. As we all know, only 1 gateway can be assigned in FreeBSD, unlike other flavors of Linux. Even the ones who don't offer single line gateway support can use IPTables to accomplish this task. But IPTables is not supported in FreeBSD. not a bad thing as long as comparable solutions exist.

Setting up static routes is not the solution. The problem with it is that many sites will not have access to the next hop info from the gateway (the next hop gateway and subnet on the other side of the router). So I cannot use static routes.

PFSense appears to support this (though not tested by me). I REALLY do not want to go that route. We have invested 3 months into adding many apps to the FreeBSD we have. PFSense is a custom FreeBSD kernel with many changes. Many message boards claim it breaks many Ports and changes other behaviors. Even if it didn't, we are under deadline and moving everything over to a new FreeBSD Version and then extensively testing everything repeatedly again would be a nightmare. I am interested in experiences with it if it becomes the last resort, though.

I have tried both PF and IPFW. Different posts around the web claim Multiple Gateway solutions using both of them. I have tried each of the recommended setups, but had no luck. If you read the last responses to each of those posts, others also state they could not duplicate what is claimed. PF looks the most promising. It has "if-bound", which is supposed to keep interface traffic on the same interface. That is a good first step. But pointing it to the gateway on that interface is still an issue. Please HELP!!! I haven't slept in days and I've been stuck for a week now!!! This is our last showstopper.

Jay
 
Looking for something like this?

setfib(1)
setfib(2)

And pf.conf(5) has the route-to and reply-to directives, of course.

[cmd=]man 5 pf.conf | less +/route-to[/cmd]
[cmd=]man 5 pf.conf | less +/reply-to[/cmd]
 
I read yesterday about setfib, but there is extremely little about it and FreeBSD anywhere that I could find. There were no examples and the man page was no help. I have played around extensively with route-to and reply-to in PF. It seems like this would work. However, the online examples from others did not seem to work and I was unable to tinker properly with the syntax to get it to work, either. It could be that I'm missing something. Again...I first started playing with PF a few days ago and while others have posted great examples, the man pages are sparse with examples. Care to help with the proper syntax? I believe you will help a lot of other people given the thousands of posts out there. I'll do my part in helping others by giving exact examples of the network to use. We can use the following example-

em0 has an IP Address of 10.0.1.10 WAN traffic leaves the network via a Cisco router gateway of 10.0.1.254 In this case, we add 10.0.1.254 as the default gateway in rc.conf- defaultrouter="10.0.1.254"

em1 has an IP address of 10.0.2.10 WAN traffic leaves the network via another Cisco router gateway of 10.0.2.254.

Here is crux of the situation. How do we make this work? I have tried many different combos in PF. But lets start with 1 example and try to figure out where I'm screwing up. In rc.conf, I have no gateway_enable="yes" statement, since we are not routing from 1 network to the next. However, others may want to enable this for routing scenarios. In pf.conf, we should be able to use a very minimal config. I am not NAting anything internally. So I would think we need to use if-bound to keep traffic from using the default route and a route-to and reply-to statement for each interface. Correct? Here is a pf.conf config which makes sense to me, but does not work. I really don't need the routes for em0 since its default, but they are here in case I remove the default gateway. Running a pfctl -f /etc/pf.conf shows no syntax errors. Also...do you need to use varibles for each IP Address and Interface or can use use the values directly in the route-to statements? It does not error out when I use direct values, but every example online uses variables instead. So I have, as well. Anyways-

Code: 
#	$FreeBSD: src/share/examples/pf/pf.conf,v 1.1.4.1.4.1 2010/06/14 02:09:06 kensmith Exp $
#	$OpenBSD: pf.conf,v 1.34 2007/02/24 19:30:59 millert Exp $
#
# See pf.conf(5) and /usr/share/examples/pf for syntax and examples.
# Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1
# in /etc/sysctl.conf if packets are to be forwarded between interfaces.

#ext_if="ext0"
#int_if="int0"
ext_if1 = "em0"
ext_if2 = "em1"
ext_ip1 = "10.0.1.10"
ext_ip2 = "10.0.2.10"
ext_gw1 = "10.0.1.254"
ext_gw2 = "10.0.2.254"

#table <spamd-white> persist

#set skip on lo
#set block-policy return
set state-policy if-bound

#scrub in

#nat-anchor "ftp-proxy/*"
#rdr-anchor "ftp-proxy/*"
#nat on $ext_if from !($ext_if) -> ($ext_if:0)
#rdr pass on $int_if proto tcp to port ftp -> 127.0.0.1 port 8021
#no rdr on $ext_if proto tcp from <spamd-white> to any port smtp
#rdr pass on $ext_if proto tcp from any to any port smtp \
#	-> 127.0.0.1 port spamd

#anchor "ftp-proxy/*"
#block in
#pass out

#pass quick on $int_if no state
#antispoof quick for { lo $int_if } 
#  general "pass out" rules for external interfaces
#pass out on $ext_if1
#pass out on $ext_if2

#  route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
#  $ext_if2 and $ext_gw2
# pass out on $ext_if1 from $ext_if2 route-to ($ext_if2 $ext_gw2)
# pass out on $ext_if2 from $ext_if1 route-to ($ext_if1 $ext_gw1) 
#pass out quick route-to (em0 10.0.1.254) from 10.0.1.10 to any
#pass out quick route-to (ext_if2 10.0.2.254) from 10.0.2.10 to any
#rdr on $ext_if1 proto tcp from any to $ext_if1 port ssh -> localhost
#rdr on $ext_if2 proto tcp from any to $ext_if2 port ssh -> localhost

#  default deny
#block in  from any to any
#block out from any to any

#  general "pass out" rules for external interfaces
pass out on $ext_if1 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if1 proto { udp, icmp } from any to any keep state
pass out on $ext_if2 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if2 proto { udp, icmp } from any to any keep state

pass in on $ext_if1 proto tcp from any to 10.0.1.10
pass in on $ext_if2 proto tcp from any to 10.0.2.10

#  route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
#  $ext_if2 and $ext_gw2
pass out log on $ext_if1 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any
pass out log on $ext_if2 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any
pass out log on $ext_if1 reply-to ($ext_if1 $ext_gw1) from $ext_if1 to any 
pass out log on $ext_if2 reply-to ($ext_if2 $ext_gw2) from $ext_if2 to any
#
 
The OpenBSD PF FAQ should have more details as well, though you should keep in mind that FreeBSD's PF is quite a long way behind, so some recent syntax additions may not fly on FreeBSD. I don't think that goes for the route/reply stuff though, which has been well-established for years. Also search these forums for setfib, I seem to remember some users running network views on e.g. services and jails.
 
You must log in or register to reply here.
Back
Top