Super noob questions about default router settings...

  • Thread starter Thread starter nx
  • Start date Start date
N

nx

What should the defaultrouter in rc.conf be for the reverse proxy and webserver in:

modem --- (WAN) pfsense (LAN) (OPT1) --- reverse proxy --- webserver

I'm trying to set up pfsense firewall as a transparent bridge by assigning bridged WAN and LAN to OPT1 (LAN IP address type is set to none).
I access pfsense's web admin via OPT1, and I assume bridged WAN traffic would then pass through to OPT1 and machines behind it.

Should the:
- reverse proxy have defaulrouter set to the LAN of pfsense or the modem?
- webserver have defaultrouter set to the single nic of the reverse proxy?
- reverse proxy have 2 nics - one for traffic from pfsense and one out to the webserver?

Please help me clear up these basic questions, thanks
 
The reverse proxy should have the default router set to the modem's LAN IP address. When bridging interfaces you treat the bridge as dumb switch. The webserver should then have the reverse proxy as its default gateway.

Your set up looks unnecessarily complicated to me. Could you turn the modem into a bridge instead of bridging the pfSense interfaces? That way you could setup pfSense with the standard NAT (or do you have a routed public subnet?) and possibly integrate the reverse proxy into the same pfSense machine.
 
Thanks kpa - you've mostly cleared that up for me!

I'm a little confused because the howto I followed for making a transparent firewall/filtered bridge with pfsense says to change the LAN IP address type to none in pfsense's web admin.
However, it still has a vmnic in the vm settings that is a host-only adapter with a static IP address.
The pfsense console of course shows the LAN as having no IP address after making the change it the web admin.

I wish I could do as you've suggested - put modem into bridge mode and use NAT on pfsense, but alas my setup is rather complicated!
pfsense and all servers behind it are virtualbox vms hosted in a mac, and it's not even possible for me to put pfsense between the modem and the mac.

I'm actually failing to get the filtered bridge working, that is, resolving the public IP/domain of the webserver.
If I have the modem forward port 80 to the reverse proxy without pfsense on, I resolve the domain externally and locally.
But if I have pfsense on, http://www.mydomain.com is being redirected to https://www.mydomain.com and my reverse proxy rule to redirect http://mydomain.com to http://www.mydomain.com isn't working, but instead redirecting to https://mydomain.com

Here is my post on the pfsense forum if you have the time to quickly scan it:
http://forum.pfsense.org/index.php/topic,53072.0.html

Skip to the second to last post (my last post) outlining how I've tested the filtered bridge, and you'll see a simple list of the behaviour happening.
Hopefully something obvious I've done to break the bridge may pop out to you!

Here is the post with my original network layout showing vms:
http://forum.pfsense.org/index.php/topic,52870.0.html

I've been stuck on this for days and really appreciate your help!

PS Originally I wanted to use NAT and not a filtered bridge, but I couldn't work out how to assign static IPs of the vms using their vm network adapters to a different subnet.
The virtualbox forum recommneds I use an internal network adapter, which I would like to use due to its isolation from the host, but no one there has answered my simple question - how to actually do this?
I suggested that 'Do I simply change the host-only adapter static IPs of each vm to be in the new subnet and add to each the same named internal network adapter?' but no one as replied - they have just given me generic answers.

And even if I did put the reverse proxy and backend servers on a different subnet with static IPs using the internal network adapter type, eg 192.168.2.0/24, I would then have the following noob issues
- how to ssh/sftp to them (via pfsense?), which I can do easily at this time because the host and vms are in the same 192.168.1.0/24 subnet.
 
About the internal network in VirtualBox. You just select "internal network" in the network adapter preferences of the VM and give it a name (defaults to "intnet"). The internal network is basically a virtual ethernet segment that is only visible to the virtual machines if you configure the network adapter(s) for the VMs to use it (and use the matching name if there are multiple internal networks) instead of NAT or bridging.

I would set the WAN interface in the pfSense VM to bridged (in the network adapter setting of the VM in virtualbox) with the real network interface of the mac. Then I would set the LAN interface of the pfSense VM to the internal network. The pfSense should be configured to use NAT with a private range static address on its LAN interface. Also the reverse proxy would be in the pfSense machine in this set up. Now the webserver VM would be on the same internal network as the pfSense VM with a statically assigned IP address on its network interface (same subnet as the pfSense LAN IP).

To access the webserver with ssh(1) you'd have to add port forwards on pfSense but that should be easy enough.


I hope this makes sense and is helpful to you.

Edit: host only network -> internal network.
 
Thanks kpa,

I've updated vmnics and IPs as per your advice:

(WAN 192.168.1.2) pfsense (LAN 192.168.2.0) --- intnet1 --- (192.168.2.1) reverse proxy --- intnet2 --- (192.168.2.2) webserver

pfsense has 3 vmnics - WAN (bridged - same as host), LAN intnet1, and OPT1 (host-only) for local web admin access.
The reverse proxy has 2 vmnics - both internal networks, one is intnet1 so it can talk with pfsense, and intnet2 so it can talk with the webserver, which has the same vmnic.


A few problems I'm stuck on:

- After changing the IP address of the reverse proxy vm from being in subnet 192.168.1.0/24 to being in 192.168.2.0/24, after booting it gives ongoing 'em0: watchdog timeout --resetting' errors in its console.
I've changed all instances of its previous IP address in the rc.conf to hosts to sshd_config file.
And to try to fix it, I've changed the mac address in the vm settings.
Nothing is fixing this error.

- I can't get ssh forwarding to work.
ssh -v just shows that 'operation timed out'.

In pfsense NAT rules I created:

If Proto Src. addr Src. ports Dest. addr Dest. ports NAT IP NAT Ports
WAN TCP/UDP * * * 22 reverseproxyalias 22

and this auto-generated a similar firewall rule for the WAN.

I assume I'm ssh-ing to the WAN of pfsense to be nat port forwarded to the reverse proxy?


Once I've fixed these I should be able to get everything working with pfsense today.

Any ideas?
 
You can't use the address ending in .0 if the subnet is a /24. Use 192.168.2.1 as the pfSense LAN IP.

You have two different virtual intnets, you can't use the same addresses on both. Change the addresses on intnet2 to for example 192.168.3.*.

You'll also have to tell pfSense how to reach the 192.168.3.* subnet. Add a static route on LAN interface with network 192.168.3.0/24 and gateway 192.168.2.x, x being the address you chose for the reverse proxy's NIC in intnet1. The reverse proxy machine should also be set to do routing, setting gateway_enable="YES" in /etc/rc.conf is enough.

This is still quite complicated, which reverse proxy you're using? Changes are that it's available on pfSense as an add-on package.
 
I think you could use bridging in pfSense but with one additional NIC.

Setup LAN as a subnet that does not conflict with the other addresses, for example 10.m.n.1/24 where m and n are numbers of your choise. The LAN net would used only for configuring pfSense. I think you could use a "host only network" in the pfSense VM network adapter settings for this, the LAN net would be only visible on the mac.

Add an OPT1 interface an internal network that you bridge with the pfSense WAN interface. Place the reverse proxy in the OPT1 network using the same addresses as the pfSense WAN network uses (192.168.1.* addresses). The reverse proxy would have to perform routing between the 192.168.1.* network and the subnet behind the reverse proxy (for example the 192.168.3.0/24 I used above) and the modem would have to told how to reach that subnet, again using a static route to 192.168.3.0/24 with gateway set to the external address of the reverse proxy.
 
Thanks kpa,

The NAT approach still looks the easiest... as I can't do much with the residential modem I'm using other than some basic telnet (local) stuff to tighten its security (eg turn of WAN telnet and all its insecure default settings). It's web admin is very restrictive in tools/settings, and I have to be careful not to break it too often as there are other users behind it.

I've made all the changes you've suggested. Now the reverse proxy vm no longer is giving the em0 watchdog errors, but after changing the webserver to be 192.168.3.x, it's giving the same error, so perhaps something still isn't right in its settings?

One thing I'm confused about is - although I have a NAT rule that forwards port 80 to the reverse proxy and an auto-generated WAN rule doing similar - do I need any related LAN rules or is this what the static route is for?

The reverse proxy and web server are nginx, and I also have db servers, apache email and dev servers (tho not in the layout diagram) - all running in vms that have had uptime of months unless I have had to reboot the mac.
Everything works fine - I can access the web/email/dev servers locally/externally - use the reverse proxy to redirect content based on country, setup loadbalancing easily for when I have more RAM to add more server vms, use private/public ssl key/certs to access squirrelmail and dev servers, etc. It's pretty easy, flexible, and has some powerful caching abilities, which is why I prefer to have the reverse proxy separate from pfsense. And when I have a better network environment, I'd like to have the same separation of firewall from reverse proxy, so it seems easier to keep them separate from the start. But I understand how your suggestion of coupling them could make things easier now. However it would probably only cause me to have to learn a new app/config/etc, which I don't have time for.

I have my first tiny app that is a pretty unique and cool service (and legit) ready to launch once I sort out pfsense, which should be getting at least 1k hits per day by the end of the first week and in the 10-15k+/day by 4ish weeks once 'word is out'. The plan is to move to a VPS and contract to a network admin after 2 months of revenue and/or before my setup fails to scale.

I want to launch on the 17th of this month, and get pfsense sorted early this week sometime, so I can spend the rest of the week doing last security tightening like rebuilding/testing the pf rules on each vm, and then have the last week to do some last refinement to the app.

If I can't get pfsense to work, I'm either going to run everything off a bootable image from usb, with backup usb image, or remove all personal stuff from my mac account and run off hard disk as I am now. If I had more RAM I'd try to run the servers from a different user account, which would give me some sandboxing and allow me to switch to my normal user account and continue dev in it.

I'll give this one last try based on your next reply - if you find time. I'm really grateful for your help! Launching in my current environment may seem crazy - but not launching would be truly insane.
 
Go with the NAT set up if it looks easier to you. I forgot one thing, the outbound NAT for the 192.168.3.x network. I'm not sure if the latest pfSense adds a suitable NAT rule automatically if you have a static route. The rule should NAT all traffic from 192.168.3.0/24 net to the pfSense WAN address. You'll have to ask this on pfSense forums.

I have no idea what could cause the watchdog errors, maybe try a different type of Intel NIC in the VM settings?
 
  • Thanks
Reactions: nx
Thanks again mate,

I've now given up on the NAT approach (LOL) as no matter which vmnic I choose I can't get rid of the em0 watchdog error after putting the webserver on a different subnet. I have no idea why the same vmnic adapter type is happy to be on a different subnet in pfsense, but it won't work for the webserver vmnic.

Running pfsense as a vm in this environment just seems too hard for me to get everything working together and really drains my energy to polish the app and get it launched. I've scraped together enough hardware to build a separate pfsense box and will put it in front of the mac in filtered bridge mode. This should be easier to setup.

I'll be back with any noob questions, but hopefully I can use all the good info you've provided thus far to get it working well.

Thanks again!
 
You must log in or register to reply here.
Back
Top