NGINX on several jails or on host?

[jjbigorra]

Hey guys, this is my first post here, I am hoping I respect all the rules of this wonderful forum.

I am setting up some services, moving from Ubuntu to FreeBSD in my company.

We have 3 environments:
- Test: all services in one server
- Acceptance: Database and Redis in one server, rest of services in another server.
- Production: same as acceptance, but with a database cluster.

I am starting with test environment transition to FreeBSD obviously, and wondering about NGINX redirecting to several jails.

The plan is to have several jails, of which the following do already work with pf firewall and IP redirection via loopback interface:
- Go application builder
- NPM website builder
- Go apps as daemons (each app its own jail)
- MariaDB jail
- Redis jail

I also need to setup a jail with that has websites available via port 80 and 443, and another jail with a PHP application ( using PHP FPM).

As it may seem obvious i need to use NGINX for both jails above mentioned.

Would it be best to :
- install NGINX on both jails and simply redirect traffic via pf to each jail’s separate NGINX
- install NGINX on the host, and work out a way to redirect traffic to the jails via NGINX
- install both websites and PHP application with fpm on one jail with its own NGINX

Thanks in advance, looking forward to learn a bit more about best practices in this area.

 

[SirDice]

- install NGINX on both jails and simply redirect traffic via pf to each jail’s separate NGINX

You can redirect port 80 (any port actually) only once. It's not possible to use PF to forward the same port 80 traffic to different backends. It's going to end up doing a round-robin type forward, that's usually not what you want to happen in this case.

install NGINX on the host, and work out a way to redirect traffic to the jails via NGINX

I'd use www/haproxy on the host. It can forward based on the HTTP1.1 Host: header to one of the jails. You can set this up with nginx too of course but with Haproxy you have much more control. Added bonus, you can see exactly which site gets what amount of traffic. Other bonuses, you can do SSL offloading on HAProxy, so there's only one place where your SSL certificates are stored.

 

[goshanecr]

I'm use 1 jail with nginx which acts as reverse proxy for other sites in other jails with lighttpd, apache , etc.

 

[jjbigorra]

You can redirect port 80 (any port actually) only once. It's not possible to use PF to forward the same port 80 traffic to different backends. It's going to end up doing a round-robin type forward, that's usually not what you want to happen in this case.


I'd use www/haproxy on the host. It can forward based on the HTTP1.1 Host: header to one of the jails. You can set this up with nginx too of course but with Haproxy you have much more control. Added bonus, you can see exactly which site gets what amount of traffic. Other bonuses, you can do SSL offloading on HAProxy, so there's only one place where your SSL certificates are stored.

Thanks you for your advice first of all. I will indeed follow your tip on using HAProxy. That will mean then using two instances of NGINX, one per jail right?
I love the idea of HAProxy, and keeping the certs in one place.

 

[SirDice]

That will mean then using two instances of NGINX, one per jail right?

Yes, each jail can run its own website (server and all). Then you can use HAProxy on the host to have it switch to the correct backend based on the Host: header. That way you can easily run site1.example.com and site2.example.com on the same machine while still keeping them completely separated. You could also use nginx in one jail and Apache in another, or whatever you fancy using.

 

[jjbigorra]

Yes, each jail can run its own website (server and all). Then you can use HAProxy on the host to have it switch to the correct backend based on the Host: header. That way you can easily run site1.example.com and site2.example.com on the same machine while still keeping them completely separated. You could also use nginx in one jail and Apache in another, or whatever you fancy using.

That is perfect, thanks for your invaluable help!

 

[tommiie]

I'd use www/haproxy on the host.

Is there a particaul reason to run www/haproxy on the host and not in a third jail? I would run all my services in jails and try to keep everything off of the host except for jail management.

 

[SirDice]

Is there a particaul reason to run www/haproxy on the host and not in a third jail?

Easiest to set up and doesn't require any firewall forwarding tricks to access. The host itself is just an obvious choice because it already sits inbetween the outside world and the jails. Besides that, you can view haproxy as an application firewall specifically for web applications (you can do lots of filtering with haproxy too). But sure, you could run haproxy in its own jail.