Basic question about Jail <-> Jail Networking

[SKull]

Howdy folks

I have a small and very basic question about Jail Networking in FreeBSD.

TLDR; What's the approach to make Jails communicate with each other.

Here's the situation:

I have the following two jails:

+-----+------+-------+--------------+------------+
| JID | NAME | STATE |   RELEASE    |    IP4     |
+=====+======+=======+==============+============+
| 6   | db   | up    | 12.1-RELEASE | 172.16.2.1 |
+-----+------+-------+--------------+------------+
| 1   | web  | up    | 12.1-RELEASE | 172.16.1.1 |
+-----+------+-------+--------------+------------+

On 'web', i have nginx is running. Nginx serves my personal homepage as well as an instance of nextcloud.

Both jails get their ips via an alias of lo1.
Ifconfig in case it matters:

lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 172.16.1.1 netmask 0xffffff00
        inet 172.16.2.1 netmask 0xffffff00
        groups: lo
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

The web jail is accessed via a port forwarding rule in my pf.conf:

ext_if="vtnet0"
ext_addr=$ext_if:0
int_if="lo1"
jail_net=$int_if:network
web_addr="172.16.1.1"
web_ports="{ 80, 443 }"

scrub all
# Define the NAT for the web jail
nat on $ext_if from $jail_net to any -> $ext_addr port 1024:65535 static-port

# redirect http traffic to web jail
rdr pass on $ext_if inet proto tcp to port $web_ports -> $web_addr

# Set the default: block everything
block all

# blacklistd
anchor "blacklistd/*" in on $ext_if
block in
pass out

# Allow the jail traffic to be translated
pass from { lo0, $jail_net } to any keep state

# Allow SSH in to the host
pass in inet proto tcp to $ext_if port ssh

# Allow outbound traffic
pass out all keep state

This all works fine and dandy.
But the nextcloud instance has an SQLite backend (the default), which of course isn't optimal for larger file collections.
So I made the second jail 'db'. It has an instance of postgresql running, ready to be used as my nextcloud backend.

Now here's the question: What's the best way of making 'web' talk to 'db'?
I guess I COULD take the same 'port forwarding' approach to make 'db' accessible from the outside, but I feel like that would be an uneccessary round trip across the internet.

I'm not expecting you guys to tell me all the necessary commands...
I can (and should have to) look them up myself, since 'aguy in the forum told me' is not the way to learn things IMHO.
But what IS the correct approach? Do I need to read up on VLANs and make my jails use VLAN? Do I need to make the roundtrip accross the internet? Is the solution even easier than I thought and I'm just not seeing it?

I did google around, and I found multiple blog posts and tutorials that MAYBE would be a solution, but again: I don't want to parrot other people's stuff, without knowing what I'm ACTUALLY doing

Thanks in advance!

[edit]I even have mwl's book on Jails, but it skips exactly this because it assumes that people now their tcp/ip stuff in detail already. I do have a CCNA, but I made it 8 years ago and since then worked as a developer. So my networking skills aren't exactly top notch anymore [/edit]

 

[Savagedlight]

Your jails are on different subnets (172.16.2/24 vs 172.16.1/24) and the jail host will need to know how to route between these, and pass traffic between them.
If either jail only has a foot in one of the subnets, pay close attention to the output of "traceroute 172.16.2.1" from the web jail, and likewise towards .1.1 from the db jail.

I suggest putting them both in the same subnet. If that's not possible, you might find some relevant information to you in my tutorial on dealing with jail hosts which are attached to multiple local networks (think VLANs or multiple NICs attached to different networks) by using fibs (multiple routing tables) here: https://savagedlight.me/2014/03/07/freebsd-jail-host-with-multiple-local-networks/

 

[PMc]

Well, as far as I understand, there is nothing needed to do: these jails will straightahead talk to each other via lo.
Without VIMAGE, there is only one routing entity (aka "IP stack") in the kernel. If one jail sends packets in there destined to the other jail, the kernel will see that we have the target address right on loX, and anything listening on that address will then catch the ball.
Thats why, with non-VIMAGE jails, one must take care that nothing on the base system would listen on that same address which is configured for a jail. (Within the jail you can listen only on that address, but in the base system one can listen on any address, including those of the jails - which would be not good, as the base system would then take traffic destined for the jail).

Thats the basic way it works. If you want to firewall between the jails, that's getting a bit more difficult, because all the traffic happens on loX. I have done this, with ipfw, and with the jail addrs defined on lo0. So I could only give you the ipfw scheme, but you're using pf. (From my investigations, I could not find significant advantage in putting them on lo1.)

Then, as @Savagedlight has noticed, You have put the jails into separate subnets. I never tried that, so I'm not certain if and how this will work. It might actually be a good idea, I just don't know. Correction: I have done this, with app+db carrying local addrs, and web a real static one - there is no problem, they should effortless talk to each other.

I'd suggest you first try and (disconnect uplink and) disable the firewall to try and understand how internal connections do basically work, and only after that configure the firewall appropriately above it.

I don't see a use-case for a VLAN here. What could be done is configure the jails as VIMAGE. In that case each jail would behave exactly like a separate host - each jail would run their own firewall, and you would need to (virtually) wire network connections between them to get any connectivity. Thats a lot more effort in configuration, but then it behaves like three individual machines which can be configured from the book.

I don't think it is even possible to have them do "a roundtrip across the internet", without using VIMAGE. With VIMAGE it is possible, and while it would not make sense to connect db with web in that fashion, there are use-cases where one wants to do that (to instantly see how the outside connection generally behaves).

 

[SKull]

Apparently I was thinking way too far ahead. I should've known that it would be simple

But thank you very much for the detailed explanations!

I managed to connect the jails as intended ?

Although I effed up my cloud data in the process, but that's a nextcoud issue, and I still have the data on several clients so it's not an issue

 

[zen0]

Hello All,

I've conceptual question about what do you think is the best way to use jails with web app + db... I'm wondering why do we need to separate WEB APP from DB with jails at all ? Suppose we have web application that is compromised. In order to talk to the DB, WEB APP will need credentials... so if web app is compromised the attacker should have access to the db, so I don't see reason to separate db from web application ? or I just misunderstand something ?


Best Regards!

 

[unitrunker]

if web app is compromised the attacker should have access to the db, so I don't see reason to separate db from web application ?

You can limit the webapp to only call specific stored procs in the DB. The app would not have full DBA level access.

 

[SKull]

Hello All,

I've conceptual question about what do you think is the best way to use jails with web app + db... I'm wondering why do we need to separate WEB APP from DB with jails at all ? Suppose we have web application that is compromised. In order to talk to the DB, WEB APP will need credentials... so if web app is compromised the attacker should have access to the db, so I don't see reason to separate db from web application ? or I just misunderstand something ?


Best Regards!

Most web applications have a parameter for 'dbuser', and that one gets full access to the application db, but no other access. So the web app's db would definitely be compromised, but nothing else.

I'm not sure if that would be enough for corporate security best practices. But it's good enough for my setup I guess

 

[PMc]

Most web applications have a parameter for 'dbuser', and that one gets full access to the application db, but no other access. So the web app's db would definitely be compromised, but nothing else.

It should not even get that. In postgres, for instance, we have the SQL grants for SELECT,INSERT,DELETE,UPDATE. These are needed to work with a database, and this is all the webuser gets.
It is an entirely different thing to add table rows, add indices, add foreign keys, etc.etc. These are part of the database design and are not changed in everyday operations. The webuser does not need to do these.

So, immediately after db creation, before creating any objects, we do just this:

SET ROLE deploy;
ALTER DEFAULT PRIVILEGES grant SELECT,INSERT,DELETE,UPDATE ON TABLES to webuser;
ALTER DEFAULT PRIVILEGES grant SELECT,UPDATE ON SEQUENCES to webuser;

deploy will succesively become the creator and owner of all the db objects, and webuser will automatically get all the necessary permissions, but nothing else.
(This obviousely implies that you have a working deploy scheme.)

Then, when you have such clear separation of duties, you can go on that path: certain internal tables can be revoked the webuser access entirely. Or, you can work with VIEWs, where e.g. the webuser is only allowed to insert data (but not read it - think credit cards), or can access only via special procedures. And so on...

I'm not sure if that would be enough for corporate security best practices.

Corporate security is mainly about fake paperwork and bogus formalism; thinking is not welcome and skill is to be avoided. Thats how shops like mastercard manage to have their customer db sold in the darknet, and why I don't even get a job. :/

 

[SKull]

and why I don't even get a job. :/

I'm so glad that my communication skills seem good. I speak fluent 'management'.
Plus, I always appear confident, despite the fact that I'm an insecure mess

Anyway, thanks for the detailed explanations!