PF Cannot limit cross-talk between jails on the same host (lo?)

M

Mikalai

I cannot limit cross-talking between jails.

Host (serv) is sitting on a network, and is supposed to serve three ips. 2222 is opened for ssh on serv's ip. Everything else is blocked by "block", except for common "skip on lo0".

Two jails (t1 and t2) are assigned ips on em0 by ezjail.
t2 has a web server.
t1 is able to connect to this server! Why?

pf.conf is this:

Code: 
set skip on lo0

ext_if = "em0"

serv = "192.168.0.10"
t1 = "192.168.0.11"
t2 = "192.168.0.12"

# removing 192.168.0.0/16 as we face such network
table <martians> const { 127.0.0.0/8, 172.16.0.0/12, \
                        10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
                        0.0.0.0/8, 240.0.0.0/4 }

block

block drop in quick on $ext_if from <martians> to any
block drop out quick on $ext_if from any to <martians>

# Allow ssh connections to serv
pass in inet proto tcp from any to $serv port { 2222 }

antispoof for $ext_if

rc.conf is this:

Code: 
hostname="serv"

defaultrouter="192.168.0.1"

ifconfig_em0="inet 192.168.0.10 netmask 255.255.255.0"

gateway_enable="YES"

pf_enable="YES"

sshd_enable="YES"

Output from ifconfig looks like this:

Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
   ether 00:00:00:00:00:00
   inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
   inet 192.168.0.11 netmask 0xffffffff broadcast 192.168.0.11
   inet 192.168.0.12 netmask 0xffffffff broadcast 192.168.0.12
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect (100baseTX <full-duplex>)
   status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
   inet6 ::1 prefixlen 128
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
   inet 127.0.0.1 netmask 0xff000000
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   groups: lo

Output from "netstat -r" is following:

Code: 
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.1        UGS         em0
localhost          link#3             UH          lo0
192.168.0.0/24     link#1             U           em0
192.168.0.10       link#1             UHS         lo0
192.168.0.11       link#1             UHS         lo0
192.168.0.11/32    link#1             U           em0
192.168.0.12       link#1             UHS         lo0
192.168.0.12/32    link#1             U           em0

Do I interpret netstat correctly, if I'll say that skipping of a block happens due to jails ips magically going through lo0, on which there is a skip?

The second question is, how to set two jails on the host so that, one may talk to another over one port, with everything else blocked?

It seems to me that most common examples are such that pf cannot block unauthorized crosstalk between jails. Please, show me example otherwise.
Put on your adversarial mood, imagine that you busted process in a jail, cannot break out of the jail, yet, try to connect to neighboring jail. If you can connect, it is the issue I am talking about, if you cannot connect, can you share your network, pf and jail settings.

I tried to put jails on lo1, but had to add skip on it as well, which allows authorized cross-talk. Experiments with giving jails ips on tap's, also added ip to lo0 relationship in netstat.
What am I missing here?

Thank you, and ... he-e-elp!

PS all of this has been tested on FreeBSD 11.0.

[UPDATE] in this thread below we have one solution that involves keeping every single jail on its own loopback, having filtering on loopbacks and skip on host's lo0. It is possible to filter inter-jails communication in this setting. The caveate is that jail cannot have external ip attached to it. So, to serve traffic from outside, port(s) need to be pf-ed inside.

Hope, this will help someone.

Do leave comments, suggestions, questions.
 
Last edited:
Mikalai said:
t1 is able to connect to this server! Why?
Click to expand...
Traffic never actually passes the interface so the firewall is completely oblivious to it. The firewall can only do its work when traffic passes through an interface.
 
How then jail networking+filtering can be arranged, so that it passes only via filtered routes?
And if the only option is to filter on lo0, how should it be done?
 
Mikalai said:
How then jail networking+filtering can be arranged, so that it passes only via filtered routes?
Click to expand...
Probably the only way to do this properly is by using VIMAGE/VNET jails instead of the 'standard' jails.
 
VIMAGE: is it only network stack, or does it bring virtualization for non-networking, i.e. heavy things which jails suppose to bypass?
 
Non-standard jail, iocage, from use in the wild: https://ramsdenj.com/2017/06/05/nextcloud-in-a-jail-on-freebsd.html
Note date June 2017, with the quote from it:
Shared ip
VIMAGE allows a virtualized NIC to be used inside of a jail, but it isn’t quite stable. Since having a virtualized interface isn’t a necessity with Nextcloud, I have turned VIMAGE off and used a shared IP instead.
Click to expand...

So, if that thing has several jails, their traffic will go through lo0, as it seems. For an attacker, it is a nice step two, for lateral movement, of cause :cool:
 
What rules should be on lo0, so that jail j1 can be served by j2, and j3 would have no connections to other jails.
On lo0, it seems that from $this_ip to $this_ip is the only way to go. But connection originating from j3 will look like connection from j2, when it goes to j2.
How can this be tamed?
 
VIMAGE: it seems to give a net stack inside of a jail. But I don't need a network stack in a jail. I want host to respect the fact of some local traffic being a little less trustworthy. Is there special network setting for this?
 
SirDice said:
Traffic never actually passes the interface so the firewall is completely oblivious to it. The firewall can only do its work when traffic passes through an interface.
Click to expand...

Do you know, can I make sockets that pass from one jails to another? This way pf can be set to shut any communication between jails via network, yet, communication will occur. Web servers and Postgres can serve on unix sockets.

And if socket can talk to jail. Can pf, or may be something else (simple and reliable), pipe networked port to a socket?
 
SirDice said:
Traffic never actually passes the interface so the firewall is completely oblivious to it. The firewall can only do its work when traffic passes through an interface.
Click to expand...
Is it possible to add and enforce some inefficient route via filtered spot? One host's ip -> filetered spot -> another host's ip.
 
Here is one possible solution. Put each jail on its own cloned lo. With 20 jails it is lo1...lo20. With two, output from netstat -r is this
Code: 
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.1        UGS         em0
localhost          link#3             UH          lo0
192.168.0.0/24     link#1             U           em0
192.168.0.10       link#1             UHS         lo0
192.168.1.1        link#5             UH          lo1
192.168.1.2        link#6             UH          lo2

Note that addresses for loX are not associated with lo0. Now, we should add some filtering.

Code: 
set skip on lo0

ext_if = "em0"

serv = "192.168.0.10"
t1 = "192.168.1.1"
t2 = "192.168.1.2"

# removing 192.168.0.0/16 as we face such network
table <martians> const { 127.0.0.0/8, 172.16.0.0/12, \
                        10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
                        0.0.0.0/8, 240.0.0.0/4 }

block

#pass quick on lo0 from 127.0.0.1 to 127.0.0.1
#pass in on lo0 from $t1 to $t2
pass quick on lo2 from $t2 to $t2
pass inet proto tcp from $t1 to $t2 port { 8080 }

block drop in quick on $ext_if from <martians> to any
block drop out quick on $ext_if from any to <martians>

# Allow ssh connections to serv
pass in inet proto tcp from any to $serv port { 2222 }

antispoof for $ext_if

In this setting, jail t2 has a web server that runs on 8080 and 8081. Host can see both. Jail t1 can see only 8080, and 8081 is not accessible. And jail t2 can talk to itself.
Tada! We have filtering of inter-jail communication without VIMAGE, and using new option of adding jails' loopback.

Of cause, if jail has to service traffic that comes on external ip, we MUST NOT give it this external ip, cause it will add route with host's lo0 to that address, and there will be a filtering hole. We only have an option to pass this traffic to jail with pf rules alone.
 
SirDice said:
Traffic never actually passes the interface so the firewall is completely oblivious to it. The firewall can only do its work when traffic passes through an interface.
Click to expand...

This is not true in my experience. All traffic between jails is sent on a lo interface (typically lo0).

All you have to do if you want to filter traffic between jails is to simply remove:
Code: 
set skip on lo0
And then write the appropriate filters to allow the traffic you do want to permit.
 
`Orum said:
This is not true in my experience. All traffic between jails is sent on a lo interface (typically lo0).

All you have to do if you want to filter traffic between jails is to simply remove:
Code: 
set skip on lo0
And then write the appropriate filters to allow the traffic you do want to permit.
Click to expand...
Can you show lines for the initial example with the following: t2 has services on 8080 and 8081; t1 should only be able to access 8080, but not 8081. Please.

And, when it works, do jails sit on external ips (on em0), like in the example, or do they sit exclusively on lo0?
 
`Orum said:
And then write the appropriate filters to allow the traffic you do want to permit.
Click to expand...
When I ssh from host (h_ip) into jail (j_ip), jail will later tell me that I was logged from j_ip. And with this, filtering on lo0 becomes tricky. So, please, show an example.
In fact, originally I also wanted to restrict traffic flow from host to jail, but this seems totally unattainable.
 
Mikalai said:
Can you show lines for the initial example with the following: t2 has services on 8080 and 8081; t1 should only be able to access 8080, but not 8081. Please.

And, when it works, do jails sit on external ips (on em0), like in the example, or do they sit exclusively on lo0?
Click to expand...
Assuming:
  • You have a default deny ruletset (you should)
  • You're using all quick rules (again, something that's a good idea)
  • The proto you want to pass is tcp
  • You have some name resolution for the jails (e.g. /etc/hosts or DNS)
Code: 
pass quick on lo0 inet proto tcp from t1 to t2 port 8080 keep state
block quick all # I recommend you put this at the end of your ruleset
Also, what do you mean "sit on"? You can assign a jail whatever IPs you want, whether they're on an external (i.e. non-loopback) or loopback interface. As far as FreeBSD is concerned, all traffic between them on a single host will go over a loopback, regardless of what interface the IPs are assigned to. If you have multiple loopback interfaces, I'm not sure which it will assign the traffic to, as I've never configured a system with more than a single loopback interface.

Mikalai said:
When I ssh from host (h_ip) into jail (j_ip), jail will later tell me that I was logged from j_ip. And with this, filtering on lo0 becomes tricky. So, please, show an example.
In fact, originally I also wanted to restrict traffic flow from host to jail, but this seems totally unattainable.
Click to expand...
You can restrict traffic from the host to the jail, but it's not very secure. The main issue with restricting traffic from the host system to the jail is that programs on the host system can bind to pretty much any address they want to (the only restrictions are on ports < 1024, which can only be bound by root), so it's hard to write a rule that can exclude traffic from the host system but still allow traffic from other legitimate hosts. By putting the other host in a jail, you restrict the source IP address(es), and you have the added bonus of restricting damage if they were able to gain root privileges in the jail (otherwise they could just modify your pf ruleset unless you were at a high securelevel).
 
You must log in or register to reply here.
Back
Top