Solved How to avoid to restart pf after starting a jail ?

ssbear

ssbear

Hi.

I host a webserver under a jail, on a FreeBSD 14.0 machine. The jail is started on boot. Here my /etc/rc.conf:
Code: 
hostname="machine.domain.tld"
ifconfig_em0="DHCP"
ifconfig_em0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"

zfs_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

nginx_enable="YES"
ntpd_enable="YES"
sendmail_enable="NONE"
smtpd_enable="YES"

jail_enable="YES"
jail_parallel_start="YES"
jail_list="my_jail"

Here's my /etc/jails.conf:
Code: 
my_jail {
  # STARTUP/LOGGING
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
  allow.raw_sockets;
  exec.clean;
  mount.devfs;

  # HOSTNAME/PATH
  host.hostname = "${name}";
  path = "/jails/containers/${name}";

  # NETWORK
  #ip4 = inherit;
  ip4.addr = "192.168.1.201";
  interface = em0;

  # MISC
  jid = 1;
  persist;
}

The jail starts on boot, the server behind the jail responds. At this point, it's ok. However, inside this jail, there is no Internet access... until I restart pf on the host machine. Example, after a
jexec my_jail, a simple ping google.fr returns a Unknown host. After a service pf restart, the jail can go to Internet (the ping is OK).

And... I got the same issue with opensmtpd inside the jail: I have to do a service smtpd restart on the host. After, the jail can send emails.

So, on this machine, after a boot, I always have to restart pf and smtpd, if I want Internet access for my jail.
Any idea? Did I miss something?


Thanks a lot for your advices.


Regards,
 
I tried a different approach: I don't start my_jail at boot, but manually, after.
And, same issue: no Internet connection. And same fix: restart pf.

So, I don't know if it's a pf issue, or a jail issue (and if it's the case, I'm probably in the wrong section).
 
Erichans said:
When ping google.fr is not successful, can you, or can't you ping anything outside the jail?
Click to expand...
I can ping the host machine (192.168.1.10) from the jail (192.168.1.201), but it seems I can't access Internet. More, emails I try to send from the jail are rejected with an invalid recipient.
If, on the host, I restart pf, ping is OK. And if, on the host, I restart smtpd, mails are correctly sent.

SirDice said:
Show us your pf.conf.
Click to expand...
Here's the file:
Code: 
#       $FreeBSD: src/etc/pf.conf,v 1.2.2.1 2006/04/04 20:31:20 mlaier Exp $
#       $OpenBSD: pf.conf,v 1.21 2003/09/02 20:38:44 david Exp $
#
# See pf.conf(5) and /usr/share/examples/pf for syntax and examples.
# Required order: options, normalization, queueing, translation, filtering.
# Macros and tables may be defined and used anywhere.
# Note that translation rules are first match while filter rules are last match.

int_if = "lo0"
ext_if = "em0"

icmp_types = "{ echoreq unreach }"

table <ips-ssh-flood> persist
table <ips-ssh-illegal> persist
table <ips-ssh-invalid> persist
table <ips-ssh-reverse> persist
table <ips-work> persist file "/etc/pf.ips-work"
table <ips-banned> persist file "/etc/pf.ips-banned"
table <ips-jails> persist file "/etc/pf.ips-jails"
table <ips-good> persist file "/etc/pf.ips-good"

set block-policy drop
set fingerprints "/etc/pf.os"
set skip on lo0

block in all
block out all

pass in quick from { <ips-work>, <ips-jails> } to any
pass in quick on $ext_if proto tcp from <ips-good> to $ext_if port 22
pass out from $ext_if to any

block drop in quick log on $ext_if proto { tcp, udp, icmp } from <ips-banned> to any
block drop in quick log on $ext_if proto { tcp, udp, icmp } from <ips-ssh-flood> to any
block drop in quick log on $ext_if proto { tcp, udp, icmp } from <ips-ssh-illegal> to any
block drop in quick log on $ext_if proto { tcp, udp, icmp } from <ips-ssh-invalid> to any
block drop in quick log on $ext_if proto { tcp, udp, icmp } from <ips-ssh-reverse> to any

pass inet proto icmp icmp-type $icmp_types
pass in on $ext_if proto tcp from any to $ext_if port 22 \
        flags S/SA keep state ( max-src-conn 5, max-src-conn-rate 5/10, \
        overload <ips-ssh-flood> flush global)
pass in on $ext_if proto tcp from any to $ext_if port 80
pass in on $ext_if proto tcp from any to $ext_if port 443
antispoof quick log for { lo0, $ext_if }

And my /etc/pf.ips-jails:
Code: 
192.168.1.201
 
Hi,
I am not an expert, but to me your last rule is misplaced:
antispoof quick log for { lo0, $ext_if }
It should come after the block out all
You don't need to included lo0 in that rule, set skip on lo0 does the same thing if i am not mistaken.

I am not sure if it's important but I would put the following lines:

pass in quick from { <ips-work>, <ips-jails> } to any
pass in quick on $ext_if proto tcp from <ips-good> to $ext_if port 22
pass out from $ext_if to any

below the block table filter:
block drop in quick log on $ext_if proto { tcp, udp, icmp } from <ips-banned> to any
...
block drop in quick log on $ext_if proto { tcp, udp, icmp } from <ips-ssh-reverse> to any

it makes more sense to me, but may be I wrong.

You should better wait for another opinion though ;)
 
ssbear said:
Code: 
pass in quick on $ext_if proto tcp from <ips-good> to $ext_if port 22
Click to expand...
The actual address(es) of the em0 interface can change dynamically (DHCP and starting/stopping the jail). Put parenthesis around the interface:
Code: 
pass in quick on $ext_if proto tcp from <ips-good> to ($ext_if) port 22

Code: 
           Host name resolution and interface to address translation are done
           at ruleset load-time.  When the address of an interface (or host
           name) changes (under DHCP or PPP, for instance), the ruleset must
           be reloaded for the change to be reflected in the kernel.
           Surrounding the interface name (and optional modifiers) in
           parentheses changes this behaviour.  When the interface name is
           surrounded by parentheses, the rule is automatically updated
           whenever the interface changes its address.  The ruleset does not
           need to be reloaded.  This is especially useful with nat.
pf.conf(5)
 
Thanks a lot regarding your advices with my /etc/pf.conf (and I assume it has to be improved ?), but the problem is still there after.
That's why I don't understand my issue: if it was related to PF, then, the issue would still be here after a restart of PF? A restart fix it, no need to stop PF. And same issue for my opensmtpd.
I'm not very confident with jails. It's strange, it seems to appear that my hail has to be start before PF and opensmtpd?
 
Yes, but there is a 'start' immediately after the 'stop'. So, initially, after the first boot, when PF and opensmtpd are started, my jail can't access Internet. After a stop and a new start of PF and opensmtpd, it's OK.

As I say, I'm not very confident with jails (and jails network).
 
ssbear said:
That's why I don't understand my issue: if it was related to PF, then, the issue would still be here after a restart of PF? A restart fix it, no need to stop PF. And same issue for my opensmtpd.
Click to expand...
If you want to make sure the problem is not coming from PF, simplify you pf.conf:
pass all
Then if your jail as well as opensmtpd work immediately you'll know that your PF config is the problem.
In this case put things back, in your config file pf.conf, step by step to identify what part is wrong.
 
gotnull said:
If you want to make sure the problem is not coming from PF, simplify you pf.conf:
pass all
Then if your jail as well as opensmtpd work immediately you'll know that your PF config is the problem.
In this case put things back, in your config file pf.conf, step by step to identify what part is wrong.
Click to expand...
Thanks, gotnull!

With a simple pass all in my /etc/pf.conf, my jail has access to Internet. More, I added a match from src "192.168.1.201" for any action "relay" in my /usr/local/etc/mail/smtpd.conf, and my jail can send email.

So, I'll investigate line by line my initial /etc/pf.conf. However, it's still a mystery why, with my initial config files, a stop & restart solves the issue.
 
Okay, solved. I had to replace this line:
Code: 
pass out from $ext_if to any

By this:
Code: 
pass out from ($ext_if) to any
 
I've just looked into this thread and the solution. Can someone please elaborate why wrapping up `$ext_if` in `()` solved the issue and made the rule work?
 
madwebness said:
I've just looked into this thread and the solution. Can someone please elaborate why wrapping up `$ext_if` in `()` solved the issue and made the rule work?
Click to expand...
As SirDice already pointed out: Parentheses around an interface `($interface)` makes pf reinterpret the rules when something changes on $interface.
 
madwebness said:
I've just looked into this thread and the solution. Can someone please elaborate why wrapping up `$ext_if` in `()` solved the issue and made the rule work?
Click to expand...
See PACKET FILTERING section, PARAMETERS sub-section, under the from sub-sub-section:

pf.conf(5)

man.freebsd.org man.freebsd.org

Host name resolution and interface to address translation are
done at ruleset load-time. When the address of an interface (or
host name) changes (under DHCP or PPP, for instance), the ruleset
must be reloaded for the change to be reflected in the kernel.
Surrounding the interface name (and optional modifiers) in paren-
theses changes this behaviour. When the interface name is sur-
rounded by parentheses, the rule is automatically updated when-
ever the interface changes its address. The ruleset does not
need to be reloaded. This is especially useful with nat.

It has already been described by other posters above but just in case you wanted to see where it is documented.
 
You must log in or register to reply here.
Back
Top