Solved IPv6 unreachable in dual-stack host (jail works)

A

algi

Hi! I have a server with FreeBSD 13.2 with dual stack. The problem is with IPv6 on host, which ends always with timeout. Curiously dual-stack jails works perfectly fine with IPv6. Can anyone please take a look at my config and point me to the right direction? Thanks!

/etc/rc.conf (only IPv6 relevant stuff, IPv4 parts removed):
Code: 
ifconfig_vtnet0_ipv6="inet6 accept_rtadv"
ipv6_gateway_enable="YES"
ipv6_defaultrouter="fe80::5400:3ff:fefb:bcb2%vtnet0"
rtsold_enable="YES"
rtsold_flags="-aF"
ipv6_activate_all_interfaces="YES"

# jail network
cloned_interfaces="bridge0"
ifconfig_bridge0_name="vtnet0bridge"
ifconfig_vtnet0bridge_ipv6="inet6 fd10:6c79:8ae5:8b91::1 -ifdisabled auto_linklocal

/etc/rtadv.conf
Code: 
vtnet0bridge:addr="fd10:6c79:8ae5:8b91::"

That's all I remember I touched. Please let me know if you need more files, I'll post them here. Thanks a lot!
 
Curiously, ping6 works even from the host, same as with traceroute6. But when I try to fetch something, it always ends on timeout error. DNS also works for IPv6. It doesn't make any sense to me, because IPv6 jail with VNET works perfectly fine.
 
I'm not sure about that, because otherwise I'd be blocked from the jail as well. But it's important to mention it anyway as I'm doing NAT on IPv6 there, so here it is (relevant parts of /etc/pf.conf):

Code: 
ext_if="vtnet0"
dmz_if="vtnet0bridge"

set block-policy drop
set skip on lo

scrub in all fragment reassemble no-df
nat on $ext_if inet6 from $dmz_if:network to any -> ($ext_if:0)

rdr pass on $ext_if proto tcp to ($ext_if) port { http https } -> $apache_ipv4
rdr pass on $ext_if proto tcp to ($ext_if) port { http https } -> $apache_ipv6

antispoof quick for $ext_if

block all
pass inet6 proto icmp6 all

pass out on $ext_if

pass on $dmz_if proto icmp from $apache_ipv4
pass on $dmz_if proto udp from { $apache_ipv4 $apache_ipv6 } to any port domain
pass on $dmz_if proto tcp from { $apache_ipv4 $apache_ipv6 } to any port { http https }
pass on $dmz_if proto tcp from any to { $apache_ipv4 $apache_ipv6 } port { http https }
 
It's definitely a firewall. When I run doas tcpdump -n -ttt -i pflog0 ip6, then suddenly I can do fetch -6 google.com without any trouble. There must be something seriously wrong with my pf.conf setup.
 
algi said:
It's definitely a firewall. When I run doas tcpdump -n -ttt -i pflog0 ip6, then suddenly I can do fetch -6 google.com without any trouble. There must be something seriously wrong with my pf.conf setup.
Click to expand...
;) I still ?got it... Unfortunately I got my own battle with PF that I've been putting off for a while (not sure how much of help I could be)....

I'll tell some things I like to do in my configs...
(Using your config as an example)

I tend not to do rules on the bridge.
I tend to
Code: 
skip set vtnet0
&
Code: 
 skip set vtnet0bridge
on the host PF rules
(why because usually the jail/bhyve will have its own PF rules)


Also from the rules you posted I am not sure if this is a bhyve, jail, host (metal)? If the rules above is the PF of the bhyve you will need to post PF config of the host.

Check this might be the culprit....

Code: 
pass in on $ext_if

Why don't you have a pass in for the vtnet0 on example above? $dmz_if is all on the bridge but how does the vtnet0 passing to the host (assuming this is a bhyve) ?
 
It's actually VM deployed on Vultr. But the issue is that it's quite unpredictable and sometimes it works and sometimes not... No wonder that nobody here's interested to solve it. When I read the book about PF, it barely mentions IPv6. My assumption is that almost nobody actually knows how it really works (including me of course, but at least I'm not sysadmin professional).

Anyway, even with basic PF config like block all in, pass all out with passing all ICMPv6 traffic, it's not enough. Sometimes it works, sometimes not... This is the most frustrating part about it. Not sure where to go from there...
 
I recently just finish The Book of PF - A No-Nonsense Guide to the OpenBSD Firewall that's why its fresh in my head.

Anyways I don't use icmp6 or ipv6 BUT here are some snippets that got my attention on ipv6 from the books that might help you debug.

In modern IPv6 networks, the updated icmp6 protocol plays a more crucial role than ever in parameter passing and even host configuration, and network admins are playing a high-stakes game while learning the finer
points of blocking or passing icmp6 traffic. To a large extent, issues that are relevant for IPv4 ICMP generally apply to IPv6 ICMP6 as well, but in addition, ICMP6 is used for several mechanisms that were handled differently in IPv4.

If you want the same free flow of messages for your IPv6 traffic, the corresponding rule is this:

Code: 
pass inet6 proto icmp6
Click to expand...



The traceroute command (and the IPv6 variant traceroute6) is useful when your users claim that the Internet isn't working. By default, Unix traceroute uses UDP connections according to a set formula based on destination. The following rules work with the traceroute and traceroute6 commands on all forms of Unix I've had access to, including GNU/Linux:

# allow out the default range for traceroute(8):
# "base+nhops*nqueries-1" (33434+64*3-1)
Code: 
pass out on egress inet proto udp to port 33433:33626 # For IPv4
Code: 
pass out on egress inet6 proto udp to port 33433:33626 # For IPv6

For IPv6, you'd probably want to let the more common ICMP6 diagnostics through, such as the following:
Code: 
icmp6_types = "{ echoreq unreach timex paramprob }"
This means that we let echo requests and destination unreachable, time exceeded, and parameter problem messages pass for IPv6 traffic.

But it's worth keeping in mind that IPv6 hosts rely on ICMP6 messages for automatic configuration-related tasks, and you may want to explicitly filter in order to allow or deny specific ICMP6 types at various points in
your network.
Click to expand...
You're saying VM, its probably a Bhyve/VM so Vultr has a metal host running it.... So it could be a Vultr issue...

Outside of this cannot be of much help... If you find a solution I would like to know.
 
I found the problem. I have followed an article that said I should configure router advertisement daemon, but I forgot to start the daemon. For a weird reason, the IPv6 traffic flowed perfectly fine inside and outside the jail, but had a problem from going directly from the host outside. After I started the daemon, everything started working perfectly...

Thanks for the moral support GoNeFast_01 !
 
algi said:
I found that turning on rtadvd won't help by itself. I had a typo in configuration, but the real problem must be the whole IPv6 setup. I don't know how to solve it, so I give up. I was trying to follow this article (https://meka.rs/blog/2022/01/22/freebsd-dual-stack-jails/), but since I'm not expert on IPv6, I haven't been able to follow it properly.
Click to expand...
I'll check it out on my end but that guide looks simple enough... You tried with those simple PF rules and nat from his guide (Unique Local Addresses)?
I like Meka works and guides ....I use Reggae. ?
 
The issue is that he's assigning public IPv6 address on the bridge and that doesn't work for me. I do have all the rules in place to allow the traffic and I do have NAT for IPv4 and 6 in place too. What I'm doing is using local address for the bridge and I do translation on it. It seems like that this simply doesn't work. My idea was to do the filtering from jails on bridge, but it seems like it's not quite viable. I start to wonder what benefit actually jails brings, if I can't do the filtering properly outside of them. I don't really want to pass PF into them, because that defeats the purpose of considering them as dispensable. My thinking was that if the attacker gains access to them, they'll be able to only do harm to the service I run there and the rest of the system should stay intact. But if I'm supposed to pass PF device there, then they can change the rules as they please, which makes jails quite pointless and just a complication IMHO.
 
Again going back to PF Book I just read.

Code: 
pass in on $ext_if inet proto tcp to $ext_if port $webports rdr-to $webserver
pass in on $ext_if inet proto tcp to $ext_if port $email rdr-to $mailserver
pass on $int_if inet proto tcp to $webserver port $webports
pass on $int_if inet proto tcp to $mailserver port $email
The last four rules here are the ones that interest us the most. If you
try to reach the services on the official address from hosts in your own
network, you’ll soon see that the requests for the redirected services from
machines in your local network most likely never reach the external inter-
face. This is because all the redirection and translation happens on the
external interface. The gateway receives the packets from your local network
on the internal interface, with the destination address set to the external
interface’s address. The gateway recognizes the address as one of its own
and tries to handle the request as if it were directed at a local service; as a
consequence, the redirections don’t quite work from the inside.
Click to expand...


REVIEW below there are some workaround, this might be an issue with reflection.
Redirection and Reflection.


What you're doing is possible just adding nat/translation in the bridge might lead to some issues, that's why for example (I presume) Meka use alias instead in bridge IP.
 
It's definitely not a firewall. I can do 2x "fetch -6 https://google.com" and it works, straight after fresh reboot. Then I run it once more and it fails on time out. What's strange is my routing table for IPv6. For some reason I've got 2 entries there for default gateway. Not sure why's that. For IPv4 I've got only one. See:

default fe80::fc00:3ff:fefb:bcb2%vtnet0 UG vtnet0
default fe80::5400:3ff:fefb:bcb2%vtnet0 UGS vtnet0

Also, this never ever fails in jail, only fails randomly in the host itself.
 
Okay, this time it's truly solved. I will post my full setup here, in case some lost soul will try to accomplish the same - working dual-stack jails with local IP addresses for both IPv4 and IPv6.

/etc/pf.conf (relevant parts)
Code: 
ext_if="vtnet0"       # external interface
dmz_if="vtnet0bridge" # cloned interface (bridge)

jail4="10.0.0.10" # private IPv4 for jail
jail6="fd10::10"  # private IPv6 for jail
jail_port="{ http https}" # ports for jail (web server)

nat on $ext_if inet  from $dmz_if:network to any -> ($ext_if)   # IPv4 NAT
nat on $ext_if inet6 from $dmz_if:network to any -> ($ext_if:0) # IPv6 NAT

rdr pass on $ext_if proto tcp to ($ext_if) port $jail_port -> $jail4
rdr pass on $ext_if proto tcp to ($ext_if) port $jail_port -> $jail6

pass quick inet6 proto icmp6 all # pass all IPv6 ICMP traffic (you can restrict it if you wish)

block all
pass from { self $dmz_if:network }

pass on $dmz_if proto udp from $jail4 port domain
pass on $dmz_if proto udp from $jail6 port domain
pass on $dmz_if proto tcp from $jail4 port $jail_port
pass on $dmz_if proto tcp from $jail6 port $jail_port

/etc/rc.conf (host, relevant parts)
Code: 
ifconfig_vtnet0_ipv6="inet6 auto_linklocal accept_rtadv"
ipv6_gateway_enable="YES"
ipv6_defaultrouter="fe80::dead:beef%vtnet0" # this is hard to get, I started with wrong address, then run '# netstat -rn6 | grep default" to get the right address and put it here

rtsold_enable="YES" # enable router solicitation
rtsold_flags="-aF" # enable all interfaces, disable IPv6 forwarding
ipv6_activate_all_interfaces="YES" # not sure if it's necessary...

cloned_interfaces="bridge0"
ifconfig_bridge0_name="vtnet0bridge"
ifconfig_vtnet0bridge="inet 10.0.0.1/24"
ifconfig_vtnet0bridge_ipv6="inet6 fd10::1 -ifdisabled auto_linklocal"

pf_enable="YES"
jail_enable="YES"

Jail's /etc/rc.conf (relevant parts)
Code: 
defaultrouter="10.0.0.10"
ipv6_defaultrouter="fd10::1"

ifconfig_e0b_jail="inet 10.0.0.10/24"
ifconfig_e0b_jail_ipv6="inet6 fd10::10 -ifdisabled auto_linklocal"

rtsold_enable="YES"
 
You must log in or register to reply here.
Back
Top