Help resolving ipv4/6 route between wireguard into jail on local homelab and vps (exit node, wg server)

srey

srey

Hi !

I'm trying to create such architecture :
- wireguard running kernel mode
- each jail running into bridged vnet using `-B` of bastille, on a 10.10.1.x subnet
- wireguard client running into one of the bastille jail, on wg ip 10.10.1.2
- wireguard server running into boxybsd vps, on wg ip 10.10.1.1,
- boxybsd run on ivp6 only at 2a13:e3c1:400e:1337::690

My boot/loader.conf :

Code: 
security.bsd.allow_destructive_dtrace=0
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
geom_mirror_load="YES"
if_wg_load="YES"


shapes at 25-12-26 16.42.18.png


My problem is actually wg client and server seems connected (see wg0 log),
but i cannot ping any ip : 10.10.1.1 (from 10.10.1.2) or 10.10.1.2 (from 10.10.1.1)

Client wg command :

Code: 
interface: wg0
public key: 3on5j0F7kEXG4dlTGrk90ONQv7X6f6yRVO3kGIGf3zQ=
private key: (hidden)
listening port: 50081

peer: OqMqYbF8Hi88IMlucdAvnDEBzxB716msCizauHJ2IXg=
  endpoint: [2a13:e3c1:400e:1337::690]:51871
  allowed ips: (none)
  latest handshake: 2 minutes, 30 seconds ago
  transfer: 17.24 KiB received, 45.37 KiB sent
  persistent keepalive: every 15 seconds

VPS wg command :

Code: 
interface: wg0
  public key: OqMqYbF8Hi88IMlucdAvnDEBzxB716msCizauHJ2IXg=
  listening port: 51871

peer: 3on5j0F7kEXG4dlTGrk90ONQv7X6f6yRVO3kGIGf3zQ=
  endpoint: [2001:861:3e10:7400:5a9c:fcff:fe10:550f]:50081
  allowed ips: 10.10.1.2/32
  latest handshake: 30 seconds ago
  transfer: 45.99 KiB received, 65.59 KiB sent

The Host pf.conf

Code: 
ext_if="re0"
int_if="bridge0"
jailnet="192.168.42.0/24"
tailnet="100.64.0.0/10"
allowed_ports = "{ 22, 80, 443, 8080 }"

set block-policy return
scrub in on $ext_if all fragment reassemble

set skip on lo
nat on $ext_if from $jailnet to any -> ($ext_if)
rdr-anchor "rdr/*"
block log (all) all
pass in quick inet6 proto icmp6 icmp6-type echoreq
pass in quick inet proto icmp icmp-type echoreq

pass out quick keep state

pass in quick on $int_if
pass out quick on $ext_if from $jailnet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

The Vps pf.conf

Code: 
ext_if="vtnet0"
wg_net="10.10.1.1/32"
wg_if="wg0"
set skip on lo

scrub in all
nat on $ext_if from $wg_net to any -> ($ext_if)

pass in quick inet6 proto icmp6 icmp6-type echoreq
pass in quick inet proto icmp icmp-type echoreq

pass in on $ext_if proto udp from any to ($ext_if) port 51871
pass quick on $wg_if
pass log all

The Host rc.conf and sysctl.conf

Code: 
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="Nostromo"
keymap="fr.kbd"
ifconfig_re0="DHCP"
ifconfig_re0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
tailscaled_enable="YES"
bastille_enable="YES"
pf_enable="YES"
pflog_enable="YES"
ipv6_activate_all_interfaces="YES"
ipv6_gateway_enable="YES"
gateway_enable="YES"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.42.1/24 addm re0 up"
ifconfig_bridge0_ipv6="inet6 0000:0000:0000:0000:0000:ffff:c0a8:2a01 addm re0 up"
rtsold_enable="YES"
rtsold_flags="-i -m bridge0"

Code: 
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
kern.randompid=1
vfs.zfs.min_auto_ashift=12
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=0
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
net.inet6.ip6.rfc6204w3=1

The jail rc.conf :

Code: 
ifconfig_e0b_wireguard_name="vnet0"
ifconfig_vnet0="inet 192.168.42.2/24"
ifconfig_vnet0_ipv6="inet 6 0000:0000:0000:0000:0000:ffff:c0a8:2a01 accept_rtadv"
ifconfig_vnet0_descr="jail interface for bridge0"
defaultrouter="192.168.42.1"
syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
cron_flags="-J 60"
cloned_interfaces="wg0"
ifconfig_wg0="inet 10.10.1.2/24"
gateway_enable="YES"
ipv6_defaultrouter="0000:0000:0000:0000:0000:ffff:c0a8:2a01"
ipv6_ipv4mapping="YES"


Bastille devfs.rules conf :

Code: 
[bastille_vnet=13]
add path 'bpf*' unhide

Netstat info for Host :

Code: 
root@wireguard:~ # netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.42.1       UGS           vnet0
10.10.1.0/24       link#10            U               wg0
10.10.1.2          link#8             UHS             lo0
127.0.0.1          link#8             UH              lo0
192.168.42.0/24    link#7             U             vnet0
192.168.42.2       link#8             UHS             lo0

root@wireguard:~ # netstat -rn6
Routing tables

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::/96                             link#8                        URS             lo0
default                           fe80::32b1:b5ff:fefd:a43c%vnet0 UG          vnet0
default                           ::ffff:192.168.42.1           UGS             lo0
::1                               link#8                        UHS             lo0
::ffff:0.0.0.0/96                 link#8                        URS             lo0
2001:861:3e10:7400::/64           link#7                        U             vnet0
2001:861:3e10:7400:5a9c:fcff:fe10:550f link#8                   UHS             lo0
fe80::%lo0/10                     link#8                        URS             lo0
fe80::%vnet0/64                   link#7                        U             vnet0
fe80::5a9c:fcff:fe10:550f%lo0     link#8                        UHS             lo0
fe80::%lo0/64                     link#8                        U               lo0
fe80::1%lo0                       link#8                        UHS             lo0
ff02::/16                         link#8                        URS             lo0

Netstat on VPS

Code: 
$ netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
10.10.1.0/24       link#4             U               wg0
10.10.1.1          link#2             UHS             lo0
10.10.1.2          link#4             UHS             wg0
127.0.0.1          link#2             UH              lo0

$ netstat -rn6
Routing tables

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::/96                             link#2                        URS             lo0
default                           2a13:e3c1::1                  UGS          vtnet0
::1                               link#2                        UHS             lo0
::ffff:0.0.0.0/96                 link#2                        URS             lo0
2a13:e3c1::/32                    link#1                        U            vtnet0
2a13:e3c1:400e:1337::690          link#2                        UHS             lo0
fe80::%lo0/10                     link#2                        URS             lo0
fe80::%vtnet0/64                  link#1                        U            vtnet0
fe80::be24:11ff:fe98:4675%lo0     link#2                        UHS             lo0
fe80::%lo0/64                     link#2                        U               lo0
fe80::1%lo0                       link#2                        UHS             lo0
ff02::/16                         link#2                        URS             lo0

The wg Conf on VPS :

Code: 
$ wg showconf wg0
[Interface]
ListenPort = 51871

[Peer]
PublicKey = 3on5j0F7kEXG4dlTGrk90ONQv7X6f6yRVO3kGIGf3zQ=
AllowedIPs = 10.10.1.2/32
Endpoint = [2001:861:3e10:7400:5a9c:fcff:fe10:550f]:50081

The wg Conf on Jail Wireguard :

Code: 
root@wireguard:~ # wg showconf wg0
[Interface]
ListenPort = 50081
PrivateKey = XXX

[Peer]
PublicKey = OqMqYbF8Hi88IMlucdAvnDEBzxB716msCizauHJ2IXg=
Endpoint = [2a13:e3c1:400e:1337::690]:51871
PersistentKeepalive = 15

I miss something but what ? ...
 
srey said:
Hi !

I'm trying to create such architecture :
- wireguard running kernel mode
- each jail running into bridged vnet using `-B` of bastille, on a 10.10.1.x subnet
- wireguard client running into one of the bastille jail, on wg ip 10.10.1.2
- wireguard server running into boxybsd vps, on wg ip 10.10.1.1,
- boxybsd run on ivp6 only at 2a13:e3c1:400e:1337::690

My boot/loader.conf :

Code: 
security.bsd.allow_destructive_dtrace=0
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
geom_mirror_load="YES"
if_wg_load="YES"


View attachment 24546

My problem is actually wg client and server seems connected (see wg0 log),
but i cannot ping any ip : 10.10.1.1 (from 10.10.1.2) or 10.10.1.2 (from 10.10.1.1)

Client wg command :

Code: 
interface: wg0
public key: 3on5j0F7kEXG4dlTGrk90ONQv7X6f6yRVO3kGIGf3zQ=
private key: (hidden)
listening port: 50081

peer: OqMqYbF8Hi88IMlucdAvnDEBzxB716msCizauHJ2IXg=
  endpoint: [2a13:e3c1:400e:1337::690]:51871
  allowed ips: (none)
  latest handshake: 2 minutes, 30 seconds ago
  transfer: 17.24 KiB received, 45.37 KiB sent
  persistent keepalive: every 15 seconds

VPS wg command :

Code: 
interface: wg0
  public key: OqMqYbF8Hi88IMlucdAvnDEBzxB716msCizauHJ2IXg=
  listening port: 51871

peer: 3on5j0F7kEXG4dlTGrk90ONQv7X6f6yRVO3kGIGf3zQ=
  endpoint: [2001:861:3e10:7400:5a9c:fcff:fe10:550f]:50081
  allowed ips: 10.10.1.2/32
  latest handshake: 30 seconds ago
  transfer: 45.99 KiB received, 65.59 KiB sent

The Host pf.conf

Code: 
ext_if="re0"
int_if="bridge0"
jailnet="192.168.42.0/24"
tailnet="100.64.0.0/10"
allowed_ports = "{ 22, 80, 443, 8080 }"

set block-policy return
scrub in on $ext_if all fragment reassemble

set skip on lo
nat on $ext_if from $jailnet to any -> ($ext_if)
rdr-anchor "rdr/*"
block log (all) all
pass in quick inet6 proto icmp6 icmp6-type echoreq
pass in quick inet proto icmp icmp-type echoreq

pass out quick keep state

pass in quick on $int_if
pass out quick on $ext_if from $jailnet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

The Vps pf.conf

Code: 
ext_if="vtnet0"
wg_net="10.10.1.1/32"
wg_if="wg0"
set skip on lo

scrub in all
nat on $ext_if from $wg_net to any -> ($ext_if)

pass in quick inet6 proto icmp6 icmp6-type echoreq
pass in quick inet proto icmp icmp-type echoreq

pass in on $ext_if proto udp from any to ($ext_if) port 51871
pass quick on $wg_if
pass log all

The Host rc.conf and sysctl.conf

Code: 
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="Nostromo"
keymap="fr.kbd"
ifconfig_re0="DHCP"
ifconfig_re0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
tailscaled_enable="YES"
bastille_enable="YES"
pf_enable="YES"
pflog_enable="YES"
ipv6_activate_all_interfaces="YES"
ipv6_gateway_enable="YES"
gateway_enable="YES"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.42.1/24 addm re0 up"
ifconfig_bridge0_ipv6="inet6 0000:0000:0000:0000:0000:ffff:c0a8:2a01 addm re0 up"
rtsold_enable="YES"
rtsold_flags="-i -m bridge0"

Code: 
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
kern.randompid=1
vfs.zfs.min_auto_ashift=12
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=0
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
net.inet6.ip6.rfc6204w3=1

The jail rc.conf :

Code: 
ifconfig_e0b_wireguard_name="vnet0"
ifconfig_vnet0="inet 192.168.42.2/24"
ifconfig_vnet0_ipv6="inet 6 0000:0000:0000:0000:0000:ffff:c0a8:2a01 accept_rtadv"
ifconfig_vnet0_descr="jail interface for bridge0"
defaultrouter="192.168.42.1"
syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
cron_flags="-J 60"
cloned_interfaces="wg0"
ifconfig_wg0="inet 10.10.1.2/24"
gateway_enable="YES"
ipv6_defaultrouter="0000:0000:0000:0000:0000:ffff:c0a8:2a01"
ipv6_ipv4mapping="YES"


Bastille devfs.rules conf :

Code: 
[bastille_vnet=13]
add path 'bpf*' unhide

Netstat info for Host :

Code: 
root@wireguard:~ # netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.42.1       UGS           vnet0
10.10.1.0/24       link#10            U               wg0
10.10.1.2          link#8             UHS             lo0
127.0.0.1          link#8             UH              lo0
192.168.42.0/24    link#7             U             vnet0
192.168.42.2       link#8             UHS             lo0

root@wireguard:~ # netstat -rn6
Routing tables

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::/96                             link#8                        URS             lo0
default                           fe80::32b1:b5ff:fefd:a43c%vnet0 UG          vnet0
default                           ::ffff:192.168.42.1           UGS             lo0
::1                               link#8                        UHS             lo0
::ffff:0.0.0.0/96                 link#8                        URS             lo0
2001:861:3e10:7400::/64           link#7                        U             vnet0
2001:861:3e10:7400:5a9c:fcff:fe10:550f link#8                   UHS             lo0
fe80::%lo0/10                     link#8                        URS             lo0
fe80::%vnet0/64                   link#7                        U             vnet0
fe80::5a9c:fcff:fe10:550f%lo0     link#8                        UHS             lo0
fe80::%lo0/64                     link#8                        U               lo0
fe80::1%lo0                       link#8                        UHS             lo0
ff02::/16                         link#8                        URS             lo0

Netstat on VPS

Code: 
$ netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
10.10.1.0/24       link#4             U               wg0
10.10.1.1          link#2             UHS             lo0
10.10.1.2          link#4             UHS             wg0
127.0.0.1          link#2             UH              lo0

$ netstat -rn6
Routing tables

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::/96                             link#2                        URS             lo0
default                           2a13:e3c1::1                  UGS          vtnet0
::1                               link#2                        UHS             lo0
::ffff:0.0.0.0/96                 link#2                        URS             lo0
2a13:e3c1::/32                    link#1                        U            vtnet0
2a13:e3c1:400e:1337::690          link#2                        UHS             lo0
fe80::%lo0/10                     link#2                        URS             lo0
fe80::%vtnet0/64                  link#1                        U            vtnet0
fe80::be24:11ff:fe98:4675%lo0     link#2                        UHS             lo0
fe80::%lo0/64                     link#2                        U               lo0
fe80::1%lo0                       link#2                        UHS             lo0
ff02::/16                         link#2                        URS             lo0

The wg Conf on VPS :

Code: 
$ wg showconf wg0
[Interface]
ListenPort = 51871

[Peer]
PublicKey = 3on5j0F7kEXG4dlTGrk90ONQv7X6f6yRVO3kGIGf3zQ=
AllowedIPs = 10.10.1.2/32
Endpoint = [2001:861:3e10:7400:5a9c:fcff:fe10:550f]:50081

The wg Conf on Jail Wireguard :

Code: 
root@wireguard:~ # wg showconf wg0
[Interface]
ListenPort = 50081
PrivateKey = XXX

[Peer]
PublicKey = OqMqYbF8Hi88IMlucdAvnDEBzxB716msCizauHJ2IXg=
Endpoint = [2a13:e3c1:400e:1337::690]:51871
PersistentKeepalive = 15

I miss something but what ? ...
Click to expand...
> The wg Conf on Jail Wireguard
You're missing AllowedIPs
 
You must log in or register to reply here.
Back
Top