Solved VNET and Jails

Hello, is this kind of setup even possible:
vServer (HOST): has a jail where Caddy runs. Host and Caddy are connected via epair0a and epai0b /30 subnet. A pf rule is set up on the host that forwards all traffic to Caddy. I have another jail called chat. It is connected to Caddy via epair1a and epair1b on the /30 network. It is no problem to reach the jail with chat from the outside, but is it possible to establish a connection from chat to the push server? I can't really use pf in Caddy on vnet interfaces.

FreeBSD 14.3.

Best regards, Walter.
 
If you bridge epair0a and epair1a on the host and enable packet forwarding on the host, things should work. Test without running a firewall first so that you can see and verify that traffic actually does flow. Once it does, there really isn't that much stopping you from using a firewall inside each jail to limit each jail's incoming traffic.
 
Thanks bvdw78
Maybe I'm missing something, but epair1a is not visible to the host. epair1a is only in the caddy and the opposite, epair1b, is only in the chat.I can access the chat jail from outside. However, it cannot reach Mattermost's push servers.
 
vServer (HOST): has a jail where Caddy runs. Host and Caddy are connected via epair0a and epai0b /30 subnet. A pf rule is set up on the host that forwards all traffic to Caddy. I have another jail called chat. It is connected to Caddy via epair1a and epair1b on the /30 network.
Could you show how you created all this with some script or some such? It makes it easier to understand your topology...
Alternatively, post output of 'jls' on host, plus ifconfig from both host and jails (redacted if you prefer in case of public IPs).
 
All my config files are located in /etc/jail.conf.d/jail_name.conf, for example chat.conf:
Code:
chat {
  vnet;

  host.hostname   = "${name}.local";
  exec.start      = "/bin/sh /etc/rc";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  exec.stop     = "/bin/sh /etc/rc.shutdown";
  exec.release  = "ifconfig epair1a destroy";
  allow.raw_sockets;
  exec.clean;
  mount.devfs;

  path = "/usr/local/jails/containers/${name}";
}
Other jails such as mariadb and icinga2 are created in this way.
The interfaces are created in an rc.d script and assigned, for example:
Code:
#!/bin/sh

# PROVIDE: epairsetup
# REQUIRE: jail
# BEFORE: LOGIN

. /etc/rc.subr

name="epairsetup"
rcvar="epairsetup_enable"
start_cmd="${name}_start"

epairsetup_start() {
    if ! ifconfig epair1 >/dev/null 2>&1; then
        ifconfig epair1 create
        ifconfig epair1a vnet caddy
        ifconfig epair1b vnet chat
        jexec caddy ifconfig epair1a 10.10.40.1/30 up
        jexec chat ifconfig epair1b 10.10.40.2/30 up
    fi
}

Only Caddy jail differs from others: caddy.conf:

Code:
caddy {
#ip4 = inherit;
#interface = vtnet0;
  $epair_interface = "${name}_b";
  vnet;
  vnet.interface = $epair_interface;
  exec.prestart = "
    epair=\$(ifconfig epair create);
    epair_base=\${epair%a};
    ifconfig \${epair_base}a inet 10.10.30.1/30 up;
    ifconfig \${epair_base}b name ${epair_interface};
  ";

  host.hostname   = "${name}.local";
  exec.start      = "/bin/sh /etc/rc";
  exec.poststart  = "jexec ${name} ifconfig ${epair_interface} inet 10.10.30.2/30 up; jexec ${name} route add default 10.10.30.1";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  exec.stop     = "/bin/sh /etc/rc.shutdown";
  exec.release  = "ifconfig ${epair_interface} destroy";

  allow.raw_sockets;
  exec.clean;
  mount.devfs;

  path = "/usr/local/jails/containers/${name}";
}

ifconfig host:
epair0a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:2b:55:5a:10:0a
inet 10.10.30.1 netmask 0xfffffffc broadcast 10.10.30.3

ifconfig caddy:
epair0b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:2b:55:5a:10:0b
inet 10.10.30.2 netmask 0xfffffffc broadcast 10.10.30.3
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair1a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:e6:64:49:7f:0a
inet 10.10.40.1 netmask 0xfffffffc broadcast 10.10.40.3
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair3b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:70:42:5b:ed:0b
inet 10.10.50.1 netmask 0xfffffffc broadcast 10.10.50.3
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

My initial idea was as follows: Request -> Host -> Cadyy -> chat like Mattermost.

So that each jail remains in a separate vnet. Isolated.

I think I'm trying to use the Caddy jail as a router, which isn't right.
 
Is it really exactly as you wrote? Sequence

ifconfig epair1 create
ifconfig epair1b vnet mon
jexec chat ifconfig epair1b 10.10.40.2/30 up

looks a bit weird to me... maybe

ifconfig epair1b vnet chat

should be used as the second one instead... it looks like you have more jails not relevant to the original question, so it could be a bit messy :)

How does 'ifconfig' output from chat and caddy jails look like?
 
Oh, Sorry, yes: ifconfig epair1b vnet. I have more Jails. I prepare the expenses and post them. How it really looks.
 
Hm, I did not see 'ifconfig' part in your message #5 when I wrote mine, weird...

Anyway, with vnet jail it is OK to use some as router, it should just work.

Maybe you need...
1. add 'gateway_enable=YES' into /etc/rc.conf file in caddy jail
2. add static route for net 10.10.40.0/30 via 10.10.30.2 to host

Output from 'netstat -rnf inet' on both host and jails would show it for sure.

Just think of vnet jails as totally separate systems, at least when composing network topology.
 
Interesting. I personally never used epair to link two jails together directly. It should probably work and give you an L2 connection. In that case you could assign something like 192.168.0.1 to epair1a and 192.168.0.2 to epair1b. If you have no firewall between them, they should "see" one another without any packet forwarding our routing between them. Something like

Code:
ifconfig_epair1a="inet 192.168.0.1 netmask 255.255.255.252"

Code:
ifconfig_epair1b="inet 192.168.0.2 netmask 255.255.255.252"

..and you should be able to ping both ways, unless I have all my notes mixed up.

I've only ever done epairs through a bridge interface on the host though. That gives you an L2 switch-like construct that you can plug in as many epairs as you like. I have multiple bridges for multiple networks, keeping things like a management and backup network separate from front door traffic. It's turtles all the way down of course, being completely virtual constructs inside a single host kernel, but it helps to keep concerns separated and firewalling is easy to reason about.
 
I've only ever done epairs through a bridge interface on the host though. That gives you an L2 switch-like construct that you can plug in as many epairs as you like.
My experience is actually the opposite - I did use epairs as L3 links connecting jails, only recently I start to use it as L2 link with if_bridge as common point. It just works for me both ways pefectly.
 
My setup.
Caddy:
cat /etc/jail.conf.d/caddy.conf:
Code:
caddy {
#ip4 = inherit;
#interface = vtnet0;
  $epair_interface = "${name}_b";
  vnet;
  vnet.interface = $epair_interface;
  exec.prestart = "
    epair=\$(ifconfig epair create);
    epair_base=\${epair%a};
    ifconfig \${epair_base}a inet 10.10.30.1/30 up;
    ifconfig \${epair_base}b name ${epair_interface};
  ";

  host.hostname   = "${name}.local";
  exec.start      = "/bin/sh /etc/rc";
  exec.poststart  = "jexec ${name} ifconfig ${epair_interface} inet 10.10.30.2/30 up; jexec ${name} route add default 10.10.30.1";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  exec.stop     = "/bin/sh /etc/rc.shutdown";
  exec.release  = "ifconfig ${epair_interface} destroy";

  allow.raw_sockets;
  exec.clean;
  mount.devfs;

  path = "/usr/local/jails/containers/${name}";
}

jexec caddy netstat -rnf inet:
Code:
Internet:
Destination        Gateway            Flags         Netif Expire
default            10.10.30.1         UGS         caddy_b
10.10.30.0/30      link#8             U           caddy_b
10.10.30.2         link#9             UHS             lo0
10.10.40.0/30      link#15            U           epair1a
10.10.40.1         link#9             UHS             lo0
10.10.50.0/30      link#20            U           epair3b
10.10.50.1         link#9             UHS             lo0
127.0.0.1          link#9             UH              lo0

jexec caddy cat /etc/rc.conf:
Code:
caddy_enable="YES"
gateway_enable="YES"
defaultrouter="10.10.30.1"

jexec caddy ifconfig:
Code:
pflog0: flags=0 metric 0 mtu 33152
    options=0
    groups: pflog
caddy_b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:2b:55:5a:10:0b
    inet 10.10.30.2 netmask 0xfffffffc broadcast 10.10.30.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair1a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:e6:64:49:7f:0a
    inet 10.10.40.1 netmask 0xfffffffc broadcast 10.10.40.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair3b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:70:42:5b:ed:0b
    inet 10.10.50.1 netmask 0xfffffffc broadcast 10.10.50.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Host:
ifconfig:

Code:
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
    options=0
    groups: pflog
wg1: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
    options=80000<LINKSTATE>
    inet 10.10.10.1 netmask 0xffffff00
    groups: wg
    nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
epair0a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:2b:55:5a:10:0a
    inet 10.10.30.1 netmask 0xfffffffc broadcast 10.10.30.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

cat /usr/local/etc/rc.d/epairsetup

Code:
#!/bin/sh

# PROVIDE: epairsetup
# REQUIRE: jail
# BEFORE: LOGIN

. /etc/rc.subr

name="epairsetup"
rcvar="epairsetup_enable"
start_cmd="${name}_start"

epairsetup_start() {
    if ! ifconfig epair1 >/dev/null 2>&1; then
        ifconfig epair1 create
        ifconfig epair1a vnet caddy
        ifconfig epair1b vnet mon
        jexec caddy ifconfig epair1a 10.10.40.1/30 up
        jexec mon ifconfig epair1b 10.10.40.2/30 up
    fi

    if ! ifconfig epair2 >/dev/null 2>&1; then
        ifconfig epair2 create
        ifconfig epair2b vnet mariadb
        ifconfig epair2a vnet chat
        jexec mariadb ifconfig epair2b 10.10.20.2/30 up
        jexec chat ifconfig epair2a 10.10.20.1/30 up
    fi

    if ! ifconfig epair3 >/dev/null 2>&1; then
        ifconfig epair3 create
        ifconfig epair3b vnet caddy
        ifconfig epair3a vnet chat
        jexec caddy ifconfig epair3b 10.10.50.1/30 up
        jexec chat ifconfig epair3a 10.10.50.2/30 up
    fi
}

Jail chat:
cat /etc/jail.conf.d/chat.conf:

Code:
chat {
#ip4 = inherit;
#interface = vtnet0;
  vnet;

  host.hostname   = "${name}.local";
  exec.start      = "/bin/sh /etc/rc";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  exec.stop     = "/bin/sh /etc/rc.shutdown";
  exec.release  = "ifconfig epair2a destroy; ifconfig epair3a destroy";
  allow.raw_sockets;
  exec.clean;
  mount.devfs;

  path = "/usr/local/jails/containers/${name}";
}

jexec chat ipconfig:

Code:
pflog0: flags=0 metric 0 mtu 33152
    options=0
    groups: pflog
epair2a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:40:bb:1f:69:0a
    inet 10.10.20.1 netmask 0xfffffffc broadcast 10.10.20.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair3a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:70:42:5b:ed:0a
    inet 10.10.50.2 netmask 0xfffffffc broadcast 10.10.50.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


jexec chat cat /etc/rc.conf:

Code:
mattermostd_enable="YES"
mattermostd_conf="/usr/local/etc/mattermost/config.json"
defaultrouter="10.10.50.1"

jexec chat netstat -rnf inet:

Code:
Internet:
Destination        Gateway            Flags         Netif Expire
10.10.20.0/30      link#17            U           epair2a
10.10.20.1         link#13            UHS             lo0
10.10.50.0/30      link#19            U           epair3a
10.10.50.2         link#13            UHS             lo0
127.0.0.1          link#13            UH              lo0
 
My setup.
Caddy:
cat /etc/jail.conf.d/caddy.conf:
Code:
caddy {
#ip4 = inherit;
#interface = vtnet0;
  $epair_interface = "${name}_b";
  vnet;
  vnet.interface = $epair_interface;
  exec.prestart = "
    epair=\$(ifconfig epair create);
    epair_base=\${epair%a};
    ifconfig \${epair_base}a inet 10.10.30.1/30 up;
    ifconfig \${epair_base}b name ${epair_interface};
  ";

  host.hostname   = "${name}.local";
  exec.start      = "/bin/sh /etc/rc";
  exec.poststart  = "jexec ${name} ifconfig ${epair_interface} inet 10.10.30.2/30 up; jexec ${name} route add default 10.10.30.1";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  exec.stop     = "/bin/sh /etc/rc.shutdown";
  exec.release  = "ifconfig ${epair_interface} destroy";

  allow.raw_sockets;
  exec.clean;
  mount.devfs;

  path = "/usr/local/jails/containers/${name}";
}

jexec caddy netstat -rnf inet:
Code:
Internet:
Destination        Gateway            Flags         Netif Expire
default            10.10.30.1         UGS         caddy_b
10.10.30.0/30      link#8             U           caddy_b
10.10.30.2         link#9             UHS             lo0
10.10.40.0/30      link#15            U           epair1a
10.10.40.1         link#9             UHS             lo0
10.10.50.0/30      link#20            U           epair3b
10.10.50.1         link#9             UHS             lo0
127.0.0.1          link#9             UH              lo0

jexec caddy cat /etc/rc.conf:
Code:
caddy_enable="YES"
gateway_enable="YES"
defaultrouter="10.10.30.1"

jexec caddy ifconfig:
Code:
pflog0: flags=0 metric 0 mtu 33152
    options=0
    groups: pflog
caddy_b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:2b:55:5a:10:0b
    inet 10.10.30.2 netmask 0xfffffffc broadcast 10.10.30.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair1a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:e6:64:49:7f:0a
    inet 10.10.40.1 netmask 0xfffffffc broadcast 10.10.40.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair3b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:70:42:5b:ed:0b
    inet 10.10.50.1 netmask 0xfffffffc broadcast 10.10.50.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Host:
ifconfig:

Code:
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
    options=0
    groups: pflog
wg1: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
    options=80000<LINKSTATE>
    inet 10.10.10.1 netmask 0xffffff00
    groups: wg
    nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
epair0a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:2b:55:5a:10:0a
    inet 10.10.30.1 netmask 0xfffffffc broadcast 10.10.30.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

cat /usr/local/etc/rc.d/epairsetup

Code:
#!/bin/sh

# PROVIDE: epairsetup
# REQUIRE: jail
# BEFORE: LOGIN

. /etc/rc.subr

name="epairsetup"
rcvar="epairsetup_enable"
start_cmd="${name}_start"

epairsetup_start() {
    if ! ifconfig epair1 >/dev/null 2>&1; then
        ifconfig epair1 create
        ifconfig epair1a vnet caddy
        ifconfig epair1b vnet mon
        jexec caddy ifconfig epair1a 10.10.40.1/30 up
        jexec mon ifconfig epair1b 10.10.40.2/30 up
    fi

    if ! ifconfig epair2 >/dev/null 2>&1; then
        ifconfig epair2 create
        ifconfig epair2b vnet mariadb
        ifconfig epair2a vnet chat
        jexec mariadb ifconfig epair2b 10.10.20.2/30 up
        jexec chat ifconfig epair2a 10.10.20.1/30 up
    fi

    if ! ifconfig epair3 >/dev/null 2>&1; then
        ifconfig epair3 create
        ifconfig epair3b vnet caddy
        ifconfig epair3a vnet chat
        jexec caddy ifconfig epair3b 10.10.50.1/30 up
        jexec chat ifconfig epair3a 10.10.50.2/30 up
    fi
}

Jail chat:
cat /etc/jail.conf.d/chat.conf:

Code:
chat {
#ip4 = inherit;
#interface = vtnet0;
  vnet;

  host.hostname   = "${name}.local";
  exec.start      = "/bin/sh /etc/rc";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  exec.stop     = "/bin/sh /etc/rc.shutdown";
  exec.release  = "ifconfig epair2a destroy; ifconfig epair3a destroy";
  allow.raw_sockets;
  exec.clean;
  mount.devfs;

  path = "/usr/local/jails/containers/${name}";
}

jexec chat ipconfig:

Code:
pflog0: flags=0 metric 0 mtu 33152
    options=0
    groups: pflog
epair2a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:40:bb:1f:69:0a
    inet 10.10.20.1 netmask 0xfffffffc broadcast 10.10.20.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair3a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:70:42:5b:ed:0a
    inet 10.10.50.2 netmask 0xfffffffc broadcast 10.10.50.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


jexec chat cat /etc/rc.conf:

Code:
mattermostd_enable="YES"
mattermostd_conf="/usr/local/etc/mattermost/config.json"
defaultrouter="10.10.50.1"

jexec chat netstat -rnf inet:

Code:
Internet:
Destination        Gateway            Flags         Netif Expire
10.10.20.0/30      link#17            U           epair2a
10.10.20.1         link#13            UHS             lo0
10.10.50.0/30      link#19            U           epair3a
10.10.50.2         link#13            UHS             lo0
127.0.0.1          link#13            UH              lo0
Caddy:
jexec caddy sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 1 -> 1

Host: gateway_enbale=YES
 
Which of your hosts/jails is going to be forwarding any traffic? Maybe I'm overlooking that, but I haven't drawn this out on paper yet so that may just be me. Seems like you should get specific routes for every subnet your systems participate in, plus a default gateway for anything non-specific.
 
What you pasted, looks basically OK and sound, I just see no default route for vnet chat, even if defaultrouter is set in its config file... please verify it is not copy/paste problem.
I see no output of 'netstat -rnf' for host.

Back to your original question, you ask about possibility of connection from chat to push server. What and where is this push server (network wise)? Also, is my assumption host has direct connection to internet, possibly with own public IP, true?
 
Caddy:
jexec caddy sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 1 -> 1

Host: gateway_enbale=YES
You see here caddy can and will act as a router, given all other conditions are met. Also, try whether you can use 'tcpdump' in a jail to capture and analyze network traffic.
 
dino1
jexec chat netstat -rnf inet
Routing tables

Internet:
Destination Gateway Flags Netif Expire
10.10.20.0/30 link#17 U epair2a
10.10.20.1 link#13 UHS lo0
10.10.30.0/30 10.10.50.1 UGS epair3a
10.10.50.0/30 link#19 U epair3a
10.10.50.2 link#13 UHS lo0
127.0.0.1 link#13 UH lo0

jexec chat cat /etc/rc.conf
mattermostd_enable="YES"
mattermostd_conf="/usr/local/etc/mattermost/config.json"
defaultrouter="10.10.50.1"
 
bvdw78 Everything should go through caddy. Caddy has many vnets with 255.255.255.252, essentially one leg with itself and the other in another jail.
 
jexec chat netstat -rnf inet
Routing tables

Internet:
Destination Gateway Flags Netif Expire
10.10.20.0/30 link#17 U epair2a
10.10.20.1 link#13 UHS lo0
10.10.30.0/30 10.10.50.1 UGS epair3a
10.10.50.0/30 link#19 U epair3a
10.10.50.2 link#13 UHS lo0
127.0.0.1 link#13 UH lo0

jexec chat cat /etc/rc.conf
mattermostd_enable="YES"
mattermostd_conf="/usr/local/etc/mattermost/config.json"
defaultrouter="10.10.50.1"
For some reason, there is no default route entry - compare it with 'jexec caddy netstat -rnf inet' output (I am looking into your message #11).
For now, as a quick fix, just issue 'route add default 10.10.50.1' command in vnet chat - full command is 'jexec chat route add default 10.10.50.1'.
Then verify routing table is properly changed.

(When working for some more time with a vnet jail, I often just start with 'jexec <jailname>', so I have a shell I can use without constantly prefixing command with that - but you can still do it with 'jexec <jailname>' prefix for each command intended to go to jail as intended, it is just more typing, possibly more typos could be produced this way.)
 
I thought isolation = more security. I'll try your suggestion from above when I get home.
If you have a better or state-of-the-art suggestion on how to operate “mattermost, icinga2, vaultwarden (only vpn interface), and cloud like owncloud” securely on a public server, then I'm all ears.:-)
 
So, i create new jail Test. Ip 10.10.100.1 epair4a and in Caddy 10.10.100.2 epair4b. Ping from caddy to Test and from Test to Caddy ok. Caddy ping to 1.1.1.1 ok. Im Test jail route add default 10.10.100.2. ping from test to 1.1.1.1 fail
 
Is the intended route 'test -> caddy -> host -> internet'?
What does 'ping 10.10.30.2' from test jail show?
And what about 'ping 10.10.30.1' from test jail?
 
Back
Top