Need help with jail networking

T

tom_h

I'm new to all this so do excuse me if this is an obvious fix, anyhow. i have a freebsd server wtih 2 jails on it:
[code
JID IP Address Hostname Path
1 10.80.0.3 web.com /usr/jail/web
2 10.80.0.2 dev.com /usr/jail/dev
[/code]
and I have a bridge set up to provide them with a network connection:

Code: 
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether de:fe:a1:b9:0c:c3
        inet 10.1.0.134 netmask 0xffffff00 broadcast 10.1.0.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:00:56:26
        inet 10.80.0.1 netmask 0xffffff00 broadcast 10.80.0.255
        inet 10.80.0.3 netmask 0xffffffff broadcast 10.80.0.3
        inet 10.80.0.2 netmask 0xffffffff broadcast 10.80.0.2
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.80.0.2 netmask 0xffffff00
        inet 10.80.0.3 netmask 0xffffff00
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x4
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

so, I have apache24 one the jails, but have just realised that they cannot communicate with outside networks, so I am unable to access the webpage, I'll share some of my config files, so hopefully that'll help.


this is the hosts rc.conf:

Code: 
ifconfig_vtnet0="inet 10.1.0.134 netmask 255.255.255.0"
defaultrouter="10.1.0.250" 
sshd_enable="YES" 
dumpdev="AUTO" 
zfs_enable="YES" 
jail_enable="YES" 
pf_enable="YES" 
gateway_enable="YES" 
apache24_enable="YES" 
cloned_interfaces="bridge0 lo1" 
ifconfig_bridge0="inet 10.80.0.1 netmask 255.255.255.0" 
ipv4_addrs_lo1="  10.80.0.2/24 10.80.0.3/24" iocage_enable="YES"

this is the hosts pf.conf:

Code: 
ext_if = "vtnet0"
bridge_if = "bridge0"

nat on $ext_if from $bridge_if:network to any -> ($ext_if)
rdr pass on $ext_if proto tcp from any to $ext_if port 2223 -> 10.80.0.2 port 2223 #dev
rdr pass on $ext_if proto tcp from any to $ext_if port 2224 -> 10.80.0.3 port 2224 #web

pass on $bridge_if all

hosts jail.conf:
Code: 
#devfs_ruleset = "2";
mount.devfs;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown jail";
allow.raw_sockets = "1";

dev {    
host.hostname = "dev.com";
ip4.addr = "10.80.0.2"; 
path = "/usr/jail/dev";
interface = "bridge0"; }

web {    
host.hostname = "web.com"; 
ip4.addr = "10.80.0.3"; 
path = "/usr/jail/web"; 
interface = "bridge0"; }

any help is much appreciated. let me know if you want to see any other files or anything like that
 
Your jails should be able to access the rest of your network (you are NATing the outgoing traffic).

tom_h said:
so I am unable to access the webpage
Click to expand...
This is incoming traffic, and you only redirected ports 2223 and 2224. Webservers typically run on ports 80 (HTTP) and/or 443 (HTTPS).
 
SirDice said:
Your jails should be able to access the rest of your network (you are NATing the outgoing traffic).


This is incoming traffic, and you only redirected ports 2223 and 2224. Webservers typically run on ports 80 (HTTP) and/or 443 (HTTPS).
Click to expand...
so, all i would have to is change those ports inside the pf.conf?
 
Dear tom_h,
I am no expert at all but I have played with vnet jails and normal ones. In my configurations I have had attached the jails either using lo1 or using a bridge and connecting the bridge by epair. You jail.conf shows that you use lo1.

What is the purpose of the bridge?
You have a bridge and lo1 with the same IP addresses. I am not sure if this is ok.
 
chrbr said:
Dear tom_h,
I am no expert at all but I have played with vnet jails and normal ones. In my configurations I have had attached the jails either using lo1 or using a bridge and connecting the bridge by epair. You jail.conf shows that you use lo1.

What is the purpose of the bridge?
You have a bridge and lo1 with the same IP addresses. I am not sure if this is ok.
Click to expand...
i have no idea, but everything breaks if the bridge isnt there
 
Oh, remove the bridge, you don't need it. Change this:
Code: 
cloned_interfaces="bridge0 lo1" 
ifconfig_bridge0="inet 10.80.0.1 netmask 255.255.255.0" 
ipv4_addrs_lo1="  10.80.0.2/24 10.80.0.3/24" 
iocage_enable="YES"
to:
Code: 
cloned_interfaces="lo1"
ifconfig_lo1="10.80.0.1 netmask 255.255.255.0"
Add interface = lo1; to your jail configurations. Your jails will attach to lo1 and will add their IP address when the jail starts. No need to already attach those addresses in rc.conf.

Change your pf.conf:
Code: 
jails_if = "lo1"

nat on $ext_if from $jails_if:network to any -> ($ext_if)
rdr pass on $ext_if proto tcp from any to $ext_if port 81 -> 10.80.0.2 port 80 #dev
rdr pass on $ext_if proto tcp from any to $ext_if port 80 -> 10.80.0.3 port 80 #web

Note that you cannot also have a webserver running on port 80 on the host. Port 80 will get redirected to the 'web' jail. Port 81 gets redirected to port 80 of 'dev'. You can only forward a port once.

Alternatively, instead of using redirections, you could install a reverse proxy (nginx, HAProxy, varnish, etc) on the host. And set up rules to switch to the correct backend based on the HTTP/1.1 Host headers. That will allow you to put multiple websites (and jails) on the same port.
 
tom_h said:
i have no idea, but everything breaks if the bridge isnt there
Click to expand...
May be because lo1 is not 100% configured. Below is a part of my /etc/rc.conf
Code: 
cloned_interfaces="lo1"
ifconfig_lo1="inet 10.0.0.254 netmask 255.255.255.0"
You may need to replace 10.0.0.254 by 10.80.0.1 for the host.
OK, SirDice has posted already :).
 
SirDice said:
Oh, remove the bridge, you don't need it. Change this:
Code: 
cloned_interfaces="bridge0 lo1"
ifconfig_bridge0="inet 10.80.0.1 netmask 255.255.255.0"
ipv4_addrs_lo1="  10.80.0.2/24 10.80.0.3/24"
iocage_enable="YES"
to:
Code: 
cloned_interfaces="lo1"
ifconfig_lo1="10.80.0.1 netmask 255.255.255.0"
Add interface = lo1; to your jail configurations. Your jails will attach to lo1 and will add their IP address when the jail starts. No need to already attach those addresses in rc.conf.

Change your pf.conf:
Code: 
jails_if = "lo1"

nat on $ext_if from $jails_if:network to any -> ($ext_if)
rdr pass on $ext_if proto tcp from any to $ext_if port 81 -> 10.80.0.2 port 80 #dev
rdr pass on $ext_if proto tcp from any to $ext_if port 80 -> 10.80.0.3 port 80 #web

Note that you cannot also have a webserver running on port 80 on the host. Port 80 will get redirected to the 'web' jail. Port 81 gets redirected to port 80 of 'dev'. You can only forward a port once.

Alternatively, instead of using redirections, you could install a reverse proxy (nginx, HAProxy, varnish, etc) on the host. And set up rules to switch to the correct backend based on the HTTP/1.1 Host headers. That will allow you to put multiple websites (and jails) on the same port.
Click to expand...
ok cool, thanks. just to double check, would that be my entire pf.conf file? if so, how would i then ssh into the jails? because before i was doing "ssh 10.1.0.134 -p 2223/2224" (using the ip of the host system)
 
SirDice said:
Just ssh to their IP address, on the host: ssh 10.80.0.2
Click to expand...
ok, im assuming i was meant to make some changes to the configs that you sent, but the jails just dont have any network connection at all now, completley unreachable.
 
Did you change the jail interface and did you add 10.80.0.1 to lo1 on the host? What is the current config that you have? Post again the output of ifconfig pf.conf and jail.conf
 
VladiBG said:
Did you change the jail interface and did you add 10.80.0.1 to lo1 on the host? What is the current config that you have? Post again the output of ifconfig pf.conf and jail.conf
Click to expand...
hosts ifconfig:
Code: 
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether de:fe:a1:b9:0c:c3
        inet 10.1.0.134 netmask 0xffffff00 broadcast 10.1.0.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:00:56:26
        inet 10.80.0.1 netmask 0xffffff00 broadcast 10.80.0.255
        inet 10.80.0.2 netmask 0xffffffff broadcast 10.80.0.2
        inet 10.80.0.3 netmask 0xffffffff broadcast 10.80.0.3
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.80.0.2 netmask 0xffffff00
        inet 10.80.0.3 netmask 0xffffff00
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x4
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

rc.conf:
Code: 
ifconfig_vtnet0="inet 10.1.0.134 netmask 255.255.255.0"
defaultrouter="10.1.0.250"
sshd_enable="YES"
dumpdev="AUTO"
zfs_enable="YES"
jail_enable="YES"
pf_enable="YES"
gateway_enable="YES"
apache24_enable="YES"
cloned_interfaces="bridge0 lo1"
ifconfig_bridge0="inet 10.80.0.1 netmask 255.255.255.0"
ipv4_addrs_lo1="10.80.0.2/24 10.80.0.3/24"
iocage_enable="YES"

pf.conf:

Code: 
ext_if = "vtnet0"
bridge_if = "bridge0"

nat pass on $ext_if from $bridge_if:network to any -> ($ext_if)

# Port forwarding rules for each jail
rdr pass on $ext_if proto tcp from any to $ext_if port 80 -> 10.80.0.2 port 80 # dev
rdr pass on $ext_if proto tcp from any to $ext_if port 80 -> 10.80.0.3 port 80 # web

rdr pass on $ext_if proto tcp from any to $ext_if port 2222 -> 10.80.0.3 port 2222 # SSH
rdr pass on $ext_if proto tcp from any to $ext_if port 2223 -> 10.80.0.2 port 2223 # SSH
pass in on $ext_if proto tcp from any to ($ext_if) port 22


jail.conf:
Code: 
#devfs_ruleset = "2";
mount.devfs;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown jail";
allow.raw_sockets = "1";

dev {
  host.hostname = "dev.com";
  ip4.addr = "10.80.0.2";
  path = "/usr/jail/dev";
  interface = "bridge0";
}

web {
  host.hostname = "web.com";
  ip4.addr = "10.80.0.3";
  path = "/usr/jail/web";
  interface = "bridge0";
}
 
i expected that you did those changes and wanted to see what did you miss. That's why i asked to share again the current configuration after the change.
 
VladiBG said:
i expected that you did those changes and wanted to see what did you miss. That's why i asked to share again the current configuration after the change.
Click to expand...
ah okay, i get you. my pf.conf was literally exactly what he put, my rc.conf was the same as what it was in the orginal post, except with those changes he suggested to get rid of the bridge interface. and my jail.conf was the same
 
After you remove the bridge you need to change the interface in jail.conf to point to lo1 instead to the bridge. That's why you lost the connection to the jails.
 
VladiBG said:
After you remove the bridge you need to change the interface in jail.conf to point to lo1 instead to the bridge. That's why you lost the connection to the jails.
Click to expand...
yeah.. i did that :/ sorry, shouldve mentioned in the previous response
 
You must log in or register to reply here.
Back
Top