PF Jails with NAT

Shadow53

Member

Reaction score: 19
Messages: 33

I'm trying to set up an Ampache media server at home and am taking the opportunity to learn how to use jails on FreeBSD. I'm trying to set up jails on a separate loopback network on the host and use the NAT features of PF to direct the traffic where it should go. I've tried following multiple different guides for this, except everything I can find refers to using ezjail and I want to do it using just jail.conf and pf.conf, if possible.

The problem is that the jails cannot access the internet. I have set up their resolv.conf files and - I thought - set up the shared network and the NAT rules to make it work. Here's my current setup:

/etc/rc.conf
Code:
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="faustus"
ifconfig_fxp0="DHCP"
ifconfig_fxp0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
ntpdate_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
kld_list="/boot/kernel/i915kms.ko"
cloned_interfaces="lo1"
#ifconfig_lo1="inet 10.0.1.1/24 netmask 255.255.255.0"
ipv4_addrs_lo1="10.0.1.1/24 10.0.1.2/24 10.0.1.3/24"
# Enabled packet forwarding between interfaces
gateway_enable="YES"
blacklistd_enable="YES"
jail_enable="YES"
pf_enable="YES"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

/etc/jail.conf
Code:
mount.devfs;
path="/usr/jails/$name";
host.hostname="$name.localdomain";
exec.clean;
exec.start="sh /etc/rc";
exec.stop="sh /etc/rc.shutdown";
exec.consolelog="/usr/jails/$name/var/log/jail.log";
interface="lo1";

nginx {
        ip4.addr="lo1|10.0.1.2";
}

ampache {
        ip4.addr="lo1|10.0.1.3";
}

/etc/pf.conf
Code:
ext_if="fxp0"
ext_net=$ext_if:network
jail_if="lo1"
jail_net=$jail_if:network

nginx_ports="{ http, https }"
nginx_ip="10.0.1.2"

# Don't filter loopback
set skip on lo0
set loginterface lo1
set block-policy return
set fail-policy  return

# Sanitize incoming data
scrub in on $ext_if all

# Route HTTP/S to nginx jail
#nat pass on $ext_if inet from $jail_net to any -> ($ext_if)
nat on $ext_if from $jail_net to any -> ($ext_if)
rdr on $ext_if proto tcp from any to ($ext_if) port $nginx_ports -> $nginx_ip

# Allow blacklistd to block stuff
#anchor "blacklistd/*" in on $ext_if
# Block incoming by default
#block in
# Allow outgoing by default
pass out
pass in
# Prevent spoofing attacks
#antispoof for $ext_if

# Allow traffic to/from jails
#pass in on $ext_if proto tcp from $ext_net to $jail_net port $nginx_ports keep state
#pass out on $jail_if proto tcp from $ext_net to $jail_net port $nginx_ports keep state
# Allow SSH
#pass in on $ext_if proto tcp from any to ($ext_if) port ssh
# Allow ICMP
#pass inet proto icmp from $jail_net to any keep state
pass in on $ext_if inet proto icmp to ($ext_if) icmp-type { unreach, redir, timex, echoreq }
pass in on $jail_if inet proto icmp to ($jail_if) icmp-type { unreach, redir, timex, echoreq }

Running tcpdump on pflog0 shows nothing when I attempt to use the host's pkg to install packages in a jail. The host has internet access. The jails' rc.conf files only disable sendmail and prevents syslogd from communicating on the network. The jails properly receive their IP address and hang when they attempt to access the internet, followed eventually by "No address record" for requests with domain names or "Operation timed out" for requests using IP addresses.

I'm new to FreeBSD and PF, so I'm not sure where to go from here. It seems like PF is blocking part of the request, response, or both, but I'm not sure how to check that or how I would fix it. Any help is appreciated.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,293
Messages: 38,810

Run tcpdump(1) on the $ext_if interface, then initiate some outgoing traffic from one of the jails. Check if there's actually anything going out and if the source address is correctly NAT'ed. Using tcpdump(1) allows you to look at the actual packets, invaluable tool when dealing with firewalls or connection problems.
 

gkontos

Daemon

Reaction score: 488
Messages: 2,160

If you want to monitor the pflog0 then you need to use[ FILE] log[/FILE] in your statements :


Code:
pass out log
pass in log
 

sko

Aspiring Daemon

Reaction score: 394
Messages: 704

I'm usually using the default loopback interface (lo0) for jails on hosts with only one IP, but besides that my NAT/rdr rules look the same, so I'm not fully convinced it is a filter/rdr/NAT problem.


Usually you don't pre-set the jails IP in the rc.conf of the host, at least that's what I've done on several jailhosts with NAT and one IP.
I only set one IP on that interface via the rc.conf that isn't connected to any jail. This IP is used by the host to reach the associated subnet on that interface. This IP is also used as the default GW by the jails.
The jail IPs are added to the interface when the jails are started.

Make sure your jails have a default route (=the hosts IP on the same loopback IF; e.g. 10.0.1.1) and resolver configured and reachable.

Then try to ping to/from the host and/or a separate host in the same network. If you still don't get any traffic from/to the jails, tcpdump on $ext_if as well as lo0 (and on the outside host you are pinging to/from), so you can find out where packets are either blocked or not properly routed and they are lost.
If ICMP gets the full way in one direction but no answers are generated, also look at ARP traffic and if requests/replies are transmitted the full way in both directions.
 
OP
S

Shadow53

Member

Reaction score: 19
Messages: 33

Again, I'm new to this sort of thing, so I couldn't really tell how to read the tcpdump output. It looks like the jail is successfully sending requests over $ext_if but isn't receiving any response. It queries the first DNS server for a domain, then the second server, then the first again for a different domain... I was attempting to install a package in the jail, so it was querying multiple pkg servers.

I was able to ping the host from the jail and the jail from the host. I was also able to ping 1.1.1.1 from the host, though attempting to ping a domain caused the same problem as the above paragraph.

Looks like the DNS server is responding to the host, but the host isn't properly directing the response to the jail. I tried a naive attempt to fix this by adding the following line to pf.conf:

Code:
rdr on $ext_if proto { tcp, udp } from any to ($ext_if) port domain -> ($jail_if)

That didn't work. I'm pretty sure that's not the correct way to redirect DNS response packets, but I thought I'd try. I feel like I shouldn't need to redirect response packets though, since PF is a stateful firewall. If I'm understanding how that works, I shouldn't need to do anything about the response packets, right?
 

gkontos

Daemon

Reaction score: 488
Messages: 2,160

Try replacing this:

nat on $ext_if from $jail_net to any -> ($ext_if) to this:
nat on $ext_if from 10.0.1.2 to any -> ($ext_if)

Assuming that 10.0.1.2 is the IP address of the jail
 
OP
S

Shadow53

Member

Reaction score: 19
Messages: 33

I decided to try spinning up a Digital Ocean droplet with FreeBSD and set up jails there. It looks like being on a home network with a NAT-ing router is causing some issues, as I have functional jail networking on the droplet but not on the home server. Both are using nearly identical configurations, with the only difference being that I can't hardcode the ip address for the home server.

I think I have an ebook copy of the Book of PF somewhere, so I might try reading through that and see if I'm missing something...
 

gutiersa

Member

Reaction score: 8
Messages: 97

May I ask if this was ever solved?
BTW, here is my 2 cents:
When my ISP provides me with a static IP address, I am not able to connect to my server from my home LAN computers. My server is a home server.
Hence the problem here may have been related to an ISP issue?
regards
 
OP
S

Shadow53

Member

Reaction score: 19
Messages: 33

It was not solved. As the post above yours says, I "solved" it by using a VPS hosted elsewhere. That's not to say it can't be solved, just that I didn't. It may be easier to use something like bastille or iocage than jail.conf directly.
 

gutiersa

Member

Reaction score: 8
Messages: 97

It was not solved. As the post above yours says, I "solved" it by using a VPS hosted elsewhere. That's not to say it can't be solved, just that I didn't. It may be easier to use something like bastille or iocage than jail.conf directly.
I am using iocage now. However, Is it possible to not require a static IP address? In my neck of the woods it gets quite pricey.
 
OP
S

Shadow53

Member

Reaction score: 19
Messages: 33

Either of the options in the iocage docs should work, since the jail IP addresses are separate from the host IP(s). "Shared IP" is arguably easier; just choose an IP address range that's one of the ones reserved for private use and is not the same range your router assigns from. VNET should work too, but for simple setups the shared IP option works fine.
 

gutiersa

Member

Reaction score: 8
Messages: 97

Either of the options in the iocage docs should work, since the jail IP addresses are separate from the host IP(s). "Shared IP" is arguably easier; just choose an IP address range that's one of the ones reserved for private use and is not the same range your router assigns from. VNET should work too, but for simple setups the shared IP option works fine.

So, if my ISP router address is 194.168.1.1, and I give my server address 192.168.1.40, then my jails should not be in the 192.168.1.0/24 range?
Thanks for replying.
Also, above, you did not need to give your jail an IP address on the server's network card, you only needed to give them an address on the lo1 clone?
 
OP
S

Shadow53

Member

Reaction score: 19
Messages: 33

So, if my ISP router address is 194.168.1.1, and I give my server address 192.168.1.40, then my jails should not be in the 192.168.1.0/24 range?
Thanks for replying.
Also, above, you did not need to give your jail an IP address on the server's network card, you only needed to give them an address on the lo1 clone?

Correct. You want the jail IP range to be something different, to avoid conflicts. And you only need addresses on the loopback interface. The jails will get access to internet by using NAT rules in your preferred firewall program.
 

wolffnx

Aspiring Daemon

Reaction score: 231
Messages: 677

Could I just make my jails get an address from the ISP's router?
Hi , I'dont use Iocage but here is something that can help you, only you need start from scratch

Ezjail from the Handbook

I'ts works for me and not have to Nat nothing in the firewall and using the Lan range from the host(your Isp router in your case)
 

Neubert

Member

Reaction score: 30
Messages: 55

I am using iocage now
[...]
Could I just make my jails get an address from the ISP's router?

If your ISP router is providing reliable firewall protection from the outside world and is giving out IP addresses to your home machines, then you can use iocage to create jails that get their IP addresses from the router. The jails will appear on the local network just like any other machine. In my case, I'm not running a firewall on the FreeBSD jail host, but am instead relying on the router to provide that protection. If I ever need to access a jail from the outside world, I'll either forward a port on the router, use a reverse proxy or install a VPN. If I ever need to segregate internal traffic for untrusted guest or IOT devices, they'll go on a separate VLAN defined on the switch with firewall rules defined on the router. For basic needs in a home environment, I think relying on the router and switch is probably good enough.

I hope the experts will jump in to correct my mistakes because I don't yet understand all of the trade-offs, but here's how I create DHCP jails with iocage.

Bridge the host interface (bge0 for my hardware):

# cat /etc/rc.conf
[...]
iocage_enable="YES"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm bge0 up"


Enable ip forwarding and disable packet filtering on the bridge:

# cat /etc/sysctl.conf
[...]
net.inet.ip.forwarding=1 # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0 # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0 # Packet filter on the bridge interface
net.link.bridge.pfil_member=0 # Packet filter on the member interface


Create an iocage jail with the bpf, dhcp, and defaultrouter parameters:

iocage create -n "j0" -r latest vnet="on" allow_raw_sockets="1" boot="on" bpf="yes" dhcp="on" defaultrouter="192.168.0.1"

Use iocage set to change those properties on existing jails if needed.

Here are the complete instructions explaining how I have been doing this for my own jails. I wish I had been able to find a complete example for this when I was starting out, so I hope this helps.
 

gutiersa

Member

Reaction score: 8
Messages: 97

I did try ezjail initially. However, I was not able to make it work. Apparently the port is not actively maintained.
That is why I changed to iocage.
I wouldn't mind starting over.
 

gutiersa

Member

Reaction score: 8
Messages: 97

If your ISP router is providing reliable firewall protection from the outside world and is giving out IP addresses to your home machines, then you can use iocage to create jails that get their IP addresses from the router. The jails will appear on the local network just like any other machine. In my case, I'm not running a firewall on the FreeBSD jail host, but am instead relying on the router to provide that protection. If I ever need to access a jail from the outside world, I'll either forward a port on the router, use a reverse proxy or install a VPN. If I ever need to segregate internal traffic for untrusted guest or IOT devices, they'll go on a separate VLAN defined on the switch with firewall rules defined on the router. For basic needs in a home environment, I think relying on the router and switch is probably good enough.

I hope the experts will jump in to correct my mistakes because I don't yet understand all of the trade-offs, but here's how I create DHCP jails with iocage.

Bridge the host interface (bge0 for my hardware):

# cat /etc/rc.conf
[...]
iocage_enable="YES"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm bge0 up"


Enable ip forwarding and disable packet filtering on the bridge:

# cat /etc/sysctl.conf
[...]
net.inet.ip.forwarding=1 # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0 # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0 # Packet filter on the bridge interface
net.link.bridge.pfil_member=0 # Packet filter on the member interface


Create an iocage jail with the bpf, dhcp, and defaultrouter parameters:

iocage create -n "j0" -r latest vnet="on" allow_raw_sockets="1" boot="on" bpf="yes" dhcp="on" defaultrouter="192.168.0.1"

Use iocage set to change those properties on existing jails if needed.

Here are the complete instructions explaining how I have been doing this for my own jails. I wish I had been able to find a complete example for this when I was starting out, so I hope this helps.
Yes, I did do this. Yes, a tutorial would be great. Perhaps that could be a project for me. I would like to contribute to the handbook.

So...The interface for the jail is the bridge, not the card from the Freebsd server where the jail is?
 

Neubert

Member

Reaction score: 30
Messages: 55

So...The interface for the jail is the bridge, not the card from the Freebsd server where the jail is?

Well, I guess you could say that, but there's more to it when using iocage. The bridge is virtual and is bound to the network card's interface. Then, iocage automatically creates a new VNET for each jail as it starts (vnet0.0-vnet0.n) and binds each of those new VNETs to the bridge as members. Inside each jail, iocage automatically creates a virtual NIC (epair0b) to give the jail a virtual network card connected to its VNET.

After you set up the bridge one time, iocage creates everything else it needs for each new jail as it starts. If you stop or delete the jail, iocage cleans everything up for you. I believe it assumes "bridge0" as the default name and creates as many "vnet0.x" as needed to match. If you need more than one bridge, you'll have to specify that for each jail using iocage set.

Follow the instructions in that link to get a working test jail in a few minutes. I might be wrong, but it does seem to work.

Run ifconfig on the host to see the interfaces created by iocage while the jails are running.

Code:
# ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:25:90:15:99:3c
        inet 192.168.1.107 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:25:90:15:99:3d
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:f4:df:4a:b0:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: j0 as nic: epair0b
        options=8<VLAN_MTU>
        ether 28:92:4a:07:ab:93
        hwaddr 02:a5:43:84:ea:0a
        inet6 fe80::2a92:4aff:fe07:ab93%vnet0.3 prefixlen 64 scopeid 0x5
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vnet0.4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: j1 as nic: epair0b
        options=8<VLAN_MTU>
        ether 28:92:4a:74:ca:1d
        hwaddr 02:2b:00:38:91:0a
        inet6 fe80::2a92:4aff:fe74:ca1d%vnet0.4 prefixlen 64 scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 

gutiersa

Member

Reaction score: 8
Messages: 97

Well, I guess you could say that, but there's more to it when using iocage. The bridge is virtual and is bound to the network card's interface. Then, iocage automatically creates a new VNET for each jail as it starts (vnet0.0-vnet0.n) and binds each of those new VNETs to the bridge as members. Inside each jail, iocage automatically creates a virtual NIC (epair0b) to give the jail a virtual network card connected to its VNET.

After you set up the bridge one time, iocage creates everything else it needs for each new jail as it starts. If you stop or delete the jail, iocage cleans everything up for you. I believe it assumes "bridge0" as the default name and creates as many "vnet0.x" as needed to match. If you need more than one bridge, you'll have to specify that for each jail using iocage set.

This is what I get from ifconfig on the box itself:

Code:
emg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether xx:xx:xx:xx:xx:xx
        inet 192.168.10.40 netmask 0xffffff00 broadcast 192.168.10.255
        inet 192.168.10.15 netmask 0xffffff00 broadcast 192.168.10.255
        inet 192.168.10.19 netmask 0xffffff00 broadcast 192.168.10.255
        inet 192.168.10.18 netmask 0xffffff00 broadcast 192.168.10.255
        inet 192.168.10.9 netmask 0xffffff00 broadcast 192.168.10.255
        inet6 xxxx::xxx:xxxx:xxxx:xxxx%emg0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::x prefixlen 128
        inet6 xxxx::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether xx:xx:xx:xx:xx:xx
        id xx:xx:xx:xx:xx:xx priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id xx:xx:xx:xx:xx:xx priority 32768 ifcost 0 port 0
        member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: emg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=1<PERFORMNUD>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: j0 as nic: epair0b
        options=8<VLAN_MTU>
        ether xx:xx:xx:xx:xx:xx
        hwaddr xx:xx:xx:xx:xx:xx
        inet6 xxxx::xxx:xxxx:xxxx:xxxx%vnet0.5 prefixlen 64 scopeid 0x5
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I have I few jails which I set up with iocage. Then I did follow the tutorial mentioned above to set up J0. I still cannot see the server which I set up in 192.168.10.9. Not sure if it is my network configuration, or an Nginx problem.
 

Neubert

Member

Reaction score: 30
Messages: 55

I have I few jails which I set up with iocage. Then I did follow the tutorial mentioned above to set up J0. I still cannot see the server which I set up in 192.168.10.9. Not sure if it is my network configuration, or an Nginx problem.

1. What is the output of iocage list?

Code:
# iocage list
+-----+---------+-------+--------------+------+
| JID |  NAME   | STATE |   RELEASE    | IP4  |
+=====+=========+=======+==============+======+
[...]

2. If you call iocage exec j0 ifconfig to get the IP address assigned to j0 (192.168.1.111 in the example below), can you see the DHCP reservation for that IP on the router's admin page? Can you ping that IP from another computer on the LAN? If the answer to those is yes, at least we'll know you can create jails using that method and there's nothing currently preventing j0 from reaching the outside.

Code:
# iocage exec j0 ifconfig
[...]
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
[...]
        inet 192.168.1.111 netmask 0xffffff00 broadcast 192.168.1.255
[...]

3. I read the thread again and I think I understand that you want to ping 192.168.10.9 from another machine on the LAN. Is that correct? You said you "can't see it" but I'm not sure what that means? See it from where?

4. Also, is 192.168.10.9 another jail on the same host that you set up by hand or using ezjail? Or is it something else?
 

michael_hackson

Active Member

Reaction score: 106
Messages: 164

Could I just make my jails get an address from the ISP's router?

Generally that is not supported by the ISP I think but you can however create aliases of the IP given to your server/station in /etc/rc.conf
Code:
ifconfig_emg0="inet 192.168.10.40 netmask 255.255.255.0"
ifconfig_emg0_alias0="inet 192.168.10.15 netmask 255.255.255.255"
ifconfig_emg0_alias1="inet 192.168.10.19 netmask 255.255.255.255"
ifconfig_emg0_alias2="inet 192.168.10.18 netmask 255.255.255.255"
ifconfig_emg0_alias3="inet 192.168.10.9 netmask 255.255.255.255"

There is a shorter way of doing this but this is what worked for me. I don't know about iocage but I guess they are similar setups. Traditionally jails are masked with 0xffffffff but as of recently it must not be this way.

I did try ezjail initially. However, I was not able to make it work. Apparently the port is not actively maintained.
That is why I changed to iocage.
I wouldn't mind starting over.

If you don't mind starting over you can have a look at Scottro's guide of how he sets up multiple jails. I think it's great because it's easy to follow and it works.
http://srobb.net/nullfsjail.html

Bear in mind to read carefully since some parts are no longer used. The most recent is the section for FreeBSD-10.

In short this method does not use ezjail nor iocage and NATing is not needed in /etc/pf.conf since the aliases are used as any regular IP given from ISP because they are given from your host.
 

gutiersa

Member

Reaction score: 8
Messages: 97

1. What is the output of iocage list?

Code:
# iocage list
+-----+---------+-------+--------------+------+
| JID |  NAME   | STATE |   RELEASE    | IP4  |
+=====+=========+=======+==============+======+
[...]

2. If you call iocage exec j0 ifconfig to get the IP address assigned to j0 (192.168.1.111 in the example below), can you see the DHCP reservation for that IP on the router's admin page? Can you ping that IP from another computer on the LAN? If the answer to those is yes, at least we'll know you can create jails using that method and there's nothing currently preventing j0 from reaching the outside.

Code:
# iocage exec j0 ifconfig
[...]
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
[...]
        inet 192.168.1.111 netmask 0xffffff00 broadcast 192.168.1.255
[...]

3. I read the thread again and I think I understand that you want to ping 192.168.10.9 from another machine on the LAN. Is that correct? You said you "can't see it" but I'm not sure what that means? See it from where?

4. Also, is 192.168.10.9 another jail on the same host that you set up by hand or using ezjail? Or is it something else?
+-----+--------------+-------+--------------+--------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+==============+=======+==============+==============+
| - | basejail | down | 12.1-RELEASE | - |
+-----+--------------+-------+--------------+--------------+
| - | bhcnj.net | down | 12.1-RELEASE | 192.168.10
Code:
+-----+--------------+-------+--------------+--------------+
| JID |     NAME     | STATE |   RELEASE    |     IP4      |
+=====+==============+=======+==============+==============+
| -   | basejail     | down  | 12.1-RELEASE | -            |
+-----+--------------+-------+--------------+--------------+
| -   | j1           | down  | 12.1-RELEASE | 192.168.10.13 |
+-----+--------------+-------+--------------+--------------+
| -   | j2           | down  | 12.1-RELEASE | -            |
+-----+--------------+-------+--------------+--------------+
| -   | empty        | down  | EMPTY        | -            |
+-----+--------------+-------+--------------+--------------+
| 5   | j0           | up    | 12.1-RELEASE | DHCP         |
+-----+--------------+-------+--------------+--------------+
| -   | j3           | down  | 12.1-RELEASE | -            |
+-----+--------------+-------+--------------+--------------+
| -   | dbjail       | down  | 12.1-RELEASE | 192.168.10.12 |
+-----+--------------+-------+--------------+--------------+
| -   | serverjail   | down  | 12.1-RELEASE | 192.168.10.57 |
+-----+--------------+-------+--------------+--------------+
| -   | dbjail2      | down  | 12.1-RELEASE | 192.168.10.80 |
+-----+--------------+-------+--------------+--------------+
| 11  | server2      | up    | 12.1-RELEASE | 192.168.10.9  |
+-----+--------------+-------+--------------+--------------+
| -   | j4           | down  | 12.1-RELEASE | -            |
+-----+--------------+-------+--------------+--------------+
| 1   | db3          | up    | 12.1-RELEASE | 192.168.10.15 |
+-----+--------------+-------+--------------+--------------+
| 4   | app          | up    | 12.1-RELEASE | 192.168.10.18 |
+-----+--------------+-------+--------------+--------------+
| -   | temp         | down  | 12.1-RELEASE | 192.168.10.20 |
+-----+--------------+-------+--------------+--------------+
| -   | thickjail    | down  | 12.1-RELEASE | -            |
+-----+--------------+-------+--------------+--------------+
| 2   | j5           | up    | 12.1-RELEASE | 192.168.10.19 |
+-----+--------------+-------+--------------+--------------+

This is the output of "iocage list" I can ping from other machines in the network and I can ping other jails from one jail. My issue is with name resolution. I can access the site by using the IP address, but I cannot access the site with its domain name. All jails were created with iocage. Also, the appropriate ports are open in the firewall.
 

Neubert

Member

Reaction score: 30
Messages: 55

My issue is with name resolution. I can access the site by using the IP address, but I cannot access the site with its domain name. All jails were created with iocage. Also, the appropriate ports are open in the firewall.

Okay, that's good info. You can currently create jails in several ways, they can communicate with other machines on the LAN using IP but not with names.

You need some kind of DNS for your local network to get names. I have never set up a DNS server, but maybe someone else will have a suggestion for that. I bet most people just rely on IP for their home networks. You could create a hosts file on each machine to give you aliases, but you'll quickly get tired of trying to keep them in sync. There might be alternate ways to get names, but I'm not familiar with them.

Your router's admin page might include a static DNS mapping table that you can fill out by hand to identify each machine in one central location:

Code:
Host name: server2.gutiersa.com
Alias: server2
IP address: 192.168.10.9
 
Top