Multiple NIC's and Jails

[DrH]

Greetings!
My FreeBSD box has 2 NIC's installed. I want to use one of the NIC's for the box itself, and dedicate the other one solely to the jails I have created. One of the jails will run a webserver and I want it to be on a different network from the base operating system. To create the jails, I have followed this link, exactly:
http://www.bsdnow.tv/tutorials/jails
I have struggled with this. After months of Googling I have two questions:

1. I think I know how to bind both cards to the base OS. Here are my settings in /etc/rc.conf:

ifconfig_re0="DHCP"
hostname="freebsd"
ifconfig_re0_alias0="inet 172.20.20.230 netmask 0xffffff00 broadcast 172.20.20.255"
ifconfig_re0_alias1="inet 172.20.20.231 netmask 0xffffff00 broadcast 172.20.20.255"
ifconfig_re0_alias2="inet 172.20.20.232 netmask 0xffffff00 broadcast 172.20.20.255"
ifconfig_re0_alias3="inet 172.20.20.233 netmask 0xffffff00 broadcast 172.20.20.255"
.
.
.

ifconfig_re1="inet 172.20.40.32 netmask 0xfffffff00 broadcast 172.20.40.255"
ifconfig_re1_alias0="inet 172.20.40.230 netmask 0xffffff00 broadcast 172.20.40.255" <--- used for the jail

Those aliases on the 172.20.20.x addresses are the jails I would like to be running on a different subnet, like 172.20.40.x, but I can't make it work.

So the first NIC, re0, is used by the main operating system in which the jails are running, and has the IP 172.20.20.32. Everything works great with that. I can install jails and give them IPs on that subnet, as shown by the alias lines above, and everything works great. I have the other IP assigned to 172.20.40.32, and it works great for connecting to the base OS. But if I have a jail that I want to put on a different subnet, such as 172.20.40.0/24, I can do that, and the jail runs, and I can connect to it via SSH, but the jail itself can't connect to the internet, only the local network.

The two NICs are each connected to a pFsense firewall. I know the NICs both function because I can unplug one and access the box from the other at the designated IP address. How do I use multiple NICs and assign them only to a specific jail (and have it work!)?

2. Once I figure out #1, how do I ensure that a particular NIC can only be used for the jail? For example, I noticed that if the jail were not running, for example, 172.20.20.230, and I tried to SSH into that jail using that IP, the SSH session would default to the main FreeBSD box and I would get a login prompt there. How do I keep it from doing that?

Thanks in advance!

 

[SirDice]

Remove all the aliases, a properly set up jail.conf will automatically add/remove the correct alias to an interface when the jail starts/stops. See the interface and ipv4.addr options in jail(8).

 

[DrH]

OK - tried that, didn't work. I commented out the alias files from /etc/rc.conf:

ifconfig_re0="DHCP"
hostname="freebsd"
#ifconfig_re0_alias0="inet 172.20.20.230 netmask 0xffffff00 broadcast 172.20.20.255"
#ifconfig_re0_alias1="inet 172.20.20.231 netmask 0xffffff00 broadcast 172.20.20.255"
#ifconfig_re0_alias2="inet 172.20.20.232 netmask 0xffffff00 broadcast 172.20.20.255"
#ifconfig_re0_alias3="inet 172.20.20.233 netmask 0xffffff00 broadcast 172.20.20.255"

Perhaps I should mention that tutorial uses ezjails to create the jails.

I did re-read the jail(8) man page and made my /etc/jail.conf file to look like this:

webjail {
        path = /usr/jails/webjail;
        mount.devfs;
        devfs_ruleset = 4;
        host.hostname = webjail.madbiochemist.com;
        ip4.addr = 172.20.20.233;
        interface = re0;
        exec.start = "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown";
}

# Output of jls command:
# JID  IP Address      Hostname                      Path
#     2  172.20.20.233   webjail                       /usr/jails/webjail

The jail starts, but it has zero network connectivity. What have I done wrong?

Keeping everything else the same as you see it but uncommenting the alias line in /etc/rc.conf allows me to access the jails again.

 

[kpa]

You need to to give the address in this form:

ip4.addr = re0|172.20.20.233/24;

This will tell jail(8) to create the additional address on re0 at jail startup.

 

[asteriskRoss]

@DrH, if, as you indicated, you are using sysutils/ezjail, for your jail management, note that it includes its own configuration files and doesn't need /etc/jail.conf. You need to configure:

• /usr/local/etc/ezjail.conf
• /usr/local/etc/ezjail/<jail_name> (initially created for you and some options can be configured through ezjail-admin commands)

You should also add to /etc/rc.conf:

ezjail_enable="YES"

Understanding how one administers jails without sysutils/ezjail is useful, but if you are planning to use it then mixing manual jail configuration with ezjail configuration is just likely to confuse you to begin with. I've been using ezjail for a while and can highly recommend it.

 

[DrH]

You need to to give the address in this form:

ip4.addr = re0|172.20.20.233/24;

This will tell jail(8) to create the additional address on re0 at jail startup.

Tried that - situation not improved.

@DrH, if, as you indicated, you are using sysutils/ezjail, for your jail management, note that it includes its own configuration files and doesn't need /etc/jail.conf. You need to configure:

• /usr/local/etc/ezjail.conf
• /usr/local/etc/ezjail/<jail_name> (initially created for you and some options can be configured through ezjail-admin commands)

You should also add to /etc/rc.conf:

ezjail_enable="YES"

Understanding how one administers jails without sysutils/ezjail is useful, but if you are planning to use it then mixing manual jail configuration with ezjail configuration is just likely to confuse you to begin with. I've been using ezjail for a while and can highly recommend it.

Yes, I am using ezjail. The problem isn't starting and administering jails - ezjail helps make that simple. The problem is that I can't figure out how to do that so that the jails have addresses on the 172.20.40.x/24 while the main box has an IP on the 172.20.20.x/24. I know that the other NIC in the main box needs to have an IP on the 172.20.40.x/24 network, and be able to communicate with the outside world. It seems to make sense to me that i should be able to give the second NIC (re1) its own IP, then make aliases off of that and assign them to jails the same way I did for the jails that were on the 172.20.20.x/24 subnet, but when I do that I have no internet connectivity by the jails.

 

[asteriskRoss]

@DrH, I agree that what you're trying to do should be possible. If I have understood you correctly, you would like to attach the jails to the re1 interface with IP addresses on the 172.20.40.0/24 subnet.

Assuming that you want to use sysutils/ezjail, make sure for /etc/rc.conf that you include:

ezjail_enable="YES"

and do not include:

jail_enable="YES" # PLEASE REMOVE -- Not needed when using only ezjail administered jails

It is actually possible to legitimately use both but certainly not for the same jails and I would be keener to see you happily up and running with sysutils/ezjail before going off-piste

The interface value should allow you to specify the network interface that should receive the required alias IP address, as you have identified in the /etc/jail.conf you posted. When using sysutils/ezjail you should add the following line to your jail's configuration file, /usr/local/etc/ezjail/webjail:

export jail_webjail_interface="re1"

There should already be a line in the configuration file for the jail's IP address so check this is the value you wanted; the ezjail script actually just passes these values to the regular jail scripts. As @SirDice said, the aliases should be added and removed automatically by the jail startup scripts, so there should be no need to configure them manually. I wonder whether you might previously have experienced some interaction with using DHCP to configure re0 that meant you needed to set up aliases manually?

You mentioned you haven't had Internet connectivity from within the jails. What did you do to test this? I ask because by default, ping cannot be used from inside jails (since raw sockets are disabled). Did you double check that /etc/resolv.conf within the jail was pointing to the right DNS server (that is, the one that is correct for the re1 interface? Could you post the output of ifconfig -a on the host once your jails have started as this should show all IP addresses (including aliases) for each interface?

 

[DrH]

Here is what I've done: As a little experiment, I decided to give the re1 interface a static IP on the 172.20.20.0/24 subnet. I added the following to /etc/rc.conf:

#ifconfig_re0="DHCP"
ifconfig_re0="inet 172.20.20.32 netmask 0xffffff00 broadcast 172.20.20.0"
ifconfig_re1="inet 172.20.20.33 netmask 0xffffff00 broadcast 172.20.20.0"
hostname="freebsd"
defaultrouter="172.20.20.1"

I removed all of the references to aliases from /etc/rc.conf. I added the interface line to /usr/local/etc/ezjail/webjail:

# To specify the start up order of your ezjails, use these lines to
# create a Jail dependency tree. See rcorder(8) for more details.
#
# PROVIDE: standard_ezjail
# REQUIRE:
# BEFORE:
#

export jail_webjail_interface="re1"
export jail_webjail_hostname="webjail"
export jail_webjail_ip="172.20.20.233"
export jail_webjail_rootdir="/usr/jails/webjail"
export jail_webjail_exec_start="/bin/sh /etc/rc"
export jail_webjail_exec_stop=""
export jail_webjail_mount_enable="YES"
export jail_webjail_devfs_enable="YES"
export jail_webjail_devfs_ruleset="devfsrules_jail"
export jail_webjail_procfs_enable="YES"
export jail_webjail_fdescfs_enable="YES"
export jail_webjail_image=""
export jail_webjail_imagetype=""
export jail_webjail_attachparams=""
export jail_webjail_attachblocking=""
export jail_webjail_forceblocking=""
export jail_webjail_zfs_datasets=""
export jail_webjail_cpuset=""
export jail_webjail_fib=""
export jail_webjail_parentzfs=""
export jail_webjail_parameters=""
export jail_webjail_post_start_script=""

Here is are the contents of /etc/resolv.conf within the webjail:

# Generated by resolvconf
search madbiochemist.com
nameserver 172.20.20.1

After a reboot, I can access the webjail and install packages:

root@webjail:~ # pkg search nano
nano-2.2.6
nano-devel-2.3.3
nanoblogger-3.4.2_1
nanoblogger-extra-3.4.2
rubygem-nanoc-3.6.9
root@webjail:~ #

I could not do that before (more on that later...) From the output of ifconfig -a, it looks as if the alias for the webjail has been assigned properly on re1:

# ifconfig -a
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 00:24:1d:74:5a:56
        inet 172.20.20.32 netmask 0xffffff00 broadcast 172.20.20.0
        inet6 fe80::224:1dff:fe74:5a56%re0 prefixlen 64 scopeid 0x1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 00:a1:b0:8c:00:44
        inet 172.20.20.33 netmask 0xffffff00 broadcast 172.20.20.0
        inet6 fe80::2a1:b0ff:fe8c:44%re1 prefixlen 64 scopeid 0x3
        inet 172.20.20.233 netmask 0xffffffff broadcast 172.20.20.233
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

It looks like I have successfully routed all jail traffic through the re1 interface. That is a step in the right direction! As a second experiment, I decided to install a third NIC into the box (rl0) and set it up exactly the same way, giving it an IP on the 172.20.40.0/24 subnet. After setting up the new NIC, /etc/rc.conf now looks like this:

# 
#ifconfig_re0="DHCP"
ifconfig_re0="inet 172.20.20.32 netmask 0xffffff00 broadcast 172.20.20.0"
hostname="freebsd"
defaultrouter="172.20.20.1"

# ssh
sshd_enable="YES"

# ntp
ntpd_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

# Enable ZFS
zfs_enable="YES"

# Enable PlexMediaServer
plexmediaserver_enable="YES"

# Enable SAMBA
samba_enable="YES"

# For NFS file sharing
mountd_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-a -t -n 4"
mountd_flags="-r"
nfs_client_enable="YES"

# For NFSv4 file sharing
nfsv_server_enable="YES"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"

# Enable ezjails
ezjail_enable="YES"

# Enable Jails - may not need this if using ezjails
#jail_enable="YES"

# Set up the second NIC on the system, re1
# Uncomment this line if using DHCP
#ifconfig_re1="DHCP"

# Comment the following line if using DHCP
ifconfig_re1="inet 172.20.20.33 netmask 0xffffff00 broadcast 172.20.20.0"

# Set up the third NIC, rl0
ifconfig_rl0="inet 172.20.40.32 netmask 0xffffff00 broadcast 172.20.40.0"

# allow the box to act like a router and pass packets back and forth between re1 and re0?
gateway_enable="YES"

Now, I created a new jail, called testjail:

#ezjail-admin create testjail 172.20.40.230

I then copied the /etc/resolv.conf file from the host machine and pointed it to what I think is the correct address for DNS:

#cp /etc/resolv.conf /usr/jails/testjail/etc/resolv.conf

I changed the nameserver IP from 172.20.20.1 to 172.20.40.1, to look like this:

# Generated by resolvconf
search mydomain.com
nameserver 172.20.40.1

I also changed the config file found in /usr/local/etc/ezjail/ for testjail to tell it to use the correct interface:

# To specify the start up order of your ezjails, use these lines to
# create a Jail dependency tree. See rcorder(8) for more details.
#
# PROVIDE: standard_ezjail
# REQUIRE:
# BEFORE:
#

export jail_testjail_interface="rl0"
export jail_testjail_hostname="testjail"
export jail_testjail_ip="172.20.40.230"
export jail_testjail_rootdir="/usr/jails/testjail"
export jail_testjail_exec_start="/bin/sh /etc/rc"
export jail_testjail_exec_stop=""
export jail_testjail_mount_enable="YES"
export jail_testjail_devfs_enable="YES"
export jail_testjail_devfs_ruleset="devfsrules_jail"
export jail_testjail_procfs_enable="YES"
export jail_testjail_fdescfs_enable="YES"
export jail_testjail_image=""
export jail_testjail_imagetype=""
export jail_testjail_attachparams=""
export jail_testjail_attachblocking=""
export jail_testjail_forceblocking=""
export jail_testjail_zfs_datasets=""
export jail_testjail_cpuset=""
export jail_testjail_fib=""
export jail_testjail_parentzfs=""
export jail_testjail_parameters=""
export jail_testjail_post_start_script=""

So now, there should be an alias created on rl0 interface for the jail, which there is:

# ifconfig -a
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 00:24:1d:74:5a:56
        inet 172.20.20.32 netmask 0xffffff00 broadcast 172.20.20.0
        inet6 fe80::224:1dff:fe74:5a56%re0 prefixlen 64 scopeid 0x1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2008<VLAN_MTU,WOL_MAGIC>
        ether 00:40:f4:ea:95:9b
        inet 172.20.40.32 netmask 0xffffff00 broadcast 172.20.40.0
        inet6 fe80::240:f4ff:feea:959b%rl0 prefixlen 64 scopeid 0x2
        inet 172.20.40.230 netmask 0xffffffff broadcast 172.20.40.230
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 00:a1:b0:8c:00:44
        inet 172.20.20.33 netmask 0xffffff00 broadcast 172.20.20.0
        inet6 fe80::2a1:b0ff:fe8c:44%re1 prefixlen 64 scopeid 0x3
        inet 172.20.20.233 netmask 0xffffffff broadcast 172.20.20.233
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The jail starts following a reboot:

# jls
   JID  IP Address      Hostname                      Path
     1  172.20.20.233   webjail                       /usr/jails/webjail
     2  172.20.40.230   testjail                       /usr/jails/testjail

I can reach a console on the testjail and try to install a package:

# ezjail-admin console testjail
root@testjail:~ # pkg update
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/freebsd:10:x86:64/latest, please wait...

And that is as far as it goes. It seems to me that what I'm doing works so long as the two interfaces are on the same subnet. I don't know if it matters, but I can ssh into the jail on the 172.20.40.0/24 subnet from the 172.20.20.0/24 subnet. I would assume 172.20.40.1 would be the correct nameserver to set for rl0 in /etc/resolv.conf of the testjail?

 

[asteriskRoss]

Whilst I applaud experimenting with technology and FreeBSD, I'm going to ignore the third interface (rl0) you have added as increasing complication makes it less likely that we'll resolve your issue quickly. I will assume your objective remains the same; to attach jails to the re1 interface with IP addresses on the 172.20.40.0/24 subnet with working DNS.

We have solved your immediate issue of assigning the IP of your jail to the interface you wanted. You asked:

I would assume 172.20.40.1 would be the correct nameserver to set for rl0 in /etc/resolv.conf of the test jail?

You need to tell us that Are you running a DNS server at 172.20.40.1? If not, I have a suspicion where your issue might be...

You have a working DNS server accessible at 172.20.20.1. Looking at your /etc/rc.conf, you have also enabled your machine to forward traffic between re0 and re1, which I assume is deliberate for reasons you haven't specified (though you also said you wanted your web server on a separate network). Rather than choosing an IP address and hoping there is a DNS server on the end of it for your jail, it would seem sensible to use the server you know exists; 172.20.20.1. With your configuration, I would expect that your machine will route requests to your DNS server.

Would you therefore try configuring your server with /etc/rc.conf containing your desired settings for the host (I've guessed these from your previous posts):

ifconfig_re0="inet 172.20.20.32 netmask 0xffffff00"
ifconfig_re1="inet 172.20.40.32 netmask 0xffffff00"

Also, try to configure your jail with the address you want (172.20.40.233) in /usr/local/etc/ezjail/webjail and configure the jail's /etc/resolv.conf to point to your working DNS server:

nameserver 172.20.20.1

Come back and let us know whether everything is working as you expect.

 

[DrH]

Are you running a DNS server at 172.20.40.1? If not, I have a suspicion where your issue might be...

You have a working DNS server accessible at 172.20.20.1. Looking at your /etc/rc.conf, you have also enabled your machine to forward traffic between re0 and re1, which I assume is deliberate for reasons you haven't specified (though you also said you wanted your web server on a separate network). Rather than choosing an IP address and hoping there is a DNS server on the end of it for your jail, it would seem sensible to use the server you know exists; 172.20.20.1. With your configuration, I would expect that your machine will route requests to your DNS server.

I think so. If I unplug the host machine from the 172.20.20.0/24 network entirely, it accesses the internet just fine through the 172.20.40.0/24 network, so I would assume there must be a nameserver there. Now, in my pFsense firewall, I have it set to use the pFsense as the nameserver, and each network by default uses the same nameserver, if I read it correctly. I think the fact that I can connect through the 172.20.40.0/24 subnet confirms that.

Would you therefore try configuring your server with /etc/rc.conf containing your desired settings for the host (I've guessed these from your previous posts):

ifconfig_re0="inet 172.20.20.32 netmask 0xffffff00"
ifconfig_re1="inet 172.20.40.32 netmask 0xffffff00"

Also, try to configure your jail with the address you want (172.20.40.233) in /usr/local/etc/ezjail/webjail and configure the jail's /etc/resolv.conf to point to your working DNS server:

nameserver 172.20.20.1

Come back and let us know whether everything is working as you expect.

I did that, in fact, I tried a couple of additional things, like for example I tried adding both

nameserver 172.20.20.1
nameserver 172.20.40.1

and restarting the jail, but that doesn't work either.

I keep reading about this option "setfib", when i search for this topic, and I'm wondering if there can be only routing for one subnet at a time? Apparently, setfib allows for the capability to add more routes, which I presume may mean that you could route packets on more than one subnet? Maybe that's a stretch?

 

[asteriskRoss]

Thanks for trying that, @DrH. I'm disappointed you didn't get a better result. Let me address a number of points.

Clarifying what you are trying to achieve
Can you say a little more about the design of the network you are trying to build? It sounds like you might be trying to have your host system route traffic through one network and have your jail(s) route traffic through a different network, both of which are connected to the Internet. This would make sense if you want to isolate network traffic to the jail from network traffic to the host. However, if that is what you want, setting up the host as a router between the two networks wouldn't seem like the right thing to do. You asked for that with the line in /etc/rc.conf:

gateway_enable="YES"

This causes your host to act as a router and pass traffic between the two networks. If you want your networks to be isolated from each other, you should remove this line. Even if routing traffic between both networks was what you wanted to do, it might be easier to turn it off whilst you are troubleshooting routing on your different networks.

Checking DNS servers
I have not used pfSense, which you said you are using as your DNS server. However, you can query a specific DNS server using drill(8) (a replacement in FreeBSD 10.0 for dig(8), which you may be familiar with). Successfully receiving a response will tell you that a DNS server is reachable at that address. For example, you could query 172.20.40.1 with the following command:

% drill @172.20.40.1 forums.freebsd.org

Understanding routing
The FreeBSD handbook has an excellent introduction to routing. You may find it useful to enable raw sockets in your jail that will allow you to use network diagnostic tools such as ping(8) and traceroute(8). You can do this with the following command in the host:

# sysctl security.jail.allow_raw_sockets=1

To retain this between reboots you can add the following to /etc/sysctl.conf:

security.jail.allow_raw_sockets=1

Also edit the following line in your jail's ezjail configuration file (for you /usr/local/etc/ezjail/webjail):

export jail_webjail_parameters="allow.raw_sockets=1"

Restart the jail for it to take effect. Once you have resolved your issues, unless you need them you may want to disallow raw sockets.

setfib
My rudimentary understanding of setfib(2) (Edit: changed from setfib(1)) is that it allows you to have multiple routing tables, rather than the usual one. If you are seeking to isolate your jail and host networks, this would be a way to set up a different default route for each interface (normally you set up a single default route for all interfaces). I see from the jail(8) man page that one of the parameters available is:

exec.fib
The FIB (routing table) to set when running commands inside the
prison.

This implies that having a jail use a different routing table from your host is possible. However, I have not used this myself, so I'll let somebody else answer and read the result with interest

 

[DrH]

@asteriskRoss, THANK YOU for your help. Let me share with you what eventually ended up working:

Apparently, we were correct in assuming that, by default, there is only one routing table permitted at a time. It must be necessary to have two, if packets are to be routed on two different networks. However, the GENERIC kernel doesn't allow for that, so I had to build a new kernel with the option

options         ROUTETABLES=2

and then install it. I used the guide in chapter 9 of the FreeBSD handbook to accomplish that. Next, I needed to specify a parameter in /boot/loader.conf so that the number of fibs could be loaded at boot time:

net.fibs=2

Next, I specified the new route with the command

# setfib 1 route add -host 172.20.40.32 172.20.40.1

Next, I added the following to /etc/rc.conf:

/sbin/route delete default
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 172.20.20.1

/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 172.20.40.1

Finally, I had to tell the jail to use fib 1 instead of the default fib by editing the following line in /usr/local/etc/ezjail/testjail

export jail_testjail_fib="1"

I rebooted, and could then install packages from within the jail using the 172.20.40.0/24 subnet! I'm not an expert at FreeBSD, so if you see anything that I did that could be harmful or may not be kosher please feel free to speak up and criticize.
Again, thank you, and the others, for the useful feedback. This is my first thread to the FreeBSD forums, I appreciate the helpfulness and generosity here.

 

[asteriskRoss]

@DrH, congratulations and you are very welcome. I am pleased that you have things working as you need. I'll have to try setfib(2) with a jail for myself when I have a moment. I see from looking at the setfib(2) man page that one may be able to use a setting in /boot/loader.conf rather than compiling a custom kernel.

The system maximum is
set in the kernel configuration file with

options ROUTETABLES=N

or in /boot/loader.conf with

net.fibs="N"

For most of my machines running FreeBSD I use a custom kernel, though if maintaining a lot of machines sticking with the GENERIC kernel can make updating with freebsd-update(8) very easy.

The only thing I would suggest for your current configuration is to consider moving the setfib(1) commands you are running directly from /etc/rc.conf into a new custom script that you place in /usr/local/etc/rc.d/. There is an article on how to write your own rc script and the rc(8) man page has a concise guide. It's actually pretty straightforward and given how far you've gotten should be a breeze! The advantage is that you can easily specify where in the boot sequence your script runs and manage any dependencies; see the rc(8) and rcorder(8) man pages. For example, you might want to ensure that the routing table your jail requires is configured before the jail starts.