OpenVPN server in jail (using a tun device)

Mattias

New Member

Thanks: 2
Messages: 13

#1
This text tries to describe how to get an OpenVPN (security/openvpn) server running inside a jail. The server will use a TUN device, but the same general procedures might work out with a TAP device.


Contents
  1. [jump=Assumptions]Assumptions[/jump]
  2. [jump=TheProblem]The problem[/jump]
  3. [jump=TheSolution]The solution[/jump]
  4. [jump=Troubleshooting]Troubleshooting[/jump]
  5. [jump=References]References[/jump]



Assumptions

  • The jail setup is similar to the one described in the handbook, 15.6 Application of Jails. [[jump=References]1[/jump]]
  • OpenVPN is installed inside the jail and configured to use a TUN device. See the official how-to [[jump=References]2[/jump]] and the sample configuration under [jump=Troubleshooting]Troubleshooting[/jump].
  • Personal customizations are taken into account. I.e firewall settings, built-in and left-out jail capabilities, security considerations etc.


The problem

On startup OpenVPN tries to (re-)establish a TUN/TAP device and some routes along with it. Since jails doesn't allow this we might be left with some of the following error messages (in log, if not on stdout):
Code:
openvpn: writing to routing socket: No such process
Cannot open TUN/TAP dev /dev/tun0: No such file or directory (errno=2)
ioctl(TUNSIFMODE): Device busy: Device busy (errno=16)
ifconfig: ioctl (set mtu): Operation not permitted
ifconfig: up: permission denied
ifconfig failed: external program exited with error status: 1


The solution
  1. Create the TUN device on the host. [[jump=References]3[/jump], [jump=References]4[/jump]]
  2. Give the jail access to the device. [[jump=References]5[/jump], [jump=References]6[/jump]]
  3. Configure the TUN interface when jail boots. [[jump=References]7[/jump]]
  4. Prevent OpenVPN from trying to configure interfaces. [[jump=References]8[/jump]]

Start off at the host by creating the TUN device...:
Code:
# ifconfig tun create
tun0
...on every boot:
# vi /etc/rc.conf

Add the following line before any jail settings:
Code:
cloned_interfaces="tun"
Check for the new tun interface:
Code:
% ifconfig
tun0: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
Back-up the file defining rules for device access...:
# cp /etc/defaults/devfs.rules /etc/defaults/devfs.rules_$(date +%F_%H%M)

...and edit it:
# vi /etc/defaults/devfs.rules

Add the following lines, but make sure it fits with your existing rules. Also, substitute <rule #> with the appropriate rule ID's and <jail name> with the name of your jail:
Code:
# Support for TUN devices
#
[devfsrules_unhide_tun=[highlight]<rule #>[/highlight]]
add path tun0 unhide

# Rules for jail [highlight]<jail name>[/highlight]
#
[devfsrules_jail_[highlight]<jail name>[/highlight]=[highlight]<rule #>[/highlight]]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_tun
Add the necessary settings for the jail:
# vi /etc/rc.conf

Depending on your current jail and OpenVPN settings you may need to change some of the values:
  • see rc.conf(5) for information on jail_⟨jname⟩_ip_multi⟨n⟩
  • the TUN settings wary with your OpenVPN server option, which in the following case is:
    Code:
    server 10.8.0.0 255.255.255.0
With that in mind add the following lines where suitable:
Code:
jail_[highlight]<jail name>[/highlight]_ip_multi0="tun0|10.8.0.1 10.8.0.2 mtu 1500 netmask 255.255.255.255"
jail_[highlight]<jail name>[/highlight]_devfs_enable="YES"
jail_[highlight]<jail name>[/highlight]_devfs_ruleset="devfsrules_jail_[highlight]<jail name>[/highlight]"
Restart the jail and step in:
# /etc/rc.d/jail restart <jail name>
# jexec <jail ID> su

In the jail, make sure the TUN interface shows up configured and ready for use by OpenVPN:
Code:
% ifconfig
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet 10.8.0.1 --> 10.8.0.2 netmask 0xffffffff
Finally, a very crucial option needs to be added to the OpenVPN configuration:
# echo 'ifconfig-noexec' >> /path/to/openvpn/server.conf

To start the server...:
# openvpn /path/to/openvpn/server.conf

...on every jail boot:
# vi /etc/rc.conf

Add the following lines, but change the path to your configuration file:
Code:
openvpn_enable="YES"
openvpn_configfile="[highlight]/path/to/openvpn/server.conf[/highlight]"
From the host, you should now be able to see the jail listening on port 1194:
Code:
# netstat -anf inet
[I][...][/I]
udp4       0      [highlight]<jail IP>[/highlight].1194       *.*
[I][...][/I]
A successful port scan from a remote host could look something like this:
Code:
# nmap -sU [highlight]<jail IP>[/highlight] -p1194-1195
[I][...][/I]
1194/udp open|filtered openvpn
1195/udp closed unknown
[I][...][/I]


Troubleshooting
  1. When following the above steps, take careful notice on possible differences in path and device names, jail name and IP, devfs rule names and ID's.

  2. Make sure the TUN device exists on the host and in the jail:
    % ls -l /dev/tun*

  3. Start off with a basic OpenVPN settings. Example:
    Code:
    local [highlight]<jail IP>[/highlight]
    port 1194
    proto udp
    dev tun0
    server 10.8.0.0 255.255.255.0
    ca /path/to/ca.crt
    cert /path/to/server.crt
    key /path/to/server.key
    dh /path/to/dh2048.pem
    keepalive 10 120
    comp-lzo
    user nobody
    group nobody
    persist-key
    persist-tun
    ifconfig-pool-persist /var/tmp/openvpn.pool
    status /var/tmp/openvpn.status
    log-append /var/log/openvpn.log
    verb 4
    mute 20
    ifconfig-noexec
  4. When asking for help include:
    • configuration files (host and jail rc.conf, OpenVPN config)
    • output from ifconfig, netstat -r and netstat -anf inet.
    • information on firewalling etc.


References
  1. The Handbook: 15.6 Application of Jails, http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails-application.html
  2. WWW: OpenVPN HOWTO, http://openvpn.net/index.php/open-source/documentation/howto.html
  3. Man page: ifconfig(8), see 'create'.
  4. Man page: rc.conf(5), see 'cloned_interfaces'.
  5. Mail: isc-dhcp3-server in a jail?, David N, http://lists.freebsd.org/pipermail/freebsd-questions/2007-June/151008.html
  6. Mail: dhcpd possible within jail?, Bjoern A. Zeeb, http://lists.freebsd.org/pipermail/freebsd-jail/2008-November/000579.html
  7. Man page: rc.conf(5), see 'jail_⟨jname⟩_ip_multi⟨n⟩'.
  8. WWW: Linux VServer FAQs, http://linux-vserver.org/Frequently_Asked_Questions#Can_I_run_an_OpenVPN_Server_in_a_guest.3F
 

ghostcorps

Well-Known Member

Thanks: 2
Messages: 295

#2
Hi :)

This is exactly what I am looking for!!! :D

But I get a bit confused here:

Mattias said:
With that in mind add the following lines where suitable:
Code:
jail_[highlight]<jail name>[/highlight]_ip_multi0="tun0|10.8.0.1 10.8.0.2 mtu 1500 netmask 255.255.255.255"
jail_[highlight]<jail name>[/highlight]_devfs_enable="YES"
jail_[highlight]<jail name>[/highlight]_devfs_ruleset="devfsrules_jail_[highlight]<jail name>[/highlight]"
I have OpenVPN working perfectly on the host, I can ssh into it etc, but I can not see Webserver (the jail) once on the VPN. What else do I need to add to rc.conf to allow this?

Here are the current configurations upto this point, as requested:

Host
/etc/defaults/devfs.rules
Code:
# Support for TUN devices
#
[devfsrules_unhide_tun=4]
add path tun0 unhide

# Devices usually found in a jail.
#
[devfsrules_jail=5]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_tun
Host
/etc/rc.conf
Code:
hostname="my.domain.com"
sshd_enable="YES"
ifconfig_em0="inet xxx.xxx.xxx.xx2 netmask 255.255.255.248"
defaultrouter="xxx.xxx.xxx.xx9"
gateway_enable="YES"
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_type="closed"
openvpn_enable="YES"
openvpn_configfile="/usr/local/etc/openvpn/server.conf"
openvpn_if="tap"
...
cloned_interfaces="tun"
jail_set_hostname_allow="NO"
jail_enable="YES"
jail_list="webserver"
jail_interface="em0"
jail_devfs_enable="YES"
jail_procfs_enable="YES"
jail_webserver_rootdir="/usr/gaols/webserver"
jail_webserver_interface="em0"
jail_webserver_hostname="webserver"
jail_webserver_ip="xxx.xxx.xxx.xx3"
jail_webserver_devfs_ruleset="devfsrules_jail"
Host
/etc/hosts
Code:
::1                     localhost localhost.my.domain
127.0.0.1               localhost localhost.my.domain my my.domain.com
xxx.xxx.xxx.xx2         my my.domain.com
192.168.254.1           host
192.168.254.2           webserver
Host
/usr/local/etc/openvpn/server.conf
Code:
port 1194
proto udp
dev tap
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/server.crt
key /usr/local/etc/openvpn/keys/server.key # This file should be kept secret
dh /usr/local/etc/openvpn/keys/dh1024.pem
server 192.168.254.0 255.255.255.0
ifconfig-pool-persist ipp.txt
peer-to-peer
keepalive 10 120
comp-lzo
max-clients 10
user root
persist-key
persist-tun
status /var/log/openvpn-status.log
log-append  /var/log/openvpn.log
verb 3
Host
#ifconfig
Code:
# ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether xx:xx:xx:xx:xx:xx
        inet xxx.xxx.xx2 netmask 0xfffffff8 broadcast xxx.xxx.xx5
        inet xxx.xxx.xx3 netmask 0xffffffff broadcast xxx.xxx.xx3
        inet 1xxx.xxx.xx4 netmask 0xffffffff broadcast xxx.xxx.xx4
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

...

tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:8e:20:00:00
        inet 192.168.254.1 netmask 0xffffff00 broadcast 192.168.254.255
        Opened by PID 20612
tun0: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
Host
# netstat -r
Code:
# netstat -r
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            174.137.139.209    UGS         2    13748    em0
localhost          link#5             UH          0        4    lo0
xxx.xxx.xxx.xx8/29 link#1             U           0        0    em0
domain             link#1             UHS         0      222    lo0
xxx.xxx.xxx.xx3    link#1             UHS         0     3673    lo0 =>
xxx.xxx.xxx.xx3/32 link#1             U           0        0    em0
xxx.xxx.xxx.xx4    link#1             UHS         0      769    lo0 =>
xxx.xxx.xxx.xx4/32 link#1             U           0        0    em0
192.168.254.0      link#6             U           1     1422   tap0
host               link#6             UHS         0        0    lo0

Internet6:
Destination        Gateway            Flags      Netif Expire
localhost          localhost          UH          lo0
fe80::%lo0         link#5             U           lo0
fe80::1%lo0        link#5             UHS         lo0
ff01:5::           fe80::1%lo0        U           lo0
ff02::%lo0         fe80::1%lo0        U           lo0
Host
# netstat -anf inet
Code:
# netstat -anf inet
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address       (state)
tcp4       0     52 192.168.254.1.22       192.168.254.3.3813     ESTABLISHED
tcp4       0      0 192.168.254.1.22       *.*                    LISTEN
tcp4       0      0 xxx.xxx.xxx.xx3.5080   *.*                    LISTEN
tcp4       0      0 xxx.xxx.xxx.xx3.1935   *.*                    LISTEN
tcp4       0      0 xxx.xxx.xxx.xx3.10085  *.*                    LISTEN
tcp4       0      0 xxx.xxx.xxx.xx3.9999   *.*                    LISTEN
tcp4       0      0 xxx.xxx.xxx.xx3.443    *.*                    LISTEN
tcp4       0      0 xxx.xxx.xxx.xx3.80     *.*                    LISTEN
tcp4       0      0 xxx.xxx.xxx.xx4.8080   *.*                    LISTEN
tcp4       0      0 xxx.xxx.xxx.xx3.21     *.*                    LISTEN
tcp4       0      0 192.168.254.1.22       192.168.254.4.1761     ESTABLISHED
tcp4       0      0 xxx.xxx.xxx.xx3.21     *.*                    LISTEN
udp4       0      0 xxx.xxx.xxx.xx2.2501   *.*
udp4       0      0 xxx.xxx.xxx.xx4.514    *.*
udp4       0      0 xxx.xxx.xxx.xx3.514    *.*
udp4       0      0 xxx.xxx.xxx.xx3.514    *.*
udp4       0      0 *.514                  *.*
Host nb. Unfortunately, ipfw refuses to be 'closed' so I need to write in all of the closed rules, hence why it is so messy.
# ipfw show
Code:
# ipfw show
00010   258   23524 allow ip from any to any via lo0
00011     0       0 allow ip from me to me
00020     0       0 check-state
00030    32    2926 allow icmp from any to any
00060  2096  325045 allow ip from any to xxx.xxx.xxx.xx2 dst-port 1024-7999
00070     0       0 allow ip from any to xxx.xxx.xxx.xx2 dst-port 8001-8088
00080     0       0 allow ip from any to xxx.xxx.xxx.xx2 dst-port 8090-9998
00090    13     960 allow ip from any to xxx.xxx.xxx.xx2 dst-port 10000-65535
00100     0       0 allow ip from any to xxx.xxx.xxx.xx3 dst-port 1024-5079
00110     0       0 allow ip from any to xxx.xxx.xxx.xx3 dst-port 5081-7999
00120     0       0 allow ip from any to xxx.xxx.xxx.xx3 dst-port 8001-8087
00120     0       0 allow ip from any to xxx.xxx.xxx.xx3 dst-port 8090-9998
00120     7    1127 allow ip from any to xxx.xxx.xxx.xx3 dst-port 10000-65535
00800     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx2 dst-port 1-1023
00800     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 1-19
00800     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 444-1934
00810     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 22-79
00810     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 1936-1023
00820     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 81-442
00830     0       0 deny ip from any to 1xxx.xxx.xxx.xx2 dst-port 8089
00840     0       0 deny ip from any to xxx.xxx.xxx.xx2 dst-port 9999
00950     0       0 deny ip from any to xxx.xxx.xxx.xx2 dst-port 8000
01010     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 5080
01020     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 8088
01030     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 8089
01040     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 8000
01050     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 9999
02000     0       0 deny ip from any to xxx.xxx.xxx.xx4
65535 36436 5479174 allow ip from any to any
Webserver (jail)
/etc/rc.conf
Code:
hostname="my.other.domain.com"
ifconfig_em0="inet xxx.xxx.xxx.xx3 netmask 255.255.255.255"
defaultrouter="xxx.xxx.xxx.xx9"
...
kern_securelevel_enable="YES"
kern_securelevel="3"
portmap_enable="NO"
log_in_vain="YES"
pureftpd_enable="YES"
red5_enable="YES"
openvpn_enable="YES"
openvpn_configfile="/usr/local/etc/openvpn/client.conf"
openvpn_if="tap"
 

Schnigges

New Member


Messages: 1

#3
First of all, thanks for the great tutorial!
Do you have any hints on how to do the setup for a bridged openvpn? I guess I would have to create a tap device on the host instead of a tun, but I am having trouble creating one (not entirely sure if this would even be the right way). When I try
Code:
ifconfig tap0 create 192.168.178.96 up
I get the following error:
Code:
ifconfig: SIOCIFCREATE2: Invalid argument
 
OP
OP
M

Mattias

New Member

Thanks: 2
Messages: 13

#4
I'm unable to recreate that error, but ifconfig's stderr isn't to be blindly trusted (see example below) :) Maybe try resetting your config related to devices (devfs.rules, rc.conf) as much as possible and make sure the device is destroyed between every try.

Code:
# ifconfig tap0 create 1.2.3[B],[/B]4 [notice the comma]
ifconfig: 1.2.3,4: bad value

# ifconfig tap0 create 1.2.3.4 [comma corrected]
ifconfig: create: bad value [[B]create[/B] is a bad value]

# ifconfig tap0
tap0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
[...]
 

lockdoc

Active Member

Thanks: 6
Messages: 124

#5
I also have a problem on this.

I am using tun1, as tun0 is for my internet connection. From the openvpn.log on the jail side I get the following error:
Code:
Tue Mar  6 13:33:45 2012 us=282808   mode = 1
Tue Mar  6 13:33:45 2012 us=282840   show_ciphers = DISABLED
Tue Mar  6 13:33:45 2012 us=282869   show_digests = DISABLED
Tue Mar  6 13:33:45 2012 us=282897   show_engines = DISABLED
Tue Mar  6 13:33:45 2012 us=282924   genkey = DISABLED
Tue Mar  6 13:33:45 2012 us=283012   key_pass_file = '[UNDEF]'
Tue Mar  6 13:33:45 2012 us=283041   show_tls_ciphers = DISABLED
Tue Mar  6 13:33:45 2012 us=283069 Connection profiles [default]:
Tue Mar  6 13:33:45 2012 us=283098   proto = udp
Tue Mar  6 13:33:45 2012 us=283125   local = '[UNDEF]'
Tue Mar  6 13:33:45 2012 us=283150   local_port = 1194
Tue Mar  6 13:33:45 2012 us=283176   remote = '[UNDEF]'
Tue Mar  6 13:33:45 2012 us=283201   remote_port = 1194
Tue Mar  6 13:33:45 2012 us=283226   remote_float = DISABLED
Tue Mar  6 13:33:45 2012 us=283293   bind_defined = DISABLED
Tue Mar  6 13:33:45 2012 us=283318   bind_local = ENABLED
Tue Mar  6 13:33:45 2012 us=283397 NOTE: --mute triggered...
Tue Mar  6 13:33:45 2012 us=283429 194 variation(s) on previous 20 message(s) suppressed by --mute
Tue Mar  6 13:33:45 2012 us=283461 OpenVPN 2.2.2 amd64-portbld-freebsd9.0 [SSL] [LZO2] [eurephia] built on Mar  6 2012
openvpn: writing to routing socket: No such process
Tue Mar  6 13:33:45 2012 us=283727 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Tue Mar  6 13:33:45 2012 us=287101 Diffie-Hellman initialized with 1024 bit key
Tue Mar  6 13:33:45 2012 us=287865 TLS-Auth MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
Tue Mar  6 13:33:45 2012 us=287916 Socket Buffers: R=[42080->65536] S=[9216->65536]
openvpn: writing to routing socket: No such process
openvpn: writing to routing socket: No such process
Tue Mar  6 13:33:45 2012 us=288149 ROUTE: default_gateway=UNDEF
Tue Mar  6 13:33:45 2012 us=288222 TUN/TAP device /dev/tun1 opened
openvpn: writing to routing socket: No such process
Tue Mar  6 13:33:45 2012 us=288346 /sbin/route add -net 10.8.0.0 10.8.0.2 255.255.255.0
route: writing to routing socket: Operation not permitted
[color="Red"]Tue Mar  6 13:33:45 2012 us=289457 ERROR: FreeBSD route add command failed: external program exited with error status: 1
[/color]Tue Mar  6 13:33:45 2012 us=289550 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Tue Mar  6 13:33:45 2012 us=289954 UDPv4 link local (bound): [undef]:1194
Tue Mar  6 13:33:45 2012 us=289998 UDPv4 link remote: [undef]
Tue Mar  6 13:33:45 2012 us=290028 MULTI: multi_init called, r=256 v=256
Tue Mar  6 13:33:45 2012 us=290128 IFCONFIG POOL: base=10.8.0.4 size=62
Tue Mar  6 13:33:45 2012 us=290166 IFCONFIG POOL LIST
Tue Mar  6 13:33:45 2012 us=290247 Initialization Sequence Completed
The client side gives me the following:
Code:
Tue Mar 06 14:39:18 2012 us=421000 OpenVPN 2.2.1 Win32-MSVC++ [SSL] [LZO2] built on Jul  1 2011
Tue Mar 06 14:39:18 2012 us=421000 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Tue Mar 06 14:39:18 2012 us=750000 LZO compression initialized
Tue Mar 06 14:39:18 2012 us=750000 Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
Tue Mar 06 14:39:18 2012 us=750000 Socket Buffers: R=[8192->8192] S=[8192->8192]
Tue Mar 06 14:39:18 2012 us=750000 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Tue Mar 06 14:39:18 2012 us=765000 Local Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-client'
Tue Mar 06 14:39:18 2012 us=765000 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-server'
Tue Mar 06 14:39:18 2012 us=765000 Local Options hash (VER=V4): '41690919'
Tue Mar 06 14:39:18 2012 us=765000 Expected Remote Options hash (VER=V4): '530fdded'
Tue Mar 06 14:39:18 2012 us=765000 UDPv4 link local: [undef]
Tue Mar 06 14:39:18 2012 us=765000 UDPv4 link remote: 192.168.0.5:1194
Tue Mar 06 14:39:25 2012 write UDPv4: Interrupted system call (WSAEINTR) (code=10004)
Tue Mar 06 14:39:33 2012 us=296000 write UDPv4: Interrupted system call (WSAEINTR) (code=10004)
Is the problem with not being able to add routes?
 

LJackson

New Member

Thanks: 1
Messages: 8

#6
In general, these instructions do appear to work with a TAP device, and without using OpenVPN's client-config-dir as discussed on http://forums.freebsd.org/showthread.php?t=24063&highlight=jail+openvpn. First, placing if_tap_load="YES" in /boot/loader.conf will prevent situations where all the interfaces appeared to be configured correctly, but the OpenVPN client stalls with errors like "Connection Reset by Peer".

Ensure OpenVPN can use the TAP device (the last two lines are key):

Code:
# sysctl -a | grep link.tap
net.link.tap.debug: 0
net.link.tap.devfs_cloning: 1
[B]net.link.tap.up_on_open: 1
net.link.tap.user_open: 1[/B]
Next, setup the bridge and configure the interfaces. The following applies to /etc/rc.conf. My host has a re0 interface (10.0.0.70), is running Samba, and has OpenVPN in a jail addressed as 10.0.0.74.

Code:
cloned_interfaces="tap0 bridge0"
ifconfig_bridge0="addm re0 addm tap0 up"
ifconfig_re0="up"                         # All interfaces must be up
ifconfig_tap0="up"

ipv4_addrs_bridge0="10.0.0.70/24" # Host IP (Samba), 74 is on re0

## Note: [B]ifconfig_bridge0[/B] should work in place of [B]ipv4_addrs_bridge0[/B]

ifconfig_re0="10.0.0.74/24"       # OpenVPN's IP
I found placing OpenVPN's IP on the bridge as an alias (ifconfig_bridge0_alias0="inet 10.0.0.74 netmask 255.255.255.255") yielded mixed results: using ssh to contact the host would still work, but Samba wouldn't. Placing both on re0 also appeared to work, but goes against convention. (If the host is to have an IP address it should go on the host. See http://www.freebsd.org/doc/handbook/network-bridging.html.) Making the TAP device sticky is interesting. (See the aforementioned handbook.)

The TAP device is set using either

Code:
ifconfig_tap0="10.8.0.50 netmask 255.255.255.0"
or
Code:
jail_<[FILE][B]OpenVPN[/B][/FILE]>_ip_multi0="tap0|10.8.0.50/24"
In my case, both appeared to work, but the latter is preferred for consistency.

Depending on how you are using your host and the bridge the gateway_enable="YES" in /etc/rc.conf may not be required.

As reported for the TUN device http://forums.freebsd.org/showthread.php?t=27934&highlight=openvpn+jail, I did find if OpenVPN is stopped the IP address assigned to the TAP device is removed. I wouldn't have thought that is possible, maybe something to do with the sysctl values?

My OpenVPN server.conf file looks like:

Code:
server-bridge 10.8.0.50 255.255.255.0 10.8.0.51 10.8.0.99

proto udp
dev tap

ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/server.crt
key /usr/local/etc/openvpn/keys/server.key
dh /usr/local/etc/openvpn/keys/dh1024.pem
tls-auth /usr/local/etc/openvpn/keys/ta.key 0

client-to-client

ifconfig-noexec

push "dhcp-option DNS 10.0.0.1" 
push "dhcp-option WINS 10.0.0.70"
push "dhcp-option NBT 8"

mssfix 1450

ping 10
ping-restart 120
push "ping 10"
push "ping-restart 60"
push "route 10.0.0.0 255.255.255.0" # Advertise Server's LAN to VPN clients

comp-lzo
user nobody
group nobody

status /var/log/openvpn/openvpn-status.log
log-append /var/log/openvpn/openvpn.log
verb 3
and the client file for a Microsoft XP system and a MacBook Pro OS X Lion looks like (The host is also acting as a Time Machine Server.)

Code:
client
dev tap
proto udp
remote <yours.com> 1194

tls-client

nobind

ca <your ca.crt>
cert <your client.crt>
key <your client.key>
tls-auth <your ta.key> 1

tun-mtu 1500
tun-mtu-extra 32
mssfix 1450

pull

comp-lzo
verb 3
 

dvl@

Aspiring Daemon
Developer

Thanks: 57
Messages: 533

#8
Why alter /etc/defaults/devfs.rules? Why not change /etc/devfs.rules instead? That's how this is supposed to work.
 

junovitch@

Daemon
Developer

Thanks: 613
Messages: 1,773

#9
Good day. I have a few additions to toss into the mix as I had a few issues getting this to work as intended on 9.1-RELEASE. I hope this helps anybody in the same situation.

  • Issue: The tunnel was not recognized in the jails initially. The devfs.rules were correct. This appeared to be because I was already using OpenVPN on the host and something about the way it was created prevented it from being shown in a jail.
  • Fix: Delete the current tunnel and create it again.
Code:
ifconfig tun0 destroy
ifconfig tun create
  • Issue: Routing was needed to access anything after connecting.
  • Fix: Add it to the host.
# vi /etc/rc.conf
Code:
cloned_interfaces="tun"
gateway_enable="YES"
  • Issue: The /etc/defaults/devfs.rules may get updated during system upgrades.
  • Fix: Modify the /etc/devfs.rules over the default. On 9.1-RELEASE there are four rules by default in /etc/defaults/devfs.rules so I used rule 5 with a single combined rule.
# vi /etc/devfs.rules
Code:
[devfsrules_jail_with_vpn=5]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path tun0 unhide
add path zfs unhide
  • Issue: My clients could connect but not ping anything.
  • Fix: A prestart routine was used to add the route on the host upon jail start.
Code:
jail_[highlight]<jail name>[/highlight]_ip_multi0="tun0|10.8.0.1 10.8.0.2 netmask 255.255.255.255"
jail_[highlight]<jail name>[/highlight]_exec_prestart0="/sbin/route add -net 10.8.0.0/24 10.8.0.2"
jail_[highlight]<jail name>[/highlight]_devfs_enable="YES"
jail_[highlight]<jail name>[/highlight]_devfs_ruleset="devfsrules_jail_with_vpn"
  • Issue: OpenVPN couldn't add routes inside the jail. This had already been solved by the prestart routine.
  • Fix: Add both noexec options to the OpenVPN configuration.
# echo 'ifconfig-noexec' >> /path/to/openvpn/server.conf
# echo 'route-noexec' >> /path/to/openvpn/server.conf
 

junovitch@

Daemon
Developer

Thanks: 613
Messages: 1,773

#11
Unfortunately restarting OpenVPN will trigger the tunnel to lose its IP address. As it cannot add it back from inside the jail the only option when I have to restart OpenVPN is to restart the whole jail. Not ideal but it's not a deal breaker as the jail is dedicated to OpenVPN and I do control the host after all.

Your post regarding jaildaemon monitoring is probably the way to go. For anybody out there who can't just restart the jail whenever they have to restart OpenVPN, I would suggest taking a look at this post and links within for a potential solution.

http://forums.freebsd.org/showpost.php?p=231691&postcount=4
 

FreeBSDNewUser

New Member


Messages: 6

#13
Good day. I have a few additions to toss into the mix as I had a few issues getting this to work as intended on 9.1-RELEASE. I hope this helps anybody in the same situation.

  • Issue: The tunnel was not recognized in the jails initially. The devfs.rules were correct. This appeared to be because I was already using OpenVPN on the host and something about the way it was created prevented it from being shown in a jail.
  • Fix: Delete the current tunnel and create it again.
Code:
ifconfig tun0 destroy
ifconfig tun create
  • Issue: Routing was needed to access anything after connecting.
  • Fix: Add it to the host.
# vi /etc/rc.conf
Code:
cloned_interfaces="tun"
gateway_enable="YES"
  • Issue: The /etc/defaults/devfs.rules may get updated during system upgrades.
  • Fix: Modify the /etc/devfs.rules over the default. On 9.1-RELEASE there are four rules by default in /etc/defaults/devfs.rules so I used rule 5 with a single combined rule.
# vi /etc/devfs.rules
Code:
[devfsrules_jail_with_vpn=5]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path tun0 unhide
add path zfs unhide
  • Issue: My clients could connect but not ping anything.
  • Fix: A prestart routine was used to add the route on the host upon jail start.
Code:
jail_[highlight]<jail name>[/highlight]_ip_multi0="tun0|10.8.0.1 10.8.0.2 netmask 255.255.255.255"
jail_[highlight]<jail name>[/highlight]_exec_prestart0="/sbin/route add -net 10.8.0.0/24 10.8.0.2"
jail_[highlight]<jail name>[/highlight]_devfs_enable="YES"
jail_[highlight]<jail name>[/highlight]_devfs_ruleset="devfsrules_jail_with_vpn"
  • Issue: OpenVPN couldn't add routes inside the jail. This had already been solved by the prestart routine.
  • Fix: Add both noexec options to the OpenVPN configuration.
# echo 'ifconfig-noexec' >> /path/to/openvpn/server.conf
# echo 'route-noexec' >> /path/to/openvpn/server.conf
Hello Junovitch,

Thanks for the help can you let me know where are these rules supposed to go rc.conf file on the host?
Code:
jail_[highlight]<jail name>[/highlight]_ip_multi0="tun0|10.8.0.1 10.8.0.2 netmask 255.255.255.255"
jail_[highlight]<jail name>[/highlight]_exec_prestart0="/sbin/route add -net 10.8.0.0/24 10.8.0.2"
jail_[highlight]<jail name>[/highlight]_devfs_enable="YES"
jail_[highlight]<jail name>[/highlight]_devfs_ruleset="devfsrules_jail_with_vpn"
Also this link does not work:
Unfortunately restarting OpenVPN will trigger the tunnel to lose its IP address. As it cannot add it back from inside the jail the only option when I have to restart OpenVPN is to restart the whole jail. Not ideal but it's not a deal breaker as the jail is dedicated to OpenVPN and I do control the host after all.

Your post regarding jaildaemon monitoring is probably the way to go. For anybody out there who can't just restart the jail whenever they have to restart OpenVPN, I would suggest taking a look at this post and links within for a potential solution.

http://forums.freebsd.org/showpost.php?p=231691&postcount=4
I would really appreciate some help on this is anyone willing to create a post with instructions from start to finish how they got VPN working inside a jail? I would be very grateful as I cannot seem to get it to work but I believe it is because I am missing ip_mulit0 because I am unsure where it goes. Does it matter what the jail ip is? and does that need to go in the openvpn.conf file? I think I may also have an issue in the VPN conf.
 

junovitch@

Daemon
Developer

Thanks: 613
Messages: 1,773

#14
The link was on the old forums. I found the post again in Thread 41691. The two links mentioned:
https://elektropost.org/ezjail/msg00982.html
http://erdgeist.org/arts/software/jaildaemon/

The jail_<jail_name>_* variables were the rc.conf style of jails. Ezjail happens to keep all these in /usr/local/etc/ezjail for each jail. If you are using sysutils/ezjail that would still apply. If not, it would help to know how you are setting up jails. I would imagine the jail.conf format would be different but the concept would be the same.

I'm not using OpenVPN in a jail anymore. I used it for a while because I had a old router that wasn't up to the job. After I replaced my router I moved the VPN duties to that device. It's still FreeBSD but I didn't jail it this time. It's just easier to manage that way. All my notes from when I did run OpenVPN in a jail I left in my notes. If you do see any inaccuracies let me know. I'll fix them up. Hope that helps.

https://github.com/junovitch/my-freebsd-build/blob/master/openvpn-jail-HOWTO
 

FreeBSDNewUser

New Member


Messages: 6

#15
Thanks a lot for the update I really appreciate it. I actually stumbled across the github link you have given here yesterday but have not have had a chance to try it out. I am very new to FreeBSD and am just wondering if I create a jail (not for the VPN) but run the VPN on the host will the jail use the tun interface or is there some way I need to set it up so that it uses the tun interface rather than the NIC card. I have my firewall setup to not allow any traffic if it is not over the VPN. I have tested and it works fine. The problem is the jail does not get out of the firewall because it is not using the tun interface and is bypassing VPN.

Long story short I want all the traffic of the jail to go through VPN just as all the traffic on the host goes through the VPN. The VPN itself does not need to be in the jail is the github link the route that I should take to achieve this or is there an easier way. Also another question do I even need to use jails or can I just install all my packages in the Freebsd FreeBSD host system as the freebsd FreeBSD host system is actually in a chroot using finch. I am using NAS4FREE 9.2 embedded but I have finch installed and the FREEBSD 9.2 filesystem on a seperate ZFS.

My shares and all my files are on a separate drive using a raid5 configuration. So if I do break anything in finch I can just format that drive and redo finch and all the packages. Is that a normal thing to do or should I stick with using jails inside the chroot? Sorry for all the questions.

Thanks.
 

FreeBSDNewUser

New Member


Messages: 6

#16
I am using finch for managing the jails and qjail console "$jailname" is how I enter the jail. I am not sure if finch uses ezjails but I think it does and that might explain some of my problems with getting it working.

Thanks
 

junovitch@

Daemon
Developer

Thanks: 613
Messages: 1,773

#17
By default, the jail would just use the host routing table. So if the VPN pushes a default route the jail traffic would leave over the tun interface. It shouldn't bypass the VPN as you mention if you are going with defaults. That would be the outbound.

The likely problem is the inbound back to the jail probably doesn't have any config to get back. I am guessing that your host is the OpenVPN client right? The server side would need a route/iroute config keyword to get back to the address or network of the jail. Alternately you can configure your firewall to NAT the outbound traffic from jail IP addresses to use the IP of the tun interface.

The VPN doesn't need to be in a jail and it only even makes sense going through the hassle if you are running it as a service. If you are a client then it's not worth the effort since it's not accepting any connections from the outside.

However if you are already using Finch to isolate an environment I don't think it makes as much sense to add another layer into the mix to complicate things. Pick one method and stick to it or if you use both use them in parallel for different environments.

An FYI, I've only looked at NAS4FREE briefly a couple years ago and sometimes advise doesn't apply. This is why we have a rule regarding threads on derivatives.
https://forums.freebsd.org/threads/pc-bsd-freenas-nas4free-and-all-other-freebsd-derivatives.7290/

Also, FreeBSD 9.2-RELEASE is also out of support so a reminder about sticking with an OS supported with security updates.
https://forums.freebsd.org/threads/topics-about-unsupported-freebsd-versions.40469/
 

FreeBSDNewUser

New Member


Messages: 6

#18
You would think that by default it uses the tun0 interface but in fact it doesn't because when I ping I get "permission denied" and not "timed out" that means the firewall is blocking it because it is not using tun0. I think it would be my fault because qjail allows to change the network interface:
Code:
new_network_interface="re0"
qjail config -c "$new_network_interface""$jailname"
I could just explicitly change the network interface for the jail and see if that works then I will probably run into the "timed out" error when the ping goes out but cannot return.

You guessed right the host is the OpenVPN client and I use Mullvad's VPN Servers. So a route/iroute on the server side is out of the question I think. How would I configure the firewall to NAT the outbound traffic from the jail IP address to use the IP of the tun interface? As far as I can tell all local traffic in 192.168.0.0 in going over the VPN.

Ok so as you say the VPN doesn't need to be in a jail but so I will run it on the host machine or on finch. Inside finch I create a jail to install deluge and plex media server will they be able to run over the VPN that is already running on the host system or will there be issues? As you say it doesn't make much sense to add another layer i.e. a jail to complicate things. Can I just install all those packages directly on my finch chroot instead of creating a jail?

I read the link and I will try some help on the NAS4FREE forums if I cant get it working. however technically inside finch is a complete FreeBSD 9.2 version of software so it uses the new package management system. I updated it and I realize that means I cannot use PKG_ADD anymore.

I will read the other link with regards to version of FreeBSD but I used the version that comes with finch and did not upgrade it because was not sure if it would work with the older system of NAS4FREE. Anyways off to work now. Thanks for all the help by the way and I am sure come this weekend I should have it up and running.
 

junovitch@

Daemon
Developer

Thanks: 613
Messages: 1,773

#19
The permission denied was because a ping is a raw socket and that requires special permissions within a jail. Here's a helpful copy/paste from the Handbook to fill you in.

Tip:
Most network services run in jails without problems. A few network services, most notably ping(8), use raw network sockets. In jails, raw network sockets are disabled by default for security. Services that require them will not work.

Occasionally, a jail genuinely needs raw sockets. For example, network monitoring applications often use ping(8) to check the availability of other computers. When raw network sockets are actually needed in a jail, they can be enabled by editing the ezjail configuration file for the individual jail,/usr/local/etc/ezjail/jailname. Modify the parameters entry:

Code:
export jail_jailname_parameters="allow.raw_sockets=1"
Do not enable raw network sockets unless services in the jail actually require them.
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails-ezjail.html

The NAT/firewall bit is covered in the Handbook. That would be a good place to start. If you have the NAT rule just NAT everything then your jails should just work.
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls.html

Regarding NAS4FREE, their websites lists there is a version 9.3 that is based on a supported release of FreeBSD 9.3.

Since this has kind of strayed outside of running an OpenVPN server in a jail, I would recommend reviewing the links above and posting if you have more specific questions on what you are doing. Specifically it would be allowing the jail or finch environment on your system to seamlessly share the OpenVPN client on the host.
 

FreeBSDNewUser

New Member


Messages: 6

#20
I have allowed raw sockets and I can ping when I disable the firewall. When I enable the firewall I can no longer ping because the jail is not using the tun0 interface and if I change the interface it doesn't work I get a error in the jail.conf because the ip is not in the range for tun0 I will create a new jail with openvpn up and running and assign it an ip and netmask that will work with the tun0 interface. If I do this I believe the jail will not work over the LAN as it should because the ip is not in the LAN range but is in the range of the vpn ip. So this will probably cause more issues.

I will read about the NAT/firewall as I am really bad in that area.

I will also try to update my NAS4FREE Version after backing up the config .xml so that I have supported versions.

Thanks for all your help I really appreciate it.
 

tenq

Member

Thanks: 1
Messages: 24

#21
Hello!

Thank you for you guide.

I've configured all as described, but getting error when starting jail.


ifconfig: ioctl (SIOCAIFADDR): Destination address required
jail: vpn_amnesi_ac: /sbin/ifconfig tun0 inet 10.8.0.1/32 alias: failed


i'm using 10.3-RELEASE-p11 amd64

Could you please make advice?
 

ucc

New Member


Messages: 10

#22
I have also followed this guide and also ran into the same Problem as tenq did. But there is a new(er) thread, wich points out the solution:
I wish I remember the reason why I changed it, but I found the multi syntax didn't work alright for me when I went from FreeBSD 9.2 to 10.0.
Spoiler: Instead of multi use:
Code:
export jail_[highlight]<jail name>[/highlight]_exec_prestart0="/sbin/ifconfig tun0 inet 10.8.0.1/32 10.8.0.2"
export jail_[highlight]<jail name>[/highlight]_exec_prestart1="/sbin/route add -net 10.8.0.1/24 10.8.0.2"
 
Top