1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

OpenVPN server in jail (using a tun device)

Discussion in 'Howtos and FAQs (Moderated)' started by Mattias, Feb 28, 2011.

  1. Mattias

    Mattias New Member

    Messages:
    13
    Thanks Received:
    2
    This text tries to describe how to get an OpenVPN (security/openvpn) server running inside a jail. The server will use a TUN device, but the same general procedures might work out with a TAP device.


    Contents
    1. [jump=Assumptions]Assumptions[/jump]
    2. [jump=TheProblem]The problem[/jump]
    3. [jump=TheSolution]The solution[/jump]
    4. [jump=Troubleshooting]Troubleshooting[/jump]
    5. [jump=References]References[/jump]



    Assumptions

    • The jail setup is similar to the one described in the handbook, 15.6 Application of Jails. [[jump=References]1[/jump]]
    • OpenVPN is installed inside the jail and configured to use a TUN device. See the official how-to [[jump=References]2[/jump]] and the sample configuration under [jump=Troubleshooting]Troubleshooting[/jump].
    • Personal customizations are taken into account. I.e firewall settings, built-in and left-out jail capabilities, security considerations etc.


    The problem

    On startup OpenVPN tries to (re-)establish a TUN/TAP device and some routes along with it. Since jails doesn't allow this we might be left with some of the following error messages (in log, if not on stdout):
    Code:
    openvpn: writing to routing socket: No such process
    Cannot open TUN/TAP dev /dev/tun0: No such file or directory (errno=2)
    ioctl(TUNSIFMODE): Device busy: Device busy (errno=16)
    ifconfig: ioctl (set mtu): Operation not permitted
    ifconfig: up: permission denied
    ifconfig failed: external program exited with error status: 1
    




    The solution
    1. Create the TUN device on the host. [[jump=References]3[/jump], [jump=References]4[/jump]]
    2. Give the jail access to the device. [[jump=References]5[/jump], [jump=References]6[/jump]]
    3. Configure the TUN interface when jail boots. [[jump=References]7[/jump]]
    4. Prevent OpenVPN from trying to configure interfaces. [[jump=References]8[/jump]]

    Start off at the host by creating the TUN device...:
    Code:
    # ifconfig tun create
    tun0
    


    ...on every boot:
    # vi /etc/rc.conf

    Add the following line before any jail settings:
    Code:
    cloned_interfaces="tun"


    Check for the new tun interface:
    Code:
    % ifconfig
    tun0: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
            options=80000<LINKSTATE>
    


    Back-up the file defining rules for device access...:
    # cp /etc/defaults/devfs.rules /etc/defaults/devfs.rules_$(date +%F_%H%M)

    ...and edit it:
    # vi /etc/defaults/devfs.rules

    Add the following lines, but make sure it fits with your existing rules. Also, substitute <rule #> with the appropriate rule ID's and <jail name> with the name of your jail:
    Code:
    # Support for TUN devices
    #
    [devfsrules_unhide_tun=<rule #>]
    add path tun0 unhide
    
    # Rules for jail <jail name>
    #
    [devfsrules_jail_<jail name>=<rule #>]
    add include $devfsrules_hide_all
    add include $devfsrules_unhide_basic
    add include $devfsrules_unhide_login
    add include $devfsrules_unhide_tun
    


    Add the necessary settings for the jail:
    # vi /etc/rc.conf

    Depending on your current jail and OpenVPN settings you may need to change some of the values:
    • see rc.conf(5) for information on jail_⟨jname⟩_ip_multi⟨n⟩
    • the TUN settings wary with your OpenVPN server option, which in the following case is:
      Code:
      server 10.8.0.0 255.255.255.0
    With that in mind add the following lines where suitable:
    Code:
    jail_<jail name>_ip_multi0="tun0|10.8.0.1 10.8.0.2 mtu 1500 netmask 255.255.255.255"
    jail_<jail name>_devfs_enable="YES"
    jail_<jail name>_devfs_ruleset="devfsrules_jail_<jail name>"


    Restart the jail and step in:
    # /etc/rc.d/jail restart <jail name>
    # jexec <jail ID> su

    In the jail, make sure the TUN interface shows up configured and ready for use by OpenVPN:
    Code:
    % ifconfig
    tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
            options=80000<LINKSTATE>
            inet 10.8.0.1 --> 10.8.0.2 netmask 0xffffffff
    


    Finally, a very crucial option needs to be added to the OpenVPN configuration:
    # echo 'ifconfig-noexec' >> /path/to/openvpn/server.conf

    To start the server...:
    # openvpn /path/to/openvpn/server.conf

    ...on every jail boot:
    # vi /etc/rc.conf

    Add the following lines, but change the path to your configuration file:
    Code:
    openvpn_enable="YES"
    openvpn_configfile="/path/to/openvpn/server.conf"
    


    From the host, you should now be able to see the jail listening on port 1194:
    Code:
    # netstat -anf inet
    [...]
    udp4       0      <jail IP>.1194       *.*
    [...]
    


    A successful port scan from a remote host could look something like this:
    Code:
    # nmap -sU <jail IP> -p1194-1195
    [...]
    1194/udp open|filtered openvpn
    1195/udp closed unknown
    [...]
    




    Troubleshooting
    1. When following the above steps, take careful notice on possible differences in path and device names, jail name and IP, devfs rule names and ID's.

    2. Make sure the TUN device exists on the host and in the jail:
      % ls -l /dev/tun*

    3. Start off with a basic OpenVPN settings. Example:
      Code:
      local <jail IP>
      port 1194
      proto udp
      dev tun0
      server 10.8.0.0 255.255.255.0
      ca /path/to/ca.crt
      cert /path/to/server.crt
      key /path/to/server.key
      dh /path/to/dh2048.pem
      keepalive 10 120
      comp-lzo
      user nobody
      group nobody
      persist-key
      persist-tun
      ifconfig-pool-persist /var/tmp/openvpn.pool
      status /var/tmp/openvpn.status
      log-append /var/log/openvpn.log
      verb 4
      mute 20
      ifconfig-noexec
      

    4. When asking for help include:
      • configuration files (host and jail rc.conf, OpenVPN config)
      • output from ifconfig, netstat -r and netstat -anf inet.
      • information on firewalling etc.


    References
    1. The Handbook: 15.6 Application of Jails, http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails-application.html
    2. WWW: OpenVPN HOWTO, http://openvpn.net/index.php/open-source/documentation/howto.html
    3. Man page: ifconfig(8), see 'create'.
    4. Man page: rc.conf(5), see 'cloned_interfaces'.
    5. Mail: isc-dhcp3-server in a jail?, David N, http://lists.freebsd.org/pipermail/freebsd-questions/2007-June/151008.html
    6. Mail: dhcpd possible within jail?, Bjoern A. Zeeb, http://lists.freebsd.org/pipermail/freebsd-jail/2008-November/000579.html
    7. Man page: rc.conf(5), see 'jail_⟨jname⟩_ip_multi⟨n⟩'.
    8. WWW: Linux VServer FAQs, http://linux-vserver.org/Frequently_Asked_Questions#Can_I_run_an_OpenVPN_Server_in_a_guest.3F
     
    Dink and ghostcorps thanked for this.
  2. ghostcorps

    ghostcorps New Member

    Messages:
    295
    Thanks Received:
    1
    Hi :)

    This is exactly what I am looking for!!! :D

    But I get a bit confused here:

    I have OpenVPN working perfectly on the host, I can ssh into it etc, but I can not see Webserver (the jail) once on the VPN. What else do I need to add to rc.conf to allow this?

    Here are the current configurations upto this point, as requested:

    Host
    /etc/defaults/devfs.rules
    Code:
    # Support for TUN devices
    #
    [devfsrules_unhide_tun=4]
    add path tun0 unhide
    
    # Devices usually found in a jail.
    #
    [devfsrules_jail=5]
    add include $devfsrules_hide_all
    add include $devfsrules_unhide_basic
    add include $devfsrules_unhide_login
    add include $devfsrules_unhide_tun
    


    Host
    /etc/rc.conf
    Code:
    hostname="my.domain.com"
    sshd_enable="YES"
    ifconfig_em0="inet xxx.xxx.xxx.xx2 netmask 255.255.255.248"
    defaultrouter="xxx.xxx.xxx.xx9"
    gateway_enable="YES"
    firewall_enable="YES"
    firewall_script="/etc/ipfw.rules"
    firewall_type="closed"
    openvpn_enable="YES"
    openvpn_configfile="/usr/local/etc/openvpn/server.conf"
    openvpn_if="tap"
    ...
    cloned_interfaces="tun"
    jail_set_hostname_allow="NO"
    jail_enable="YES"
    jail_list="webserver"
    jail_interface="em0"
    jail_devfs_enable="YES"
    jail_procfs_enable="YES"
    jail_webserver_rootdir="/usr/gaols/webserver"
    jail_webserver_interface="em0"
    jail_webserver_hostname="webserver"
    jail_webserver_ip="xxx.xxx.xxx.xx3"
    jail_webserver_devfs_ruleset="devfsrules_jail"
    


    Host
    /etc/hosts
    Code:
    ::1                     localhost localhost.my.domain
    127.0.0.1               localhost localhost.my.domain my my.domain.com
    xxx.xxx.xxx.xx2         my my.domain.com
    192.168.254.1           host
    192.168.254.2           webserver
    


    Host
    /usr/local/etc/openvpn/server.conf
    Code:
    port 1194
    proto udp
    dev tap
    ca /usr/local/etc/openvpn/keys/ca.crt
    cert /usr/local/etc/openvpn/keys/server.crt
    key /usr/local/etc/openvpn/keys/server.key # This file should be kept secret
    dh /usr/local/etc/openvpn/keys/dh1024.pem
    server 192.168.254.0 255.255.255.0
    ifconfig-pool-persist ipp.txt
    peer-to-peer
    keepalive 10 120
    comp-lzo
    max-clients 10
    user root
    persist-key
    persist-tun
    status /var/log/openvpn-status.log
    log-append  /var/log/openvpn.log
    verb 3
    


    Host
    #ifconfig
    Code:
    # ifconfig
    em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
            options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
            ether xx:xx:xx:xx:xx:xx
            inet xxx.xxx.xx2 netmask 0xfffffff8 broadcast xxx.xxx.xx5
            inet xxx.xxx.xx3 netmask 0xffffffff broadcast xxx.xxx.xx3
            inet 1xxx.xxx.xx4 netmask 0xffffffff broadcast xxx.xxx.xx4
            media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    
    ...
    
    tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
            options=80000<LINKSTATE>
            ether 00:bd:8e:20:00:00
            inet 192.168.254.1 netmask 0xffffff00 broadcast 192.168.254.255
            Opened by PID 20612
    tun0: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
            options=80000<LINKSTATE>
    


    Host
    # netstat -r
    Code:
    # netstat -r
    Routing tables
    
    Internet:
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            174.137.139.209    UGS         2    13748    em0
    localhost          link#5             UH          0        4    lo0
    xxx.xxx.xxx.xx8/29 link#1             U           0        0    em0
    domain             link#1             UHS         0      222    lo0
    xxx.xxx.xxx.xx3    link#1             UHS         0     3673    lo0 =>
    xxx.xxx.xxx.xx3/32 link#1             U           0        0    em0
    xxx.xxx.xxx.xx4    link#1             UHS         0      769    lo0 =>
    xxx.xxx.xxx.xx4/32 link#1             U           0        0    em0
    192.168.254.0      link#6             U           1     1422   tap0
    host               link#6             UHS         0        0    lo0
    
    Internet6:
    Destination        Gateway            Flags      Netif Expire
    localhost          localhost          UH          lo0
    fe80::%lo0         link#5             U           lo0
    fe80::1%lo0        link#5             UHS         lo0
    ff01:5::           fe80::1%lo0        U           lo0
    ff02::%lo0         fe80::1%lo0        U           lo0
    


    Host
    # netstat -anf inet
    Code:
    # netstat -anf inet
    Active Internet connections (including servers)
    Proto Recv-Q Send-Q  Local Address          Foreign Address       (state)
    tcp4       0     52 192.168.254.1.22       192.168.254.3.3813     ESTABLISHED
    tcp4       0      0 192.168.254.1.22       *.*                    LISTEN
    tcp4       0      0 xxx.xxx.xxx.xx3.5080   *.*                    LISTEN
    tcp4       0      0 xxx.xxx.xxx.xx3.1935   *.*                    LISTEN
    tcp4       0      0 xxx.xxx.xxx.xx3.10085  *.*                    LISTEN
    tcp4       0      0 xxx.xxx.xxx.xx3.9999   *.*                    LISTEN
    tcp4       0      0 xxx.xxx.xxx.xx3.443    *.*                    LISTEN
    tcp4       0      0 xxx.xxx.xxx.xx3.80     *.*                    LISTEN
    tcp4       0      0 xxx.xxx.xxx.xx4.8080   *.*                    LISTEN
    tcp4       0      0 xxx.xxx.xxx.xx3.21     *.*                    LISTEN
    tcp4       0      0 192.168.254.1.22       192.168.254.4.1761     ESTABLISHED
    tcp4       0      0 xxx.xxx.xxx.xx3.21     *.*                    LISTEN
    udp4       0      0 xxx.xxx.xxx.xx2.2501   *.*
    udp4       0      0 xxx.xxx.xxx.xx4.514    *.*
    udp4       0      0 xxx.xxx.xxx.xx3.514    *.*
    udp4       0      0 xxx.xxx.xxx.xx3.514    *.*
    udp4       0      0 *.514                  *.*
    


    Host nb. Unfortunately, ipfw refuses to be 'closed' so I need to write in all of the closed rules, hence why it is so messy.
    # ipfw show
    Code:
    # ipfw show
    00010   258   23524 allow ip from any to any via lo0
    00011     0       0 allow ip from me to me
    00020     0       0 check-state
    00030    32    2926 allow icmp from any to any
    00060  2096  325045 allow ip from any to xxx.xxx.xxx.xx2 dst-port 1024-7999
    00070     0       0 allow ip from any to xxx.xxx.xxx.xx2 dst-port 8001-8088
    00080     0       0 allow ip from any to xxx.xxx.xxx.xx2 dst-port 8090-9998
    00090    13     960 allow ip from any to xxx.xxx.xxx.xx2 dst-port 10000-65535
    00100     0       0 allow ip from any to xxx.xxx.xxx.xx3 dst-port 1024-5079
    00110     0       0 allow ip from any to xxx.xxx.xxx.xx3 dst-port 5081-7999
    00120     0       0 allow ip from any to xxx.xxx.xxx.xx3 dst-port 8001-8087
    00120     0       0 allow ip from any to xxx.xxx.xxx.xx3 dst-port 8090-9998
    00120     7    1127 allow ip from any to xxx.xxx.xxx.xx3 dst-port 10000-65535
    00800     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx2 dst-port 1-1023
    00800     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 1-19
    00800     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 444-1934
    00810     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 22-79
    00810     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 1936-1023
    00820     0       0 deny log logamount 5 ip from any to xxx.xxx.xxx.xx3 dst-port 81-442
    00830     0       0 deny ip from any to 1xxx.xxx.xxx.xx2 dst-port 8089
    00840     0       0 deny ip from any to xxx.xxx.xxx.xx2 dst-port 9999
    00950     0       0 deny ip from any to xxx.xxx.xxx.xx2 dst-port 8000
    01010     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 5080
    01020     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 8088
    01030     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 8089
    01040     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 8000
    01050     0       0 deny ip from any to xxx.xxx.xxx.xx3 dst-port 9999
    02000     0       0 deny ip from any to xxx.xxx.xxx.xx4
    65535 36436 5479174 allow ip from any to any
    


    Webserver (jail)
    /etc/rc.conf
    Code:
    hostname="my.other.domain.com"
    ifconfig_em0="inet xxx.xxx.xxx.xx3 netmask 255.255.255.255"
    defaultrouter="xxx.xxx.xxx.xx9"
    ...
    kern_securelevel_enable="YES"
    kern_securelevel="3"
    portmap_enable="NO"
    log_in_vain="YES"
    pureftpd_enable="YES"
    red5_enable="YES"
    openvpn_enable="YES"
    openvpn_configfile="/usr/local/etc/openvpn/client.conf"
    openvpn_if="tap"
    
     
  3. Schnigges

    Schnigges New Member

    Messages:
    1
    Thanks Received:
    0
    First of all, thanks for the great tutorial!
    Do you have any hints on how to do the setup for a bridged openvpn? I guess I would have to create a tap device on the host instead of a tun, but I am having trouble creating one (not entirely sure if this would even be the right way). When I try
    Code:
    ifconfig tap0 create 192.168.178.96 up
    

    I get the following error:
    Code:
    ifconfig: SIOCIFCREATE2: Invalid argument
     
  4. Mattias

    Mattias New Member

    Messages:
    13
    Thanks Received:
    2
    I'm unable to recreate that error, but ifconfig's stderr isn't to be blindly trusted (see example below) :) Maybe try resetting your config related to devices (devfs.rules, rc.conf) as much as possible and make sure the device is destroyed between every try.

    Code:
    # ifconfig tap0 create 1.2.3,4 [notice the comma]
    ifconfig: 1.2.3,4: bad value
    
    # ifconfig tap0 create 1.2.3.4 [comma corrected]
    ifconfig: create: bad value [create is a bad value]
    
    # ifconfig tap0
    tap0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    [...]
    
     
  5. lockdoc

    lockdoc New Member

    Messages:
    124
    Thanks Received:
    6
    I also have a problem on this.

    I am using tun1, as tun0 is for my internet connection. From the openvpn.log on the jail side I get the following error:
    Code:
    Tue Mar  6 13:33:45 2012 us=282808   mode = 1
    Tue Mar  6 13:33:45 2012 us=282840   show_ciphers = DISABLED
    Tue Mar  6 13:33:45 2012 us=282869   show_digests = DISABLED
    Tue Mar  6 13:33:45 2012 us=282897   show_engines = DISABLED
    Tue Mar  6 13:33:45 2012 us=282924   genkey = DISABLED
    Tue Mar  6 13:33:45 2012 us=283012   key_pass_file = '[UNDEF]'
    Tue Mar  6 13:33:45 2012 us=283041   show_tls_ciphers = DISABLED
    Tue Mar  6 13:33:45 2012 us=283069 Connection profiles [default]:
    Tue Mar  6 13:33:45 2012 us=283098   proto = udp
    Tue Mar  6 13:33:45 2012 us=283125   local = '[UNDEF]'
    Tue Mar  6 13:33:45 2012 us=283150   local_port = 1194
    Tue Mar  6 13:33:45 2012 us=283176   remote = '[UNDEF]'
    Tue Mar  6 13:33:45 2012 us=283201   remote_port = 1194
    Tue Mar  6 13:33:45 2012 us=283226   remote_float = DISABLED
    Tue Mar  6 13:33:45 2012 us=283293   bind_defined = DISABLED
    Tue Mar  6 13:33:45 2012 us=283318   bind_local = ENABLED
    Tue Mar  6 13:33:45 2012 us=283397 NOTE: --mute triggered...
    Tue Mar  6 13:33:45 2012 us=283429 194 variation(s) on previous 20 message(s) suppressed by --mute
    Tue Mar  6 13:33:45 2012 us=283461 OpenVPN 2.2.2 amd64-portbld-freebsd9.0 [SSL] [LZO2] [eurephia] built on Mar  6 2012
    openvpn: writing to routing socket: No such process
    Tue Mar  6 13:33:45 2012 us=283727 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
    Tue Mar  6 13:33:45 2012 us=287101 Diffie-Hellman initialized with 1024 bit key
    Tue Mar  6 13:33:45 2012 us=287865 TLS-Auth MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
    Tue Mar  6 13:33:45 2012 us=287916 Socket Buffers: R=[42080->65536] S=[9216->65536]
    openvpn: writing to routing socket: No such process
    openvpn: writing to routing socket: No such process
    Tue Mar  6 13:33:45 2012 us=288149 ROUTE: default_gateway=UNDEF
    Tue Mar  6 13:33:45 2012 us=288222 TUN/TAP device /dev/tun1 opened
    openvpn: writing to routing socket: No such process
    Tue Mar  6 13:33:45 2012 us=288346 /sbin/route add -net 10.8.0.0 10.8.0.2 255.255.255.0
    route: writing to routing socket: Operation not permitted
    Tue Mar  6 13:33:45 2012 us=289457 ERROR: FreeBSD route add command failed: external program exited with error status: 1
    Tue Mar  6 13:33:45 2012 us=289550 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
    Tue Mar  6 13:33:45 2012 us=289954 UDPv4 link local (bound): [undef]:1194
    Tue Mar  6 13:33:45 2012 us=289998 UDPv4 link remote: [undef]
    Tue Mar  6 13:33:45 2012 us=290028 MULTI: multi_init called, r=256 v=256
    Tue Mar  6 13:33:45 2012 us=290128 IFCONFIG POOL: base=10.8.0.4 size=62
    Tue Mar  6 13:33:45 2012 us=290166 IFCONFIG POOL LIST
    Tue Mar  6 13:33:45 2012 us=290247 Initialization Sequence Completed
    


    The client side gives me the following:
    Code:
    Tue Mar 06 14:39:18 2012 us=421000 OpenVPN 2.2.1 Win32-MSVC++ [SSL] [LZO2] built on Jul  1 2011
    Tue Mar 06 14:39:18 2012 us=421000 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
    Tue Mar 06 14:39:18 2012 us=750000 LZO compression initialized
    Tue Mar 06 14:39:18 2012 us=750000 Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
    Tue Mar 06 14:39:18 2012 us=750000 Socket Buffers: R=[8192->8192] S=[8192->8192]
    Tue Mar 06 14:39:18 2012 us=750000 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
    Tue Mar 06 14:39:18 2012 us=765000 Local Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-client'
    Tue Mar 06 14:39:18 2012 us=765000 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-server'
    Tue Mar 06 14:39:18 2012 us=765000 Local Options hash (VER=V4): '41690919'
    Tue Mar 06 14:39:18 2012 us=765000 Expected Remote Options hash (VER=V4): '530fdded'
    Tue Mar 06 14:39:18 2012 us=765000 UDPv4 link local: [undef]
    Tue Mar 06 14:39:18 2012 us=765000 UDPv4 link remote: 192.168.0.5:1194
    Tue Mar 06 14:39:25 2012 write UDPv4: Interrupted system call (WSAEINTR) (code=10004)
    Tue Mar 06 14:39:33 2012 us=296000 write UDPv4: Interrupted system call (WSAEINTR) (code=10004)
    


    Is the problem with not being able to add routes?
     
  6. LJackson

    LJackson New Member

    Messages:
    1
    Thanks Received:
    0
    In general, these instructions do appear to work with a TAP device, and without using OpenVPN's client-config-dir as discussed on http://forums.freebsd.org/showthread.php?t=24063&highlight=jail+openvpn. First, placing if_tap_load="YES" in /boot/loader.conf will prevent situations where all the interfaces appeared to be configured correctly, but the OpenVPN client stalls with errors like "Connection Reset by Peer".

    Ensure OpenVPN can use the TAP device (the last two lines are key):

    Code:
    # sysctl -a | grep link.tap
    net.link.tap.debug: 0
    net.link.tap.devfs_cloning: 1
    net.link.tap.up_on_open: 1
    net.link.tap.user_open: 1


    Next, setup the bridge and configure the interfaces. The following applies to /etc/rc.conf. My host has a re0 interface (10.0.0.70), is running Samba, and has OpenVPN in a jail addressed as 10.0.0.74.

    Code:
    cloned_interfaces="tap0 bridge0"
    ifconfig_bridge0="addm re0 addm tap0 up"
    ifconfig_re0="up"                         # All interfaces must be up
    ifconfig_tap0="up"
    
    ipv4_addrs_bridge0="10.0.0.70/24" # Host IP (Samba), 74 is on re0
    
    ## Note: ifconfig_bridge0 should work in place of ipv4_addrs_bridge0
    
    ifconfig_re0="10.0.0.74/24"       # OpenVPN's IP
    


    I found placing OpenVPN's IP on the bridge as an alias (ifconfig_bridge0_alias0="inet 10.0.0.74 netmask 255.255.255.255") yielded mixed results: using ssh to contact the host would still work, but Samba wouldn't. Placing both on re0 also appeared to work, but goes against convention. (If the host is to have an IP address it should go on the host. See http://www.freebsd.org/doc/handbook/network-bridging.html.) Making the TAP device sticky is interesting. (See the aforementioned handbook.)

    The TAP device is set using either

    Code:
    ifconfig_tap0="10.8.0.50 netmask 255.255.255.0"


    or
    Code:
    jail_<OpenVPN>_ip_multi0="tap0|10.8.0.50/24"


    In my case, both appeared to work, but the latter is preferred for consistency.

    Depending on how you are using your host and the bridge the gateway_enable="YES" in /etc/rc.conf may not be required.

    As reported for the TUN device http://forums.freebsd.org/showthread.php?t=27934&highlight=openvpn+jail, I did find if OpenVPN is stopped the IP address assigned to the TAP device is removed. I wouldn't have thought that is possible, maybe something to do with the sysctl values?

    My OpenVPN server.conf file looks like:

    Code:
    server-bridge 10.8.0.50 255.255.255.0 10.8.0.51 10.8.0.99
    
    proto udp
    dev tap
    
    ca /usr/local/etc/openvpn/keys/ca.crt
    cert /usr/local/etc/openvpn/keys/server.crt
    key /usr/local/etc/openvpn/keys/server.key
    dh /usr/local/etc/openvpn/keys/dh1024.pem
    tls-auth /usr/local/etc/openvpn/keys/ta.key 0
    
    client-to-client
    
    ifconfig-noexec
    
    push "dhcp-option DNS 10.0.0.1" 
    push "dhcp-option WINS 10.0.0.70"
    push "dhcp-option NBT 8"
    
    mssfix 1450
    
    ping 10
    ping-restart 120
    push "ping 10"
    push "ping-restart 60"
    push "route 10.0.0.0 255.255.255.0" # Advertise Server's LAN to VPN clients
    
    comp-lzo
    user nobody
    group nobody
    
    status /var/log/openvpn/openvpn-status.log
    log-append /var/log/openvpn/openvpn.log
    verb 3 
    


    and the client file for a Microsoft XP system and a MacBook Pro OS X Lion looks like (The host is also acting as a Time Machine Server.)

    Code:
    client
    dev tap
    proto udp
    remote <yours.com> 1194
    
    tls-client
    
    nobind
    
    ca <your ca.crt>
    cert <your client.crt>
    key <your client.key>
    tls-auth <your ta.key> 1
    
    tun-mtu 1500
    tun-mtu-extra 32
    mssfix 1450
    
    pull
    
    comp-lzo
    verb 3
    
     
  7. Dink

    Dink New Member

    Messages:
    1
    Thanks Received:
    0
    Thank you very much. Nice post.

    Got my OpenVPN up and running using jail.
     
  8. dvl@

    dvl@ Member Developer

    Messages:
    424
    Thanks Received:
    18
    Why alter /etc/defaults/devfs.rules? Why not change /etc/devfs.rules instead? That's how this is supposed to work.
     
  9. junovitch

    junovitch Moderator Staff Member Moderator

    Messages:
    863
    Thanks Received:
    201
    Good day. I have a few additions to toss into the mix as I had a few issues getting this to work as intended on 9.1-RELEASE. I hope this helps anybody in the same situation.

    • Issue: The tunnel was not recognized in the jails initially. The devfs.rules were correct. This appeared to be because I was already using OpenVPN on the host and something about the way it was created prevented it from being shown in a jail.
    • Fix: Delete the current tunnel and create it again.
    Code:
    ifconfig tun0 destroy
    ifconfig tun create
    


    • Issue: Routing was needed to access anything after connecting.
    • Fix: Add it to the host.
    # vi /etc/rc.conf
    Code:
    cloned_interfaces="tun"
    gateway_enable="YES"
    


    • Issue: The /etc/defaults/devfs.rules may get updated during system upgrades.
    • Fix: Modify the /etc/devfs.rules over the default. On 9.1-RELEASE there are four rules by default in /etc/defaults/devfs.rules so I used rule 5 with a single combined rule.
    # vi /etc/devfs.rules
    Code:
    [devfsrules_jail_with_vpn=5]
    add include $devfsrules_hide_all
    add include $devfsrules_unhide_basic
    add include $devfsrules_unhide_login
    add path tun0 unhide
    add path zfs unhide
    


    • Issue: My clients could connect but not ping anything.
    • Fix: A prestart routine was used to add the route on the host upon jail start.
    Code:
    jail_<jail name>_ip_multi0="tun0|10.8.0.1 10.8.0.2 netmask 255.255.255.255"
    jail_<jail name>_exec_prestart0="/sbin/route add -net 10.8.0.0/24 10.8.0.2"
    jail_<jail name>_devfs_enable="YES"
    jail_<jail name>_devfs_ruleset="devfsrules_jail_with_vpn"
    


    • Issue: OpenVPN couldn't add routes inside the jail. This had already been solved by the prestart routine.
    • Fix: Add both noexec options to the OpenVPN configuration.
    # echo 'ifconfig-noexec' >> /path/to/openvpn/server.conf
    # echo 'route-noexec' >> /path/to/openvpn/server.conf
     
  10. dvl@

    dvl@ Member Developer

    Messages:
    424
    Thanks Received:
    18
    What happens if you restart OpenVPN?
     
  11. junovitch

    junovitch Moderator Staff Member Moderator

    Messages:
    863
    Thanks Received:
    201
    Unfortunately restarting OpenVPN will trigger the tunnel to lose its IP address. As it cannot add it back from inside the jail the only option when I have to restart OpenVPN is to restart the whole jail. Not ideal but it's not a deal breaker as the jail is dedicated to OpenVPN and I do control the host after all.

    Your post regarding jaildaemon monitoring is probably the way to go. For anybody out there who can't just restart the jail whenever they have to restart OpenVPN, I would suggest taking a look at this post and links within for a potential solution.

    http://forums.freebsd.org/showpost.php?p=231691&postcount=4
     
  12. dvl@

    dvl@ Member Developer

    Messages:
    424
    Thanks Received:
    18
    Sorry, I thought I'd already posted that one here... Thanks.
     
  13. FreeBSDNewUser

    FreeBSDNewUser New Member

    Messages:
    6
    Thanks Received:
    0
    Hello Junovitch,

    Thanks for the help can you let me know where are these rules supposed to go rc.conf file on the host?
    Code:
    jail_<jail name>_ip_multi0="tun0|10.8.0.1 10.8.0.2 netmask 255.255.255.255"
    jail_<jail name>_exec_prestart0="/sbin/route add -net 10.8.0.0/24 10.8.0.2"
    jail_<jail name>_devfs_enable="YES"
    jail_<jail name>_devfs_ruleset="devfsrules_jail_with_vpn"
    


    Also this link does not work:
    I would really appreciate some help on this is anyone willing to create a post with instructions from start to finish how they got VPN working inside a jail? I would be very grateful as I cannot seem to get it to work but I believe it is because I am missing ip_mulit0 because I am unsure where it goes. Does it matter what the jail ip is? and does that need to go in the openvpn.conf file? I think I may also have an issue in the VPN conf.
     
  14. junovitch

    junovitch Moderator Staff Member Moderator

    Messages:
    863
    Thanks Received:
    201
    The link was on the old forums. I found the post again in Thread 41691. The two links mentioned:
    https://elektropost.org/ezjail/msg00982.html
    http://erdgeist.org/arts/software/jaildaemon/

    The jail_<jail_name>_* variables were the rc.conf style of jails. Ezjail happens to keep all these in /usr/local/etc/ezjail for each jail. If you are using sysutils/ezjail that would still apply. If not, it would help to know how you are setting up jails. I would imagine the jail.conf format would be different but the concept would be the same.

    I'm not using OpenVPN in a jail anymore. I used it for a while because I had a old router that wasn't up to the job. After I replaced my router I moved the VPN duties to that device. It's still FreeBSD but I didn't jail it this time. It's just easier to manage that way. All my notes from when I did run OpenVPN in a jail I left in my notes. If you do see any inaccuracies let me know. I'll fix them up. Hope that helps.

    https://github.com/junovitch/my-freebsd-build/blob/master/openvpn-jail-HOWTO
     
    FreeBSDNewUser thanks for this.
  15. FreeBSDNewUser

    FreeBSDNewUser New Member

    Messages:
    6
    Thanks Received:
    0
    Thanks a lot for the update I really appreciate it. I actually stumbled across the github link you have given here yesterday but have not have had a chance to try it out. I am very new to FreeBSD and am just wondering if I create a jail (not for the VPN) but run the VPN on the host will the jail use the tun interface or is there some way I need to set it up so that it uses the tun interface rather than the NIC card. I have my firewall setup to not allow any traffic if it is not over the VPN. I have tested and it works fine. The problem is the jail does not get out of the firewall because it is not using the tun interface and is bypassing VPN.

    Long story short I want all the traffic of the jail to go through VPN just as all the traffic on the host goes through the VPN. The VPN itself does not need to be in the jail is the github link the route that I should take to achieve this or is there an easier way. Also another question do I even need to use jails or can I just install all my packages in the Freebsd FreeBSD host system as the freebsd FreeBSD host system is actually in a chroot using finch. I am using NAS4FREE 9.2 embedded but I have finch installed and the FREEBSD 9.2 filesystem on a seperate ZFS.

    My shares and all my files are on a separate drive using a raid5 configuration. So if I do break anything in finch I can just format that drive and redo finch and all the packages. Is that a normal thing to do or should I stick with using jails inside the chroot? Sorry for all the questions.

    Thanks.
     
  16. FreeBSDNewUser

    FreeBSDNewUser New Member

    Messages:
    6
    Thanks Received:
    0
    I am using finch for managing the jails and qjail console "$jailname" is how I enter the jail. I am not sure if finch uses ezjails but I think it does and that might explain some of my problems with getting it working.

    Thanks
     
  17. junovitch

    junovitch Moderator Staff Member Moderator

    Messages:
    863
    Thanks Received:
    201
    By default, the jail would just use the host routing table. So if the VPN pushes a default route the jail traffic would leave over the tun interface. It shouldn't bypass the VPN as you mention if you are going with defaults. That would be the outbound.

    The likely problem is the inbound back to the jail probably doesn't have any config to get back. I am guessing that your host is the OpenVPN client right? The server side would need a route/iroute config keyword to get back to the address or network of the jail. Alternately you can configure your firewall to NAT the outbound traffic from jail IP addresses to use the IP of the tun interface.

    The VPN doesn't need to be in a jail and it only even makes sense going through the hassle if you are running it as a service. If you are a client then it's not worth the effort since it's not accepting any connections from the outside.

    However if you are already using Finch to isolate an environment I don't think it makes as much sense to add another layer into the mix to complicate things. Pick one method and stick to it or if you use both use them in parallel for different environments.

    An FYI, I've only looked at NAS4FREE briefly a couple years ago and sometimes advise doesn't apply. This is why we have a rule regarding threads on derivatives.
    https://forums.freebsd.org/threads/pc-bsd-freenas-nas4free-and-all-other-freebsd-derivatives.7290/

    Also, FreeBSD 9.2-RELEASE is also out of support so a reminder about sticking with an OS supported with security updates.
    https://forums.freebsd.org/threads/topics-about-unsupported-freebsd-versions.40469/
     
    FreeBSDNewUser thanks for this.
  18. FreeBSDNewUser

    FreeBSDNewUser New Member

    Messages:
    6
    Thanks Received:
    0
    You would think that by default it uses the tun0 interface but in fact it doesn't because when I ping I get "permission denied" and not "timed out" that means the firewall is blocking it because it is not using tun0. I think it would be my fault because qjail allows to change the network interface:
    Code:
    new_network_interface="re0"
    qjail config -c "$new_network_interface""$jailname"
    

    I could just explicitly change the network interface for the jail and see if that works then I will probably run into the "timed out" error when the ping goes out but cannot return.

    You guessed right the host is the OpenVPN client and I use Mullvad's VPN Servers. So a route/iroute on the server side is out of the question I think. How would I configure the firewall to NAT the outbound traffic from the jail IP address to use the IP of the tun interface? As far as I can tell all local traffic in 192.168.0.0 in going over the VPN.

    Ok so as you say the VPN doesn't need to be in a jail but so I will run it on the host machine or on finch. Inside finch I create a jail to install deluge and plex media server will they be able to run over the VPN that is already running on the host system or will there be issues? As you say it doesn't make much sense to add another layer i.e. a jail to complicate things. Can I just install all those packages directly on my finch chroot instead of creating a jail?

    I read the link and I will try some help on the NAS4FREE forums if I cant get it working. however technically inside finch is a complete FreeBSD 9.2 version of software so it uses the new package management system. I updated it and I realize that means I cannot use PKG_ADD anymore.

    I will read the other link with regards to version of FreeBSD but I used the version that comes with finch and did not upgrade it because was not sure if it would work with the older system of NAS4FREE. Anyways off to work now. Thanks for all the help by the way and I am sure come this weekend I should have it up and running.
     
  19. junovitch

    junovitch Moderator Staff Member Moderator

    Messages:
    863
    Thanks Received:
    201
    The permission denied was because a ping is a raw socket and that requires special permissions within a jail. Here's a helpful copy/paste from the Handbook to fill you in.

    https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails-ezjail.html

    The NAT/firewall bit is covered in the Handbook. That would be a good place to start. If you have the NAT rule just NAT everything then your jails should just work.
    https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls.html

    Regarding NAS4FREE, their websites lists there is a version 9.3 that is based on a supported release of FreeBSD 9.3.

    Since this has kind of strayed outside of running an OpenVPN server in a jail, I would recommend reviewing the links above and posting if you have more specific questions on what you are doing. Specifically it would be allowing the jail or finch environment on your system to seamlessly share the OpenVPN client on the host.
     
    FreeBSDNewUser thanks for this.
  20. FreeBSDNewUser

    FreeBSDNewUser New Member

    Messages:
    6
    Thanks Received:
    0
    I have allowed raw sockets and I can ping when I disable the firewall. When I enable the firewall I can no longer ping because the jail is not using the tun0 interface and if I change the interface it doesn't work I get a error in the jail.conf because the ip is not in the range for tun0 I will create a new jail with openvpn up and running and assign it an ip and netmask that will work with the tun0 interface. If I do this I believe the jail will not work over the LAN as it should because the ip is not in the LAN range but is in the range of the vpn ip. So this will probably cause more issues.

    I will read about the NAT/firewall as I am really bad in that area.

    I will also try to update my NAS4FREE Version after backing up the config .xml so that I have supported versions.

    Thanks for all your help I really appreciate it.