Can you bind amd (automount daemon) to a specific IP?

J

jimeha

Hi all,

I am setting up a number of jails on a server (OS spec's below) and would like to run amd (automount daemon) to auto mount/dismount one of our test lab's NFS shares. It works fine on the jail host, but I cannot find a way to bind it to a specific IP, so it can run in the jails. Can this be done or do I need to use nullfs mounts for this like you do for NFS mounts*?

* We needed eight NFS mounts to various servers in each jail, but discovered NFS is not a 'jail friendly' OS according to lsvfs (quote Google). I also had a failed attempt to add VFCF_JAIL to src/sys/nfsclient/nfs_vfsops.c to make NFS 'jail friendly'. I'm using nullfs mounts to the jail sub-directories to get around this now. It's ugly, but it works, but will be uglier when we have eight jails with eight mounts a piece; 64 nullfs mounts + 8 nfs mounts on the host.

* I would love to avoid the whole use of nullfs mounts and just let the jails have their own mounts as they please so feel free to pop in a hint if anyone knows of a way to do this. :)

* This is a segregated environment where security is essentially disabled for build and testing purposes.

Jail host:
- FreeBSD 8.1 RC2 amd64 (updated to RELENG_8_1 about 2 weeks ago)
- The Lab IP/hostname I want to bind amd to is 10.128.6.11 (build.test.name.com) for example
- The Corporate IP and hostname I don't need bound

Jail guest #1 for example:
- FreeBSD 6.1 i386 (updated to RELENG_6_1 about two weeks ago)
- The Lab IP/hostname I want to bind amd to is 10.128.6.12 (build2.test.name.com) for example
- The Corporate IP and hostname I don't need bound

Thank very much, Dylan
 
This looks like a good candidate to try:
localhost_address (string, default to localhost or 127.0.0.1)
Specify the name or IP address for amd(8) to use when connecting the sockets for the local NFS server and the RPC server. This defaults to 127.0.0.1 or whatever the host reports as its local address. This parameter is useful on hosts with multiple addresses where you want to force amd(8) to connect to a specific address.
Click to expand...

Taken from amd.conf(5)

As for NFS, rpc, mountd and nfsd can be bound to a specific IP address. It does at least on 8.x. For 6.x I'm not sure, I can indeed remember that this wasn't possible on an older version.
 
Thanks SirDice for the help. I added localhost_address to /etc/amd.conf (below) and while /var/log/all.log says 'My ip addr is 10.128.6.11' during amd's startup (below), sockstat still shows amd to be listening on *:<635,904,1022,& 1023> instead of 10.128.6.11:<635,904,1022,& 1023>.

I did successfully get nfsd, mountd, and rpcbind to listen on 10.128.6.11 the other day so they could run bound to the jail IP as well, but am unable to NFS mount anything because NFS is not a jail-friendly fs according to lsvfs. Running 'mount -a' to attempt to mount NFS mounts in the jail's /etc/fstab, just results in 'Operation not permitted' as root in the jail with the mount sysctl tweaked to 1 on the jail host. I'm abandoning this for now since the nullfs mounts are working; ugly, but working. :) I imagine there's a FreeBSD PR ticket to add NFS mount functionality to jails, probably as another security.jail sysctl, but I'm terrible at using the PR system to find anything.

So for now I'll focus on getting amd working, but I think I may be barking up the same 'no nfs mounts in jails' tree.

In /var/log/all.log:
Code: 
Aug  4 16:30:55 build amd[96535]: Copyright (c) 1997-2006 Erez Zadok
Aug  4 16:30:55 build amd[96535]: Copyright (c) 1990 Jan-Simon Pendry
Aug  4 16:30:55 build amd[96535]: Copyright (c) 1990 Imperial College of Science, Technology & Medicine
Aug  4 16:30:55 build amd[96535]: Copyright (c) 1990 The Regents of the University of California.
Aug  4 16:30:55 build amd[96535]: am-utils version 6.1.5 (build 801000).
Aug  4 16:30:55 build amd[96535]: Report bugs to [url]https://bugzilla.am-utils.org/[/url] or [email]am-utils@am-utils.org[/email].
Aug  4 16:30:55 build amd[96535]: Configured by David O'Brien <obrien@FreeBSD.org> on date 4-December-2007 PST.
Aug  4 16:30:55 build amd[96535]: Built by [email]root@build.test.name.com[/email].
Aug  4 16:30:55 build amd[96535]: cpu=amd64 (little-endian), arch=amd64, karch=amd64.
Aug  4 16:30:55 build amd[96535]: full_os=freebsd8.1, os=freebsd8, osver=8.1, vendor=undermydesk,distro=The FreeBSD Project.
Aug  4 16:30:55 build amd[96535]: domain=test.name.com, host=build, hostd=build.test.name.com.
Aug  4 16:30:55 build amd[96535]: Map support for: root, passwd, union, nis, ndbm, file, exec, error.
Aug  4 16:30:55 build amd[96535]: AMFS: nfs, link, nfsx, nfsl, host, linkx, program, union, ufs, cdfs,
Aug  4 16:30:55 build amd[96535]:       pcfs, auto, direct, toplvl, error, inherit.
Aug  4 16:30:55 build amd[96535]: FS: cd9660, nfs, nfs3, nullfs, msdosfs, ufs, unionfs.
Aug  4 16:30:55 build amd[96535]: Network 1: wire="192.168.221.0" (netnumber=192.168.221).
...<other 'network numbers' of jail IPs aliased on host interface>
Aug  4 16:30:55 build amd[96535]: localhost_address "10.128.6.11" requested, using 10.128.6.11
Aug  4 16:30:55 build amd[96535]: My ip addr is 10.128.6.11
Aug  4 16:30:55 build amd[96536]: released controlling tty using setsid()
Aug  4 16:30:55 build amd[96536]: file server localhost, type local, state starts up
Aug  4 16:30:55 build amd[96536]: creating mountpoint directory '/amdmnt'
Aug  4 16:30:55 build amd[96537]: /amdmnt: disabling nfs congestion window
Aug  4 16:30:55 build amd[96536]: initializing amd.conf map /etc/amd.amdmnt.map of type file
Aug  4 16:30:55 build amd[96536]: first time load of map /etc/amd.amdmnt.map succeeded
Aug  4 16:30:55 build amd[96536]: /etc/amd.amdmnt.map mounted fstype toplvl on /amdmnt
Aug  4 16:30:55 build amd[96536]: /amdmnt set to never timeout

$ sockstat|grep amd:
Code: 
root     amd        96536 3  dgram  -> /var/run/logpriv
root     amd        96536 4  udp4   *:1023                *:*
root     amd        96536 5  udp4   *:1022                *:*
root     amd        96536 6  tcp4   *:635                 *:*
root     amd        96536 7  udp4   *:904                 *:*

$ grep localhost /etc/amd.conf
Code: 
localhost_address =     10.128.6.11
$ ps -auxww|grep amd
Code: 
root 96536  0.0  0.0  6992  1856  ??  Is    4:30PM   0:00.00 /usr/sbin/amd -p -F /etc/amd.conf
root 96543  0.0  0.0  9092  1428   1  S+    4:31PM   0:00.00 grep amd
 
Solved

I found a workaround to this issue using a nullfs mount to my amd mounted file-system; same format as my nullfs mounts for my NFS mounted file-systems.

- I added this to my build2 jail's fstab stored on the jail host build1 (/etc/fstab.build2):
Code: 
/amdmnt/home  /d2/jails/build2/amdmnt/home  nullfs rw,nosuid 0 0

- The jail auto-magically picks up the actual amd auto_dir mount directory when the jail is started:
Code: 
[root@build2 ~]# mount|grep home
/.amd_mnt/lab-home1/exports/home on /amdmnt/home (nullfs, nosuid)

- Build1's /etc/amd.conf looks like this:
Code: 
[ global ]
...
auto_dir =              /.amd_mnt
...
[ /amdmnt ]
map_name =              amd.amdmnt.map

- Build2 doesn't need any /etc/amd.conf or amd /etc/rc.conf entries (i.e. amd_enable) at all to make this work.

Thanks for the hints!
 
I'm just adding this in case anyone is curious about other work-arounds to this issue not involving nullfs mounts. As stated in my original post I had found a forum post on Google stating you could add VFCF_JAIL to src/sys/nfsclient/nfs_vfsops.c on the jail host and it does in fact work. I had tried and failed to do it right the first time, hence this thread. :) However, all sorts of strange mount behaviour occurs as a result of this.

Here is a link to part of the original post: http://markmail.org/message/xwcsyupq6v3ux37n

Brief Instructions:
  1. Obtain FreeBSD sources
  2. Edit src/sys/nfsclient/nfs_vfsops.c so the line with:
    Code: 
    VFS_SET(nfs_vfsops, nfs, VFCF_NETWORK);
    looks like:
    Code: 
    VFS_SET(nfs_vfsops, nfs, VFCF_NETWORK|VFCF_JAIL);
  3. Re-build and install world: make buildworld; make installworld; reboot
  4. The output of 'lsvfs' should now show 'nfs' as 'network, jail'
  5. Construct your jail(s) and add the desired NFS mounts to the jail's fstab*

Gentle security warning!
  • Adding the nfs mount to the jail's /etc/fstab or just using 'mount -t nfs...' will result in the nfs mount appearing in the jail host's 'mount' ouptput as if is was mounted on the host (i.e. under /mnt) but will only be accessible in the jail's /mnt directory (i.e. /jails/jailname/mnt)! It's weird, but yes, you read that correctly. As a result, the NFS mounts do not appear in the output of 'mount' in the jail, if the jail host's /mnt isn't a sub-directory in the jail. The jail host's /mnt is unaffected by this though.
  • However, adding the nfs mount to the jail's fstab on the jail host, for example, jailhost:/etc/fstab.jailname requires the full path to the jail mount (i.e. /jails/jailname/mnt) and will appear properly in the output of 'mount' in the jail.
  • The NFS mount cannot be unmounted with 'umount' inside the jail it just fails with 'Invalid Argument'. The mount can only be unmounted from the jail host using either, for example, 'umount /mnt' or 'umount /jail/jailname/mnt'. Either command will unmount the mount created in the jail and the jail host's /mnt will be unaffected.
  • This likely opens up a slew of bugs, issues, and security holes, as noted by the above strange behaviour, so this should be used only as a last resort work-around where nullfs mounts cannot be used for some reason.
  • In the cases where security isn't a concern, using the nullfs mount solution in my previous post is, in my opinion, a much safer and more functional solution.

Enjoy!
 
umount

Add
Code: 
security.jail.enforce_statfs=1
to /etc/sysctl.conf (sysctl.conf(5)) before starting up the jail and that should allow you to use umount(8) within the jail.

Also, I'm not certain of this, but I think it's also prudent to modify sys/fs/nfsclient/nfs_clvfsops.c in the same manner as sys/nfsclient/nfs_vfsops.c
 
/etc/fstab.jailname

If you put the NFS mount entries into the master-host's /etc/fstab.jailname file, there is no need to patch the kernel to make NFS `jail-friendly'.

As of FreeBSD-8.1, simply putting the following into /etc/rc.conf on the master-host will allow you to perform any mount entered into /etc/fstab.jailname and the kicker is that /etc/rc.d/jail will automatically perform those mounts when the jail is started AND will automatically unmount those mounts when the jail is stopped:

Code: 
jail_jailname_mount_enable=YES

NOTE: Despite similarities, the above has no effect on the sysctl MIB jail.security.mount_allowed ... rather the above is a boolean that tells /etc/rc.d/jail to mount the entries in /etc/fstab.jailname prior to a start and to umount the same on a stop.

However, if you want to perform the NFS mount from WITHIN the jail, you will need the above kernel patch AND the following in /etc/sysctl.conf:

Code: 
security.jail.mount_allowed=1

And if you want to allow umount'ing of `jail-friendly' vfs-types from within the jail, also add:

Code: 
security.jail.enforce_statfs=0
 
devinteske said:
Add
Code: 
security.jail.enforce_statfs=1
to /etc/sysctl.conf (sysctl.conf(5)) before starting up the jail and that should allow you to use umount(8) within the jail.

Also, I'm not certain of this, but I think it's also prudent to modify sys/fs/nfsclient/nfs_clvfsops.c in the same manner as sys/nfsclient/nfs_vfsops.c
Click to expand...

Oops, that was
Code: 
security.jail.enforce_statfs=0
(fingers faster than mind in the post-lunch food-coma)
 
dmartin said:
  1. Adding the nfs mount to the jail's /etc/fstab or just using 'mount -t nfs...' will result in the nfs mount appearing in the jail host's 'mount' ouptput as if is was mounted on the host (i.e. under /mnt) but will only be accessible in the jail's /mnt directory (i.e. /jails/jailname/mnt)! It's weird, but yes, you read that correctly. As a result, the NFS mounts do not appear in the output of 'mount' in the jail, if the jail host's /mnt isn't a sub-directory in the jail. The jail host's /mnt is unaffected by this though.
  2. The NFS mount cannot be unmounted with 'umount' inside the jail it just fails with 'Invalid Argument'. The mount can only be unmounted from the jail host using either, for example, 'umount /mnt' or 'umount /jail/jailname/mnt'. Either command will unmount the mount created in the jail and the jail host's /mnt will be unaffected.
Click to expand...

I've verified that
Code: 
security.jail.enforce_statfs=0
allows you to BOTH see the mount AND umount the mount. However, beware that when this is set in sysctl.conf(5) on the master-host, the jail can see *ALL* the mounts from the master-host. This is potentially dangerous (but in our case, necessary).

Here's how you can verify for yourself...

0. Execute (on master-host) # sysctl security.jail.enforce_statfs (should be non-zero)

1. Execute # jls to get JID of jail

2. Execute (as root) # jexec JID tcsh to "log into" the jail

3. Execute # df (should show restricted output for the jail)

4. Exit the jail (`exit' will do)

5. Execute (as root) # jexec JID sh /etc/rc.shutdown

6. Execute (as root) # /etc/rc.d/jail stop jailname

7. Execute (as root) # sysctl security.jail.enforce_statfs=0

8. Execute (as root) [cmd=]/etc/rc.d/jail start jailname[/cmd]

9. Execute # jls to get new JID of jail

10. Execute (as root) # jexec JID tcsh to "log into" the jail

11. Execute # df (should show full unrestricted output from master-host)

NOTE: The above is also a clean way of restarting a jail.

NOTE: You can inspect whether the jail shutdown cleanly by listing all the processes associated with the jail, like-so:

# ps axo pid,jid,command | grep '\<JID\>' | grep -v '\<0\>'
# lists all processes associated with jail associated with JID (use `jls' to get the JID)

In addition, you can easily kill-off all jail processes associated with a given JID like-so:

# killall -9 -j JID

Though, that should be reserved as a "sledge-hammer" solution... it's much cleaner to call # rc.shutdown within the jail before finally using /etc/rc.d/jail to "deconfigure" the alias'd interfaces and clean-up the JID files.
 
Hi there, thanks for the info and the test to back it up. :) Since we're actually using our jails for chroot builds that, when kicked off, automatically mount devfs and such too so the solution was to also patch devfs to be jail friendly and live with statfs set to '0'. We saved the extra nfs mounts by nfs mounting once on the host and adding nullfs mounts in the jail fstab's. Same with amd mounts on the host and a nullfs mount for the jails. Have a good one!
 
devfs within jail

First, thanks DutchMaster for fixing my post for me =) laziness is no excuse for improper formatting ^_^

You shouldn't have to patch devfs to be `jail-friendly'. All that should be required to get devfs mounted within the jail is to add the following to /etc/rc.conf (on the master-host):

Code: 
jail_NAME_devfs_enable="YES"

Similarly, you can have /proc (procfs(5)) mounted automatically at jail startup too, despite the fact that procfs(5) is no longer required by standard utilities such as ps(1), top(1), and others (because procfs is considered a security risk these days). If you need /proc mounted within the jail (for example, if you're jailing an older FreeBSD distribution that still requires it), add the following to /etc/rc.conf:

Code: 
jail_NAME_procfs_enable="YES"

When either of these options are set in /etc/rc.conf, the jail startup script (/etc/rc.d/jail) will automatically mount the required filesystem within the jail. Specifically, when jail_NAME_devfs_enable is set, the devfs(5) filesystem will be mounted on $jail_NAME_rootdir/dev and similarly, if jail_NAME_procfs_enable is set, the procfs(5) filesystem will be mounted on $jail_NAME_rootdir/proc. Executing df(1) on the master-host will show something like the following (taken from an actual system):

Code: 
Filesystem              Size    Used   Avail Capacity  Mounted on
/dev/da0s1a             989M    305M    605M    34%    /
devfs                   1.0K    1.0K      0B   100%    /dev
/dev/da0s1e             989M     10K    910M     0%    /tmp
/dev/da0s1f              17G    2.6G     13G    16%    /usr
/dev/da0s1d             989M     23M    887M     3%    /var
devfs                   1.0K    1.0K      0B   100%    /usr/jail/www/dev

In the above example, /usr/jail/www is the root directory to our web-server jail, running FreeBSD-8.1 i386 within a FreeBSD-8.1 amd64 master-host.

Here's sample output to show the configuration in working order:

Code: 
dteske@oos0a ~$ file /sbin/init
/sbin/init: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 8.1, stripped
dteske@oos0a ~$ jls
   JID  IP Address      Hostname                      Path
     3  10.14.154.132   ipu0a.vbsd.vicor.com          /usr/vm/ipu0a.vbsd.vicor.com
dteske@oos0a ~$ sudo jexec 3 file /sbin/init
/sbin/init: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), statically linked, for FreeBSD 8.1, stripped
 
You must log in or register to reply here.
Back
Top