At a loss...which network sharing protocol can I use?

J

joeyea323

I store all of my FreeBSD jails on a single fileserver at this time and have been trying for a while now to get an active/active clustered filesystem setup. I've strictly been using GlusterFS on Ubuntu for the clustered filesystem part, but am having nothing but problems getting those files successfully shared to the FreeBSD servers. My one gotcha is that I need locking to be clusterable for me to be able to actually create clusters of web/email/etc. jails. So far I've had no problems sharing those files via NFS, SMB, and SSHFS, however each has presented its own problems which, so far, have prevented me from finally picking one to use:
  • NFSv4 exported via Ubuntu NFS Server: I can mount the share and access files, however locking does not work despite nlockmgr showing as running on both sides via rpcinfo -p.
  • NFSv3 exported via Gluster NFS Server: As of Gluster v3.3 locking is supported over NFS. lockd is running client side and nlockmgr is verified running on the server side, however locking a file will indefinitely hang the FreeBSD server. Turning on debugging for statd and lockd show nothing in debug.log beyond it indicating it found a new host (the Ubuntu server).
  • SMB with CTDB to make locking clusterable: Can mount the share and browse, however the creation of symbolic links errors as "Operation not supported" and chown/chgrp/chmod don't error but don't work. The big gotcha there is the symbolic links as the creation of the jail's base filesystem depends on being able to create symbolic links. Seems to be a client side issue as the server side samba logs indicate nothing regarding the attempt to create a symbolic link.
  • SSHFS: I'm first mounting the Gluster volume to a directory on the Gluster server via the Gluster client. Then I mount that folder over sshfs to the FreeBSD server. I'm assuming that lock requests will be sent to the remote server which allows clustered locking to work as that will be handled via the Gluster client. All operations work, however untaring my jail base filesystem will kernel panic the FreeBSD server everytime so this is very unstable.
I'm really at a loss here as I thought at least one of these options would work. At this point I'm contemplating doing a hybrid setup where I use NFS with no locking to create the jails and SMB to actually run the jails. This has its own issues as SMB seems to also not support unix sockets (which affects MySQL, Postfix, and Dovecot to name a few) and this becomes unusable the first time a service running in the jail wants to create a symbolic link. I also wondered if there was a way to put FreeBSD's NFS locking mechanism on clustered storage in much the same way that CTDB recommends setting things up to cluster NFS.

FYI - FreeBSD servers are 8.3-RELEASE and 9.0-RELEASE at latest patch level. Ubuntu server is 12.04 running both Gluster 3.2.5 and 3.3.0 during this testing.
 
Hi,

Did you get anywhere at the end ?
I'm looking at creating web cluster servers hosting WordPress..so need replicate fullness when plugging are updating
Thank you
 
You must log in or register to reply here.
Back
Top