NFS not playing nice after I create new ZFS fs

[rowan194]

I've had an NFSv4 setup working for several years on my NAS. There's a 4 drive RAIDZ1 mounted on /nas01.

The path /nas01/backups is used by several servers.

Recently, I upgraded from 12.x-R to 14.1-R, and set up a new ZFS file system with extreme compression (zstd-19) on /nas01/backups/livefs

There's two issues here.

1. After the upgrade, FreeBSD has started reporting Warning: exporting /nas01/backups exports entire /nas01 file system - presumably a security issue if untrusted clients can access the next level up, but things seem to work otherwise

2. I cannot find a valid line that will export the new file system /nas01/backups/livefs. The usual error is bad exports list line '/nas01/backups /nas01/backups/livefs': fsid mismatch. There's very little information about "fsid mismatch" that I can find, and most just seems to be source code, so I am having difficulty understanding this error.

I think what's happening is that mountd doesn't like me trying to export a mount point whose path is within another mount point?

Just for curiosity, I tried explicitly listing the base mount point /nas01 (and only that) as a valid export, but that didn't work.

Is there some way to tell NFS to export the damn thing? Is it something to do with exporting both /nas01/backups (which is not a mount point) and /nas01/backups/livefs (which is a mount point)?

Here's my /etc/exports (slightly censored)

/nas01/backups /nas01/backups/livefs -mapall=nas:nas 192.168.xx.xx 192.168.xx.xx ......
/nas01/backups -mapall=nas:nas 192.168.xx.xx

Here's the ZFS filesystems mounted that reference /nas01:

zfs18                           /nas01
zfs18/livefs                    /nas01/backups/livefs

Thanks in advance.

 

[cy@]

I've had an NFSv4 setup working for several years on my NAS. There's a 4 drive RAIDZ1 mounted on /nas01.

The path /nas01/backups is used by several servers.

Recently, I upgraded from 12.x-R to 14.1-R, and set up a new ZFS file system with extreme compression (zstd-19) on /nas01/backups/livefs

There's two issues here.

1. After the upgrade, FreeBSD has started reporting "Warning: exporting /nas01/backups exports entire /nas01 file system" - presumably a security issue if untrusted clients can access the next level up, but things seem to work otherwise

It's always been that way, ever since I started using FreeBSD 2.0.5. And the man page discusses this. The O'Reilly book Managing NFS and NIS also talks about this.

You can circumvent this by exporting a nullfs for /nas01/backups. NFS exports the root inode filehandle.

I vaguely recall my Solaris days (20 years ago) it behaved differently.

2. I cannot find a valid line that will export the new file system /nas01/backups/livefs. The usual error is "bad exports list line '/nas01/backups /nas01/backups/livefs': fsid mismatch" . There's very little information about "fsid mismatch" that I can find, and most just seems to be source code, so I am having difficulty understanding this error.

You should export that using a nullfs too. Or, export the entire /nas01/backups with -alldirs.

I think what's happening is that mountd doesn't like me trying to export a mount point whose path is within another mount point?

Just for curiosity, I tried explicitly listing the base mount point /nas01 (and only that) as a valid export, but that didn't work.

Is /nas01 a zpool?

ZFS datasets would also mitigate this.

Is there some way to tell NFS to export the damn thing? Is it something to do with exporting both /nas01/backups (which is not a mount point) and /nas01/backups/livefs (which is a mount point)?

I'm guessing these are separate ZFS datasets, the answer is no. If UFS, the answer is yes.

Here's my /etc/exports (slightly censored)

/nas01/backups /nas01/backups/livefs -mapall=nas:nas 192.168.xx.xx 192.168.xx.xx ......
/nas01/backups -mapall=nas:nas 192.168.xx.xx

Here's the ZFS filesystems mounted that reference /nas01:

zfs18                           /nas01
zfs18/livefs                    /nas01/backups/livefs

This is why. /nas01 and /nas01/backup/lifefs are different datasets. ZFS datasets are different filesystems just like a UFS filesystem mounted (grafted) to another. It's the UNIX way.

Thanks in advance.

 

[usdmatt]

The exports documentation shows the following -

In a mount entry, the first field(s) specify the directory path(s) within a server file system that can be mounted on by the corresponding client(s).

So for a start the first line is trying to list 2 directories to export that are actually in different filesystems. This is I assume why you get the "fsid mismatch" error. It's simply telling you that the multiple entries you are trying to export are on different file systems.

It's not obvious to me with the information given why just exporting /nas01 would not work, although it may have an issue with the next line in the exports file that is effectively exporting the same filesystem.

Personally I would try to follow the advice further down in the documentation that suggests all exports should be actual mount points of file systems on the server and not just directories.

Note well that exporting a directory on the server does not guarantee that only files below the exported directory will be accessi-
ble. This is true even in the absence of the -alldirs flag. To provide this guarantee, the exported directories must be local file system
mount points on the server. For example, if one exports /home, and /home is not a file system mount point, then clients will be able to access arbitrary files on the root file system. As such, to avoid confusion with respect to what is exported, it may be prudent to limit exported directories to server local file system mount points.

 

[rowan194]

Sorry, I'm still confused.

I have simplified exports to only a single line, and added -alldirs.

/nas01 -mapall=nas:nas -alldirs 192.168.xx.xx .....

With this config:

- clients can mount and access /nas01/backups without any change in command (even though the normal export config is /nas01/backups, and this config is /nas01 only)
- /nas01/backups/livefs can be navigated to on clients, but it's empty
- attempting to mount /nas01/backups/livefs on a client fails - mount request denied

If I change exports from /nas01 to /nas01/backups/livefs, I can mount /nas01/backups/livefs on clients, and the data shows...but exports won't let me have both listed. I still don't understand why I cannot cleanly export two separate mount points.

EDIT - POSSIBLE WORKAROUND

Splitting the two mounts into separate lines seems to have worked:

/nas01/backups/livefs -mapall=nas:nas -alldirs 192.168.xx.xx
/nas01/backups -mapall=nas:nas -alldirs 192.168.xx.xx

There is still the warning about exporting all of /nas01 - remember that /nas01/backups is just a normal dir off the /nas01 mount - but I can now mount both /nas01/backups and /nas01/backups/livefs via NFS.

I'm still interested in a solution that doesn't require me to mount the two separate paths on each client, rather, just have a single seamless tree off /nas01/backups. Can this be done?

 

[usdmatt]

No, NFS works at the filesystem level. You cannot cross mountpoints with a single NFS mount. If you export /nas01, the /nas01/backups/live dataset is not available on the client, as this is a completely different filesystem.

If you want it to be a separate dataset, it will have to be mounted separately on the client.