ZFS NFS: read-only does not allow reading

[PMc]

Lets assume we have accidentially deleted a file. Luckily, it is zfs and there is a snapshot, so one might just copy the file back from .zfs/snapshot.
But this doesn't work on NFS:

pmc@disp:2516:1~/Media$ cp .zfs/snapshot/zar.20531/Thiere.txt .
cp: .zfs/snapshot/zar.20531/Thiere.txt: Read-only file system

 

[ralphbsz]

Which of the two file systems (.zfs/... and .) is NFS? Which one is read only? How are they mounted? I don't have enough information to help understand or debug this.

 

[bakul]

I don't see this problem (local 15.0-RELEASE-p4, remote (nfs host): 15.0-STABLE). See if "date > ./foo" works.

 

[PMc]

Which of the two file systems (.zfs/... and .) is NFS? Which one is read only? How are they mounted? I don't have enough information to help understand or debug this.

It is only one filesystem, the toplevel of a ZFS fileset, wich is nfs exported, and then mounted from remote.
A zfs fileset has on the toplevel an invisible .zfs directory, which contains the snapshots, and therefore is readonly.

Apparently nfs gets confused because the same filesystem/mountpoint is in part writeable and in part readonly.
Most likely this is another effect of the "copy-file-range" weirdness, because a traditional copy would treat source and dest independently from each other, but copy-file-range on the same filesystem tries to optimize things by not doing a copy at all when a range-hardlink is possible.
Proof: dd (which doesn't use copy-file-range) works normally as expected.

 

[PMc]

I don't see this problem (local 15.0-RELEASE-p4, remote (nfs host): 15.0-STABLE). See if "date > ./foo" works.

That has nothing to do with it. One can read the source, one can write the destination, only one cannot copy from source to destination with cp.

 

[PMc]

Slowly but surely that copy-file-range gimmick begins to annoy me.

There was this one: https://forums.freebsd.org/threads/why-is-stdout-locked-against-read.99421/post-722715
Rick Macklem wrote a workaround, I am not sure if that was incorporated into HEAD.

Then there is this one, which apparently was fixed: https://forums.freebsd.org/threads/cp-interrupted-system-call.101308/post-738926

And now there is this one here...

 

[AlfredoLlaquet]

Try this:

dd if=.zfs/snapshot/zar.20531/Thiere.txt of=./Thiere.txt bs=128k

 

[cracauer@]

"cp .zfs/snapshot/zar.20531/Thiere.txt ."

Obviously the target is the ZFS filesystem, otherwise it would not have .zfs in it. You say the ZFS filesystem is mounted readonly. So why do you expect this would work.

mv(1) has confusing error messages. The filename it reports in an error message is always the source, regardless of whether the error occured on the source or target side.

 

[Alain De Vos]

zfs snapshots are read-only , so copying will always fail...
Note, your cp command is incomplete ,what is source and destination, and how are they mounted.

 

[bakul]

That has nothing to do with it. One can read the source, one can write the destination, only one cannot copy from source to destination with cp.

I actually tried something very similar:
$ df .
Filesystem 1K-blocks Used Avail Capacity Mounted on
fileserver:/usr/remotes 4747615348 467707228 4279908120 10% /usr/remotes
$ mount |grep /usr/remotes
fileserver:/usr/remotes on /usr/remotes (nfs)
$ cp .zfs/snapshot/20250321/github/NetBSD/src/README.md .
$ ls -l README.md
-rw-r--r-- 1 bakul wheel 1748 Mar 21 13:36 README.md


You might want to post on the fs mailing list or create a PR.

 

[cracauer@]

You folks need to read the OP carefully.

 

[mer]

Lets assume we have accidentially deleted a file. Luckily, it is zfs and there is a snapshot, so one might just copy the file back from .zfs/snapshot.
But this doesn't work on NFS:

pmc@disp:2516:1~/Media$ cp .zfs/snapshot/zar.20531/Thiere.txt .
cp: .zfs/snapshot/zar.20531/Thiere.txt: Read-only file system

So source of the cp is a zfs snapshot, destination is NFS?
I would ask "what is under the NFS file system?"
To me, it's obvious the zfs snapshot is readonly (because that's part of the definition of a snapshot).
I'm not sure what that error message is actually referring to. Is it the zfs snapshot source or the NFS destination?
Does copying to NFS try to modify the source like with atime or something?

 

[AlfredoLlaquet]

You folks need to read the OP carefully.

Yes, sir! Right away, sir!

 

[PMc]

Try this:

dd if=.zfs/snapshot/zar.20531/Thiere.txt of=./Thiere.txt bs=128k

*Laugh*
Sure, that works.

"cp .zfs/snapshot/zar.20531/Thiere.txt ."

Obviously the target is the ZFS filesystem, otherwise it would not have .zfs in it. You say the ZFS filesystem is mounted readonly. So why do you expect this would work.

I didn't, and it isn't.
Only the .zfs subdirectory is readonly (always, maintained by zfs itself)

mv(1) has confusing error messages. The filename it reports in an error message is always the source, regardless of whether the error occured on the source or target side.

It woudn't happen on the target, because the target is writeable.
Using dd (like Alfredo showed) does work. Copying in two steps (from the .zfs subdirectory to somewhere else, then from there back into the zfs filesystem) does also work.

zfs snapshots are read-only , so copying will always fail...
Note, your cp command is incomplete ,what is source and destination, and how are they mounted.

The command is not incomlete. The destination is the single dot at the end of the line, i.e. the current location.

Which of the two file systems (.zfs/... and .) is NFS? Which one is read only? How are they mounted? I don't have enough information to help understand or debug this.

It is the same filesystem. There is only one mount:

pmc@disp:2505:1~/Media$ df . .zfs
Filesystem        1K-blocks      Used     Avail Capacity  Mounted on
edge-e:/media/pmc 390882496 244693252 146189244    63%    /home/pmc/Media
edge-e:/media/pmc 390882496 244693252 146189244    63%    /home/pmc/Media
pmc@disp:2510:1~/Media$ /sbin/mount | grep Media
edge-e:/media/pmc on /home/pmc/Media (nfs, nfsv4acls)
pmc@disp:2511:1~/Media$ grep Media /etc/fstab
edge-e:/media/pmc       /home/pmc/Media nfs     nfsv4,rw        0       0

 

[bakul]

I blame nfsv4

 

[PMc]

So source of the cp is a zfs snapshot, destination is NFS?

Both source and destination is inside the same NFS mount of the same ZFS fileset.

I would ask "what is under the NFS file system?"
To me, it's obvious the zfs snapshot is readonly (because that's part of the definition of a snapshot).

Exactly. So one and the same NFS mount (of the same ZFS fileset) is partially readonly (the part with the snapshots) and partially writeable (the regular fileset part).

I'm not sure what that error message is actually referring to. Is it the zfs snapshot source or the NFS destination?
Does copying to NFS try to modify the source like with atime or something?

Interesting. That might be possible. But then still on a r/o device such would just be silently ignored.

But, I can tell you at least what it tries to do: it detects that source and destination are on the same device, and that the unterlying filesystem is ZFS (which supports deduplication).
Consequentially, it tries to not do a copy at all, but instead just record the newly to be created destination file blocks as duplicates of the source file blocks (and that way spare the entire i/o traffic and be tremendously faster).
Obviousely, such a scheme would need some special handling when the actual copy happens from a snapshot to the main filesystem (if it works at all in that way).
But then, handling that whole matter through NFS makes things even more complicated,
and then apparently at some point things fail.

 

[Alain De Vos]

Migrate from NFS to ISCSI
Seriously , i find understand NFS mount does not mount the zfs snapshot correctly.

 

[AlfredoLlaquet]

*Laugh*

Oh, well, I was just trying to help. I guess the boss was right and my little brain couldn't comprehend the full complexity of the matter. Next time I won't even try because it's unpleasant to be laughed at.

 

[cracauer@]

I didn't, and it isn't.

You didn't make clear that the actual NFS mount is read-write. If you re-read your header and first post you crealy waffle on about a readonly NFS volume.

In any case, it is an unusual situation to have one mount that is partially a readonly filesystem and partially read-write. I don't think the kernel has a concept of that. You are probably the first one to experience this.

 

[PMc]

Oh, well, I was just trying to help. I guess the boss was right and my little brain couldn't comprehend the full complexity of the matter. Next time I won't even try because it's unpleasant to be laughed at.

Don't worry. I didn't laugh at you. I was enjoying that You found the same workaround I figured for these issues.

 

[PMc]

You didn't make clear that the actual NFS mount is read-write. If you re-read your header and first post you crealy waffle on about a readonly NFS volume.

In any case, it is an unusual situation to have one mount that is partially a readonly filesystem and partially read-write. I don't think the kernel has a concept of that. You are probably the first one to experience this.

Oh, You're right. The header is indeed misleading when read from someone elses perspective.

 

[mer]

Maybe a quick test would be create a clone off that snapshot (because clones are writeable) and then try the copy.

 

[Alain De Vos]

or cp -axfvR / clone /rsync of the snapshot. Have the impression problem is at the source not destination.

 

[AlfredoLlaquet]

Don't worry. I didn't laugh at you. I was enjoying that You found the same workaround I figured for these issues.

That's a very kind answer, thank you. You are a wholesome person.

Here's an excerpt of what ChatGPT 5.4 Plus Extended Thinking generated about this issue in case it casts some light. If it says anything outrageous, please shout at it (I'm speaking to everybody, not to the OP), not at me. I'm only trying to help. Also, be sure it's really outrageous before shouting, because it may not be once considered carefully.

---

This looks much more like a cp(1)() fast-path problem than a basic NFS permissions problem. ZFS snapshots are exposed under .zfs/snapshot, and those snapshots are read-only. FreeBSD’s copy_file_range(2)() is an in-kernel copy primitive that may use a file-system-specific technique when source and destination appear to be on the same file system, and recent FreeBSD discussion still refers to cp(1)() trying to use copy_file_range(2)(). That combination fits your symptom very well: reading the snapshot works, writing the live file works, but cp(1)() fails with Read-only file system when it tries the optimized copy path across the snapshot/live boundary.

So the likely failure is not “NFS cannot handle a partly writable tree” in the simple sense. It is more specific: the exported namespace contains both the live dataset and the snapshot view, and cp(1)() is trying an in-kernel copy operation instead of a plain read/write loop. Also, the closer FreeBSD term here is block cloning or an in-kernel copy optimization, not a “range-hardlink.” copy_file_range(2)() explicitly allows file-system-specific techniques, and VOP_COPY_FILE_RANGE(9)() is the vnode-level hook behind that mechanism.