Solved Cannot delete symlink: rm: Operation not permitted

Anyone care to have a stab at this?

I have a soft symlink residing on a ZFS dataset which I can't delete. It's pointing to a directory on a different ZFS dataset which I deleted (the directory, not the dataset).

Here's some output from my attempts:

Code:
# ls -ald media/
ls: media/: No such file or directory

# ls -ald media
lrwxr-xr-x  1 root  incoming    21B 31 Mar  2017 media@ -> /z/media-05/incoming/

# ls -al /z/media-05/incoming/
ls: /z/media-05/incoming/: No such file or directory

# rm media
rm: media: Operation not permitted

# rm -f media
rm: media: Operation not permitted

# chflags -R noschg media
# rm -f media
rm: media: Operation not permitted

# stat media
2875054362 310538 lrwxr-xr-x 1 root incoming 4294967295 21 "Mar 31 22:12:41 2017" "Mar 31 22:12:41 2017" "Mar 31 22:16:01 2017" "Mar 31 22:12:41 2017" 4096 1 0x100800 media

Thanks,
Scott
 
Last edited by a moderator:
Make sure you're not accidentally cd'ed into that symlink directory before you removed it. It's possible you may have some other terminal screen open that's still in there.
 
Duffyx:

Code:
# unlink media
unlink: media: Operation not permitted

SirDice:

I had also tried a reboot, no dice:

Code:
# lsof | grep media
lsof: WARNING: compiled for FreeBSD release 11.0-RELEASE-p8; this is 11.1-RELEASE-p4.
[irrelevant output removed]
 
Last edited by a moderator:
Also just tried single user. Same error.

If I recreate the target, the symlink behaves normally. (I can cd into it, etc.) It's also the exact size of the target pathname. But I still can't delete it.

Thanks.
 
Perhaps it has an ACL (check with getfacl(1))? Also verify if it's perhaps NFS exported, see /etc/exports.
 
Thanks for your ideas. Still no dice.

Under single mode there would be no NFS, yes? The parent (/z/incoming) IS exported and so is the old symlink target. But again, under single mode I think that's a non-starter.

I can chmod and chown the file successfully (with -h).

As for getfacl:

Code:
# getfacl -h /z/incoming/media
# file: /z/incoming/media
# owner: root
# group: wheel
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:rwxp--a-R-c--s:-------:allow

I've never used ACLs - does the above read ok?

Scott
 
Last edited by a moderator:
Under single mode there would be no NFS, yes? The parent (/z/incoming) IS exported and so is the old symlink target. But again, under single mode I think that's a non-starter.
Yes, agreed. It was a bit of a long shot.
 
Should I post on freebsd-questions or freebsd-fs?

Any thoughts as to whether its ZFS? Is there a way to force ZFS to do a metadata check?

Scott
 
Look at the actual symlink with getfacl(1), not where the symlink was/is pointing to. Or is /z/incoming/media the symlink? It's a little hard to tell, that context is missing from the first post.

Is there a way to force ZFS to do a metadata check?
Yes, you can try scrubbing the pool. That should fix (or at least report) any filesystem errors that may be preventing the deletion.
 
Yep sorry, /z/incoming/media is the symlink, the target is /z/media-05/incoming/.

When I ran getfacl I used -h (see above).

Scrub. Duh. I'll fire off a scrub... :)

Thanks,
Scott
 
6 hours later... same result

Code:
# zpool status -v ZP-01
  pool: ZP-01
 state: ONLINE
  scan: scrub repaired 0 in 5h42m with 0 errors on Thu Jan 25 07:12:21 2018
config:

        NAME                                            STATE     READ WRITE CKSUM
        ZP-01                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/ebe89f2e-f421-11e7-85d8-3cd92b042b6f  ONLINE       0     0     0
            gptid/ed674e21-f421-11e7-85d8-3cd92b042b6f  ONLINE       0     0     0

errors: No known data errors
# rm media
rm: media: Operation not permitted
 
Last edited by a moderator:
Thanks. Bit of a long shot - my understanding is that a symlink is just a text file whose contents are the target and whose directory entry is marked with a symlink flag. Unlinking it should be independent of the contents. Here's the result:

Code:
# cd /incoming
# ls -ald media
lrwxrwxrwx  1 root  wheel    21B 31 Mar  2017 media@ -> /z/media-05/incoming/

# ls -ald media/
ls: media/: No such file or directory

# mkdir -p /z/media-05/incoming/
# touch /z/media-05/incoming/test

# ls -al media/
total 10
drwxr-xr-x  2 root   wheel     3B 25 Jan 12:39 ./
drwxr-xr-x  5 media  wheel     6B 25 Jan 12:39 ../
-rw-r--r--  1 root   wheel     0B 25 Jan 12:39 test

# rm media
rm: media: Operation not permitted

# unlink media
unlink: media: Operation not permitted

Scott
 
Last edited by a moderator:
What's this:

lsof: WARNING: compiled for FreeBSD release 11.0-RELEASE-p8; this is 11.1-RELEASE-p4.
[irrelevant output removed]
That doesn't look quite healthy to me. Note: the context you're giving is that this happened after a reboot. If so then I think you could have bigger issues.

What does: sysctl kern.securelevel tell you? (I know you tried in single user mode, I'd like to try and cover all bases, including possible oversights).

But speaking of single user (once again: trying to cover possible mishaps): did you account for the filesystem to be mounted readonly right after single user mode?

And what happens if you use a rescue CD?
 
Thanks ShelLuser,

Ignore the lsof warning. That’s expected as I upgraded to 11.1 recently and need to rebuild it.

I don’t know when the fault occurred. Not necessarily immediately after a reboot. I had been moving zfs datasets around between pools and systems before I noticed the error but it could have been there for months. I had a few snapshots from days before and when cloned they all behaved the same. Same error.

Code:
# sysctl kern.securelevel
kern.securelevel: -1

When I booted into single user the root mount was RO but I manually mounted the dataset. Which was RW.
 
What are the permissions of /incoming?

Can you touch and then rm a file in the /incoming directory? Same question for a new symbolic link in /incoming?

How about the zfs dataset properties for /incoming? It's not set readonly is it?
 
Thanks droeders

Code:
# ls -ald /z/incoming/
drwxrwxr-t  84 incoming  incoming    93B 25 Jan 01:49 /z/incoming//

# cd /z/incoming/
# touch testfile
# ls -al /z/incoming/testfile 
-rw-r--r--  1 root  incoming     0B 26 Jan 10:23 /z/incoming/testfile

# rm testfile 
testfile

# ls -al /z/incoming/testfile
ls: /z/incoming/testfule: No such file or directory

Code:
# zfs get all /z/incoming

NAME            PROPERTY              VALUE                  SOURCE
ZP-01/incoming  type                  filesystem             -
ZP-01/incoming  creation              Mon Jan 22 12:21 2018  -
ZP-01/incoming  used                  488G                   -
ZP-01/incoming  available             2.34T                  -
ZP-01/incoming  referenced            414G                   -
ZP-01/incoming  compressratio         1.12x                  -
ZP-01/incoming  mounted               yes                    -
ZP-01/incoming  quota                 none                   default
ZP-01/incoming  reservation           none                   default
ZP-01/incoming  recordsize            128K                   default
ZP-01/incoming  mountpoint            /z/incoming            local
ZP-01/incoming  sharenfs              off                    default
ZP-01/incoming  checksum              on                     default
ZP-01/incoming  compression           on                     inherited from ZP-01
ZP-01/incoming  atime                 off                    inherited from ZP-01
ZP-01/incoming  devices               on                     default
ZP-01/incoming  exec                  on                     default
ZP-01/incoming  setuid                on                     default
ZP-01/incoming  readonly              off                    default
ZP-01/incoming  jailed                off                    default
ZP-01/incoming  snapdir               hidden                 default
ZP-01/incoming  aclmode               discard                default
ZP-01/incoming  aclinherit            restricted             default
ZP-01/incoming  canmount              on                     default
ZP-01/incoming  xattr                 off                    temporary
ZP-01/incoming  copies                1                      default
ZP-01/incoming  version               5                      -
ZP-01/incoming  utf8only              off                    -
ZP-01/incoming  normalization         none                   -
ZP-01/incoming  casesensitivity       sensitive              -
ZP-01/incoming  vscan                 off                    default
ZP-01/incoming  nbmand                off                    default
ZP-01/incoming  sharesmb              off                    default
ZP-01/incoming  refquota              none                   default
ZP-01/incoming  refreservation        none                   default
ZP-01/incoming  primarycache          all                    default
ZP-01/incoming  secondarycache        all                    default
ZP-01/incoming  usedbysnapshots       74.0G                  -
ZP-01/incoming  usedbydataset         414G                   -
ZP-01/incoming  usedbychildren        0                      -
ZP-01/incoming  usedbyrefreservation  0                      -
ZP-01/incoming  logbias               latency                default
ZP-01/incoming  dedup                 off                    default
ZP-01/incoming  mlslabel                                     -
ZP-01/incoming  sync                  standard               default
ZP-01/incoming  refcompressratio      1.11x                  -
ZP-01/incoming  written               39.5G                 -
ZP-01/incoming  logicalused           546G                   -
ZP-01/incoming  logicalreferenced     461G                   -
ZP-01/incoming  volmode               default                default
ZP-01/incoming  filesystem_limit      none                   default
ZP-01/incoming  snapshot_limit        none                   default
ZP-01/incoming  filesystem_count      none                   default
ZP-01/incoming  snapshot_count        none                   default
ZP-01/incoming  redundant_metadata    all
 
I was wanting you to check the permissions (and touch/symlink, get zfs properties) on the directory where the media symlink resides - presumably /incoming from your previous posts.

What's the full path to the media symlink that you can't remove?
 
That destination has been deleted. In the posts above you’ll see I recreated it for testing then deleted it again. So the system knows it’s a link - that bit works.

I haven’t read the source code yet for rm (I’m camping for a couple of days) but I’d be surprised if it even checked the contents of a symlink when unlinking.

The symlink’s full path /z/incoming/media. The contents of the symlink file (the destination) is /z/media-05/incoming. See above for tests showing me recreating the destination.
 
I'm confused because of these conflicting posts, where one seems to use /incoming and the other is using /z/incoming:

Code:
# cd /incoming
# ls -ald media
lrwxrwxrwx  1 root  wheel    21B 31 Mar  2017 media@ -> /z/media-05/incoming/

Code:
# ls -ald /z/incoming/
drwxrwxr-t  84 incoming  incoming    93B 25 Jan 01:49 /z/incoming//

# cd /z/incoming/
# touch testfile
# ls -al /z/incoming/testfile
-rw-r--r--  1 root  incoming     0B 26 Jan 10:23 /z/incoming/testfile

# rm testfile
testfile

# ls -al /z/incoming/testfile
ls: /z/incoming/testfule: No such file or directory

I see the sticky bit set on the /z/incoming directory, but that shouldn't prevent deletions by root. Your shell prompt seems to indicate so, but you're running these commands as root, correct?

Can you create and remove a new symlink from the same directory that media is located in?
 
Sorry. Confusing but not conflicting./incoming is a symlink to /z/incoming. All my recent posts are from the latter to avoid confusion.

Here’s your suggestions:

Code:
# cd /z/incoming/

# ln -s /z/media-05/incoming/ media-test

# ls -al media-test/
total 10
drwxr-xr-x  2 root   wheel     3B 25 Jan 12:39 ./
drwxr-xr-x  5 media  wheel     6B 25 Jan 12:39 ../
-rw-r--r--  1 root   wheel     0B 25 Jan 12:39 test

# rm media-test
media-test

# chmod uog-t .
# ls -ald .
drwxrwxr-x  84 incoming  incoming    93B 26 Jan 14:26 ./

# rm media
rm: media: Operation not permitted

# chmod uog+t .
# ls -ald .
drwxrwxr-t  84 incoming  incoming    93B 26 Jan 14:26 ./

# rm media
rm: media: Operation not permitted
 
Code:
# cd /z/incoming/
# ls -lo media
lrwxrwxrwx  1 root  wheel  sunlnk,uarch   21B 31 Mar  2017 media@ -> /z/media-05/incoming/

# ls -lod ./
drwxrwxr-t  84 incoming  incoming  uarch   93B 26 Jan 14:26 ./

I put -d on the last one to reduce the output.
 
Back
Top