• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Solved Cannot delete symlink: rm: Operation not permitted

scotia

Member

Thanks: 2
Messages: 39

#1
Anyone care to have a stab at this?

I have a soft symlink residing on a ZFS dataset which I can't delete. It's pointing to a directory on a different ZFS dataset which I deleted (the directory, not the dataset).

Here's some output from my attempts:

Code:
# ls -ald media/
ls: media/: No such file or directory

# ls -ald media
lrwxr-xr-x  1 root  incoming    21B 31 Mar  2017 media@ -> /z/media-05/incoming/

# ls -al /z/media-05/incoming/
ls: /z/media-05/incoming/: No such file or directory

# rm media
rm: media: Operation not permitted

# rm -f media
rm: media: Operation not permitted

# chflags -R noschg media
# rm -f media
rm: media: Operation not permitted

# stat media
2875054362 310538 lrwxr-xr-x 1 root incoming 4294967295 21 "Mar 31 22:12:41 2017" "Mar 31 22:12:41 2017" "Mar 31 22:16:01 2017" "Mar 31 22:12:41 2017" 4096 1 0x100800 media
Thanks,
Scott
 
Last edited by a moderator:

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,089
Messages: 26,977

#3
Make sure you're not accidentally cd'ed into that symlink directory before you removed it. It's possible you may have some other terminal screen open that's still in there.
 

scotia

Member

Thanks: 2
Messages: 39

#4
Duffyx:

Code:
# unlink media
unlink: media: Operation not permitted
SirDice:

I had also tried a reboot, no dice:

Code:
# lsof | grep media
lsof: WARNING: compiled for FreeBSD release 11.0-RELEASE-p8; this is 11.1-RELEASE-p4.
[irrelevant output removed]
 
Last edited by a moderator:

scotia

Member

Thanks: 2
Messages: 39

#5
Also just tried single user. Same error.

If I recreate the target, the symlink behaves normally. (I can cd into it, etc.) It's also the exact size of the target pathname. But I still can't delete it.

Thanks.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,089
Messages: 26,977

#6
Perhaps it has an ACL (check with getfacl(1))? Also verify if it's perhaps NFS exported, see /etc/exports.
 

scotia

Member

Thanks: 2
Messages: 39

#7
Thanks for your ideas. Still no dice.

Under single mode there would be no NFS, yes? The parent (/z/incoming) IS exported and so is the old symlink target. But again, under single mode I think that's a non-starter.

I can chmod and chown the file successfully (with -h).

As for getfacl:

Code:
# getfacl -h /z/incoming/media
# file: /z/incoming/media
# owner: root
# group: wheel
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:rwxp--a-R-c--s:-------:allow
I've never used ACLs - does the above read ok?

Scott
 
Last edited by a moderator:

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,089
Messages: 26,977

#8
Under single mode there would be no NFS, yes? The parent (/z/incoming) IS exported and so is the old symlink target. But again, under single mode I think that's a non-starter.
Yes, agreed. It was a bit of a long shot.
 

scotia

Member

Thanks: 2
Messages: 39

#9
Should I post on freebsd-questions or freebsd-fs?

Any thoughts as to whether its ZFS? Is there a way to force ZFS to do a metadata check?

Scott
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,089
Messages: 26,977

#10
Look at the actual symlink with getfacl(1), not where the symlink was/is pointing to. Or is /z/incoming/media the symlink? It's a little hard to tell, that context is missing from the first post.

Is there a way to force ZFS to do a metadata check?
Yes, you can try scrubbing the pool. That should fix (or at least report) any filesystem errors that may be preventing the deletion.
 

scotia

Member

Thanks: 2
Messages: 39

#11
Yep sorry, /z/incoming/media is the symlink, the target is /z/media-05/incoming/.

When I ran getfacl I used -h (see above).

Scrub. Duh. I'll fire off a scrub... :)

Thanks,
Scott
 

scotia

Member

Thanks: 2
Messages: 39

#12
6 hours later... same result

Code:
# zpool status -v ZP-01
  pool: ZP-01
 state: ONLINE
  scan: scrub repaired 0 in 5h42m with 0 errors on Thu Jan 25 07:12:21 2018
config:

        NAME                                            STATE     READ WRITE CKSUM
        ZP-01                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/ebe89f2e-f421-11e7-85d8-3cd92b042b6f  ONLINE       0     0     0
            gptid/ed674e21-f421-11e7-85d8-3cd92b042b6f  ONLINE       0     0     0

errors: No known data errors
# rm media
rm: media: Operation not permitted
 
Last edited by a moderator:

scotia

Member

Thanks: 2
Messages: 39

#13
Thanks. Bit of a long shot - my understanding is that a symlink is just a text file whose contents are the target and whose directory entry is marked with a symlink flag. Unlinking it should be independent of the contents. Here's the result:

Code:
# cd /incoming
# ls -ald media
lrwxrwxrwx  1 root  wheel    21B 31 Mar  2017 media@ -> /z/media-05/incoming/

# ls -ald media/
ls: media/: No such file or directory

# mkdir -p /z/media-05/incoming/
# touch /z/media-05/incoming/test

# ls -al media/
total 10
drwxr-xr-x  2 root   wheel     3B 25 Jan 12:39 ./
drwxr-xr-x  5 media  wheel     6B 25 Jan 12:39 ../
-rw-r--r--  1 root   wheel     0B 25 Jan 12:39 test

# rm media
rm: media: Operation not permitted

# unlink media
unlink: media: Operation not permitted
Scott
 
Last edited by a moderator:

ShelLuser

Son of Beastie

Thanks: 1,262
Messages: 2,683

#14
What's this:

lsof: WARNING: compiled for FreeBSD release 11.0-RELEASE-p8; this is 11.1-RELEASE-p4.
[irrelevant output removed]
That doesn't look quite healthy to me. Note: the context you're giving is that this happened after a reboot. If so then I think you could have bigger issues.

What does: sysctl kern.securelevel tell you? (I know you tried in single user mode, I'd like to try and cover all bases, including possible oversights).

But speaking of single user (once again: trying to cover possible mishaps): did you account for the filesystem to be mounted readonly right after single user mode?

And what happens if you use a rescue CD?
 

scotia

Member

Thanks: 2
Messages: 39

#15
Thanks ShelLuser,

Ignore the lsof warning. That’s expected as I upgraded to 11.1 recently and need to rebuild it.

I don’t know when the fault occurred. Not necessarily immediately after a reboot. I had been moving zfs datasets around between pools and systems before I noticed the error but it could have been there for months. I had a few snapshots from days before and when cloned they all behaved the same. Same error.

Code:
# sysctl kern.securelevel
kern.securelevel: -1
When I booted into single user the root mount was RO but I manually mounted the dataset. Which was RW.
 

droeders

New Member

Thanks: 3
Messages: 6

#16
What are the permissions of /incoming?

Can you touch and then rm a file in the /incoming directory? Same question for a new symbolic link in /incoming?

How about the zfs dataset properties for /incoming? It's not set readonly is it?
 

scotia

Member

Thanks: 2
Messages: 39

#17
Thanks droeders

Code:
# ls -ald /z/incoming/
drwxrwxr-t  84 incoming  incoming    93B 25 Jan 01:49 /z/incoming//

# cd /z/incoming/
# touch testfile
# ls -al /z/incoming/testfile 
-rw-r--r--  1 root  incoming     0B 26 Jan 10:23 /z/incoming/testfile

# rm testfile 
testfile

# ls -al /z/incoming/testfile
ls: /z/incoming/testfule: No such file or directory
Code:
# zfs get all /z/incoming

NAME            PROPERTY              VALUE                  SOURCE
ZP-01/incoming  type                  filesystem             -
ZP-01/incoming  creation              Mon Jan 22 12:21 2018  -
ZP-01/incoming  used                  488G                   -
ZP-01/incoming  available             2.34T                  -
ZP-01/incoming  referenced            414G                   -
ZP-01/incoming  compressratio         1.12x                  -
ZP-01/incoming  mounted               yes                    -
ZP-01/incoming  quota                 none                   default
ZP-01/incoming  reservation           none                   default
ZP-01/incoming  recordsize            128K                   default
ZP-01/incoming  mountpoint            /z/incoming            local
ZP-01/incoming  sharenfs              off                    default
ZP-01/incoming  checksum              on                     default
ZP-01/incoming  compression           on                     inherited from ZP-01
ZP-01/incoming  atime                 off                    inherited from ZP-01
ZP-01/incoming  devices               on                     default
ZP-01/incoming  exec                  on                     default
ZP-01/incoming  setuid                on                     default
ZP-01/incoming  readonly              off                    default
ZP-01/incoming  jailed                off                    default
ZP-01/incoming  snapdir               hidden                 default
ZP-01/incoming  aclmode               discard                default
ZP-01/incoming  aclinherit            restricted             default
ZP-01/incoming  canmount              on                     default
ZP-01/incoming  xattr                 off                    temporary
ZP-01/incoming  copies                1                      default
ZP-01/incoming  version               5                      -
ZP-01/incoming  utf8only              off                    -
ZP-01/incoming  normalization         none                   -
ZP-01/incoming  casesensitivity       sensitive              -
ZP-01/incoming  vscan                 off                    default
ZP-01/incoming  nbmand                off                    default
ZP-01/incoming  sharesmb              off                    default
ZP-01/incoming  refquota              none                   default
ZP-01/incoming  refreservation        none                   default
ZP-01/incoming  primarycache          all                    default
ZP-01/incoming  secondarycache        all                    default
ZP-01/incoming  usedbysnapshots       74.0G                  -
ZP-01/incoming  usedbydataset         414G                   -
ZP-01/incoming  usedbychildren        0                      -
ZP-01/incoming  usedbyrefreservation  0                      -
ZP-01/incoming  logbias               latency                default
ZP-01/incoming  dedup                 off                    default
ZP-01/incoming  mlslabel                                     -
ZP-01/incoming  sync                  standard               default
ZP-01/incoming  refcompressratio      1.11x                  -
ZP-01/incoming  written               39.5G                 -
ZP-01/incoming  logicalused           546G                   -
ZP-01/incoming  logicalreferenced     461G                   -
ZP-01/incoming  volmode               default                default
ZP-01/incoming  filesystem_limit      none                   default
ZP-01/incoming  snapshot_limit        none                   default
ZP-01/incoming  filesystem_count      none                   default
ZP-01/incoming  snapshot_count        none                   default
ZP-01/incoming  redundant_metadata    all
 

droeders

New Member

Thanks: 3
Messages: 6

#18
I was wanting you to check the permissions (and touch/symlink, get zfs properties) on the directory where the media symlink resides - presumably /incoming from your previous posts.

What's the full path to the media symlink that you can't remove?
 

scotia

Member

Thanks: 2
Messages: 39

#19
That destination has been deleted. In the posts above you’ll see I recreated it for testing then deleted it again. So the system knows it’s a link - that bit works.

I haven’t read the source code yet for rm (I’m camping for a couple of days) but I’d be surprised if it even checked the contents of a symlink when unlinking.

The symlink’s full path /z/incoming/media. The contents of the symlink file (the destination) is /z/media-05/incoming. See above for tests showing me recreating the destination.
 

droeders

New Member

Thanks: 3
Messages: 6

#20
I'm confused because of these conflicting posts, where one seems to use /incoming and the other is using /z/incoming:

Code:
# cd /incoming
# ls -ald media
lrwxrwxrwx  1 root  wheel    21B 31 Mar  2017 media@ -> /z/media-05/incoming/
Code:
# ls -ald /z/incoming/
drwxrwxr-t  84 incoming  incoming    93B 25 Jan 01:49 /z/incoming//

# cd /z/incoming/
# touch testfile
# ls -al /z/incoming/testfile
-rw-r--r--  1 root  incoming     0B 26 Jan 10:23 /z/incoming/testfile

# rm testfile
testfile

# ls -al /z/incoming/testfile
ls: /z/incoming/testfule: No such file or directory
I see the sticky bit set on the /z/incoming directory, but that shouldn't prevent deletions by root. Your shell prompt seems to indicate so, but you're running these commands as root, correct?

Can you create and remove a new symlink from the same directory that media is located in?
 

scotia

Member

Thanks: 2
Messages: 39

#21
Sorry. Confusing but not conflicting./incoming is a symlink to /z/incoming. All my recent posts are from the latter to avoid confusion.

Here’s your suggestions:

Code:
# cd /z/incoming/

# ln -s /z/media-05/incoming/ media-test

# ls -al media-test/
total 10
drwxr-xr-x  2 root   wheel     3B 25 Jan 12:39 ./
drwxr-xr-x  5 media  wheel     6B 25 Jan 12:39 ../
-rw-r--r--  1 root   wheel     0B 25 Jan 12:39 test

# rm media-test
media-test

# chmod uog-t .
# ls -ald .
drwxrwxr-x  84 incoming  incoming    93B 26 Jan 14:26 ./

# rm media
rm: media: Operation not permitted

# chmod uog+t .
# ls -ald .
drwxrwxr-t  84 incoming  incoming    93B 26 Jan 14:26 ./

# rm media
rm: media: Operation not permitted
 

scotia

Member

Thanks: 2
Messages: 39

#23
Code:
# cd /z/incoming/
# ls -lo media
lrwxrwxrwx  1 root  wheel  sunlnk,uarch   21B 31 Mar  2017 media@ -> /z/media-05/incoming/

# ls -lod ./
drwxrwxr-t  84 incoming  incoming  uarch   93B 26 Jan 14:26 ./
I put -d on the last one to reduce the output.
 
Top