"No such file or directory" though "File exists"?

Hi,

I cannot upgrade libnotify on one on my servers. The message is:

Bash:
root@magrathea:/usr/local/bin # pkg install libnotify
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
pkg: php56-mbstring has a missing dependency: oniguruma6
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        libnotify: 0.7.7_1
        py27-acme: 0.22.2,1

Number of packages to be installed: 2

The process will require 1 MiB more space.

Proceed with this action? [y/N]: y
[1/2] Installing libnotify-0.7.7_1...
[1/2] Extracting libnotify-0.7.7_1: 100%
pkg: Fail to rename /usr/local/bin/.notify-send.C6hnQOXTnP1f -> /usr/local/bin/notify-send:No such file or directory

Checking for the files in question I can see the following:

Bash:
root@magrathea:/usr/local/bin # ls -la /usr/local/bin/.notify-send.C6hnQOXTnP1f                                                             
-rwxr-xr-x  1 root  wheel  11848 May 10 08:08 /usr/local/bin/.notify-send.C6hnQOXTnP1f                                                     
root@magrathea:/usr/local/bin # file /usr/local/bin/.notify-send.C6hnQOXTnP1f                                                               
/usr/local/bin/.notify-send.C6hnQOXTnP1f: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.1, FreeBSD-style, stripped

Bash:
root@magrathea:/usr/local/bin # ls -la /usr/local/bin/notify-send                                                                           
ls: /usr/local/bin/notify-send: No such file or directory                                                                                   
root@magrathea:/usr/local/bin # file /usr/local/bin/notify-send                                                                             
/usr/local/bin/notify-send: cannot open `/usr/local/bin/notify-send' (No such file or directory)

When I try to move/rename the file myself I get the same message:
Bash:
root@magrathea:~ # mv /usr/local/bin/.notify-send.C6hnQOXTnP1f /usr/local/bin/notify-send
mv: rename /usr/local/bin/.notify-send.C6hnQOXTnP1f to /usr/local/bin/notify-send: No such file or directory

Symlinking or touching the destination file says the following though:
Bash:
root@magrathea:~ # ln -s /usr/local/bin/.notify-send.C6hnQOXTnP1f /usr/local/bin/notify-send
ln: /usr/local/bin/notify-send: File exists
root@magrathea:~ # touch /usr/local/bin/notify-send
touch: /usr/local/bin/notify-send: File exists

Trying to edit the file in vim fails with:
Code:
"/usr/local/bin/notify-send" E212: Can't open file for writing

Clearing flags won't help either, because the file does not exist for these tools. But somehow this file still exists in a sort of limbo?
Bash:
root@magrathea:~ # chflags nouchg /usr/local/bin/notify-send
chflags: /usr/local/bin/notify-send: No such file or directory

Does anybody know what I can do here? The zpool is healthy and even a scrub did not return any errors.

Thanks for your help.
 
I'm guessing /usr/local/bin/notify-send is a symlink that points to a non-existing file. I.e. it's a broken symlink.
 
Unfortunately it is not. "ls -la" does not show the file and an "rm" tells me there is no file. If it was a symlink then I would be able to delete it. And a "file" on a broken symlink would state that it is a broken symlink. At least that's my experience.

Must be something else. Maybe more low level? And I also don't think it has anything to do with the port. I am pretty puzzled. Never had this kind of behavior ever
 
First: what kind of ZFS pool do you use? In specific: is it a mirror, a raidz or such, so does it use multiple storage media? It is definitely kind of bizarre, but one of the things ZFS relies on when it comes to protecting data integrity is to compare across multiple storage media. As such my question.

Leading up to: what happens if you use a rescue system to check the pool? So: boot using a rescue CD and then important / mount your pool manually and check the status of <mount point>usr/local/bin/notify-send? That should also rule out the option of anything being left in a write cache somewhere.
 
It's a simple mirror with two rotating disks. No L2 cache and no ZIL just the basic stuff. I have to reboot for a kernel update anyway. So I will do that in the next couple of days. Maybe I manage to do it tonight. And maybe it is just something stuck in the RAM. But then it looks like a bug.

But for completeness, some data:

ZFS Storage pool Version: 5000
ZFS Filesystem Version: 5

Code:
  pool: zroot                                                                                                                              
state: ONLINE                                                                                                                            
  scan: scrub repaired 0 in 0h47m with 0 errors on Tue May 22 14:33:43 2018                                                                
config:                                                                                                                                    
                                                                                                                                           
        NAME                               STATE     READ WRITE CKSUM                                                                      
        zroot                              ONLINE       0     0     0                                                                      
          mirror-0                         ONLINE       0     0     0                                                                      
            diskid/DISK-WD-WX41A8200573p4  ONLINE       0     0     0                                                                      
            diskid/DISK-WD-WX71A96R6CC9p4  ONLINE       0     0     0                                                                      
                                                                                                                                           
errors: No known data errors

Code:
# zpool get all zroot                                                                        
NAME   PROPERTY                       VALUE                          SOURCE                                                                
zroot  size                           456G                           -                                                                    
zroot  capacity                       12%                            -                                                                    
zroot  altroot                        -                              default                                                              
zroot  health                         ONLINE                         -                                                                    
zroot  guid                           3463442437644807507            default                                                              
zroot  version                        -                              default                                                              
zroot  bootfs                         zroot/ROOT/default             local                                                                
zroot  delegation                     on                             default                                                              
zroot  autoreplace                    off                            default                                                              
zroot  cachefile                      -                              default                                                              
zroot  failmode                       wait                           default                                                              
zroot  listsnapshots                  off                            default                                                              
zroot  autoexpand                     off                            default                                                              
zroot  dedupditto                     0                              default                                                              
zroot  dedupratio                     1.00x                          -                                                                    
zroot  free                           401G                           -                                                                    
zroot  allocated                      55.4G                          -                                                                    
zroot  readonly                       off                            -                                                                    
zroot  comment                        -                              default                                                              
zroot  expandsize                     -                              -                                                                    
zroot  freeing                        0                              default                                                              
zroot  fragmentation                  16%                            -                                                                    
zroot  leaked                         0                              default                                                              
zroot  feature@async_destroy          enabled                        local                                                                
zroot  feature@empty_bpobj            active                         local                                                                
zroot  feature@lz4_compress           active                         local                                                                
zroot  feature@multi_vdev_crash_dump  enabled                        local                                                                
zroot  feature@spacemap_histogram     active                         local                                                                
zroot  feature@enabled_txg            active                         local                                                                
zroot  feature@hole_birth             active                         local                                                                
zroot  feature@extensible_dataset     enabled                        local                                                                
zroot  feature@embedded_data          active                         local                                                                
zroot  feature@bookmarks              enabled                        local                                                                
zroot  feature@filesystem_limits      enabled                        local                                                                
zroot  feature@large_blocks           enabled                        local                                                                
zroot  feature@sha512                 enabled                        local                                                                
zroot  feature@skein                  enabled                        local

Code:
 # zfs get all zroot                                                                          
NAME   PROPERTY                VALUE                   SOURCE                                                                              
zroot  type                    filesystem              -                                                                                  
zroot  creation                Tue Jul 11 13:16 2017   -                                                                                  
zroot  used                    55.4G                   -                                                                                  
zroot  available               386G                    -                                                                                  
zroot  referenced              96K                     -                                                                                  
zroot  compressratio           1.62x                   -                                                                                  
zroot  mounted                 yes                     -                                                                                  
zroot  quota                   none                    default                                                                            
zroot  reservation             none                    default                                                                            
zroot  recordsize              128K                    default                                                                            
zroot  mountpoint              /zroot                  local                                                                              
zroot  sharenfs                off                     default                                                                            
zroot  checksum                on                      default                                                                            
zroot  compression             lz4                     local                                                                              
zroot  atime                   off                     local                                                                              
zroot  devices                 on                      default                                                                            
zroot  exec                    on                      default                                                                            
zroot  setuid                  on                      default                                                                            
zroot  readonly                off                     default                                                                            
zroot  jailed                  off                     default                                                                            
zroot  snapdir                 hidden                  default                                                                            
zroot  aclmode                 discard                 default                                                                            
zroot  aclinherit              restricted              default                                                                            
zroot  canmount                on                      default                                                                            
zroot  xattr                   off                     temporary                                                                          
zroot  copies                  1                       default                                                                            
zroot  version                 5                       -                                                                                  
zroot  utf8only                off                     -                                                                                  
zroot  normalization           none                    -                                                                                  
zroot  casesensitivity         sensitive               -                                                                                  
zroot  vscan                   off                     default                                                                            
zroot  nbmand                  off                     default                                                                            
zroot  sharesmb                off                     default                                                                            
zroot  refquota                none                    default                                                                            
zroot  refreservation          none                    default                                                                            
zroot  primarycache            all                     default                                                                            
zroot  secondarycache          all                     default                                                                            
zroot  usedbysnapshots         0                       -                                                                                  
zroot  usedbydataset           96K                     -                                                                                  
zroot  usedbychildren          55.4G                   -                                                                                  
zroot  usedbyrefreservation    0                       -                                                                                  
zroot  logbias                 latency                 default                                                                            
zroot  dedup                   off                     default                                                                            
zroot  mlslabel                                        -                                                                                  
zroot  sync                    standard                default                                                                            
zroot  refcompressratio        1.00x                   -                                                                                  
zroot  written                 0                       -                                                                                  
zroot  logicalused             83.5G                   -                                                                                  
zroot  logicalreferenced       40.5K                   -                                                                                  
zroot  volmode                 default                 default                                                                            
zroot  filesystem_limit        none                    default                                                                            
zroot  snapshot_limit          none                    default                                                                            
zroot  filesystem_count        none                    default                                                                            
zroot  snapshot_count          none                    default                                                                            
zroot  redundant_metadata      all                     default                                                                            
zroot  org.freebsd.ioc:active  no                      local                                                                              
zroot  com.sun:auto-snapshot   true                    local

smartctl data is also fine. All error counts are zero. SMART data might not always be trusted. But it is a good indicator nonetheless.
 
Unfortunately it is not. "ls -la" does not show the file and an "rm" tells me there is no file. If it was a symlink then I would be able to delete it. And a "file" on a broken symlink would state that it is a broken symlink. At least that's my experience.
Yes, I agree.
Must be something else. Maybe more low level? And I also don't think it has anything to do with the port.
Yeah, I don't think it's anything specific to the port, it just happens to involve a file the port/package tries to write to.
I am pretty puzzled. Never had this kind of behavior ever
Neither have I, not on UFS, UFS2 or ZFS.

Does fstat /usr/local/bin/notify-send output anything? Or does it complain about a non-existing file too?
(I'm thinking there's an application running that's keeping the file descriptor open while the file itself has already been removed.)
 
Back
Top