can't remove files in lost+found after fsck creates them on usb drive

K

ko56

I am running 15.0-RELEASE, and have a problem with an external USB drive. I had to run fsck_ffs on it because I disconnected it improperly.
After fsck fixed everything, it created a lost+found/ with files in it such as

Code: 
# ls -l | head
total 636
-rwSr-S---  1 root       91458984                    0 Nov 24  1972 #14337884
-rw-------  1 65536      4294836269                  0 Dec 20  151635083 #14954722
l-wxrw---T  1 root       wheel                       0 Apr  8  1991 #14999341 ->
s-wx-wx--T  1 daemon     676643492                   0 bad date val #15000653
-rw-------  1 root       95614408                    0 Jan 11  1973 #16406834
-rw-------  1 root       95616632                    0 Jan 11  1973 #16442418
-rw-------  1 root       3010720                  4096 Feb  4  1970 #18230170
-rw-------  1 root       3011282                  4096 Feb  4  1970 #18230171

I want to remove these files, but it is proving to be challenging :)
root can't remove them, perhaps because they have strange permission flags set. But chflags has problems:

Code: 
# chflags -v 0 \#14337884
chflags: #14337884: Operation not permitted
# chflags -v nouunlnk \#14337884
chflags: #14337884: Operation not supported

I am running ZFS and I know that there are ZFS issues with chflags and unsupported flags,
but I tried the above using a live CD where FreeBSD runs ufs.

Any help would be appreciated.
 
ko56 said:
I am running ZFS
Click to expand...
Contradictory statement. ZFS doesn't create a lost+found and neither can you run fsck(8) on it. It's the external disk that's using UFS? Why do you think it matters which filesystem the host itself uses?

ko56 said:
I know that there are ZFS issues with chflags and unsupported flags,
Click to expand...
And? What does that have to do with an external disk that's formatted with UFS?

Keep in mind that the files in lost+found are, by definition, corrupted.
 
SirDice said:
Contradictory statement.
Click to expand...
Yes. But only apparently. I expressed myself badly.
What I meant to say is that my system runs ZFS (which may be irrelevant as you say below). The USB drive has a UFS file system on it, which was corrupted.
SirDice said:
ZFS doesn't create a lost+found and neither can you run fsck(8) on it. It's the external disk that's using UFS? Why do you think it matters which filesystem the host itself uses?
Click to expand...
Yes, ZFS and fsck don't mix. The USB drive had the corrupted UFS file system. And running fsck fixed it. It also created the lost+found, whose (junk) contents I now want to remove.
SirDice said:
And? What does that have to do with an external disk that's formatted with UFS?
Click to expand...
Apparently a confusion on my part. I read in the forums about other people having similar problems with removing files (UFS or ZFS, I'm not clear), and other people having chflags problems with ZFS.
This led me to believe that there is some mysterious connection between the file system of the host and that of the external drive.
SirDice said:
Keep in mind that the files in lost+found are, by definition, corrupted.
Click to expand...
Of course.

I hope I've made things clear. Do you have any advice at this point?
 
SirDice said:
Did you run fsck(8) before or after it was mounted?
Click to expand...
I did this whole fsck business a while ago, and I really don't remember whether the file system was mounted or not.
(Now the file system seems OK, I use it for weekly backups, and they seem fine. I just can't clean up lost+found.)
 
You might want to run a check on the disk itself too. While unplugging without unmounting isn't the greatest, it generally doesn't cause this much problems, certainly not if happened only once. So there might be other reasons for the filesystem corruption.

But I'd do a full fsck(8) once more, making sure the filesystem isn't mounted. fsck(8) can't fix certain issues when the filesystem is mounted read/write.
 
SirDice said:
You might want to run a check on the disk itself too. While unplugging without unmounting isn't the greatest, it generally doesn't cause this much problems, certainly not if happened only once. So there might be other reasons for the filesystem corruption.

But I'd do a full fsck(8) once more, making sure the filesystem isn't mounted. fsck(8) can't fix certain issues when the filesystem is mounted read/write.
Click to expand...
OK, thanks, will do and will report later today. (How would you check the disk itself?)
 
ko56 said:
I did this whole fsck business a while ago, and I really don't remember whether the file system was mounted or not.
(Now the file system seems OK, I use it for weekly backups, and they seem fine. I just can't clean up lost+found.)
Click to expand...
it is an external USB disk, right? copy the data from it and reformat it. problem solved.
 
SirDice said:
You might want to run a check on the disk itself too. While unplugging without unmounting isn't the greatest, it generally doesn't cause this much problems, certainly not if happened only once. So there might be other reasons for the filesystem corruption.

But I'd do a full fsck(8) once more, making sure the filesystem isn't mounted. fsck(8) can't fix certain issues when the filesystem is mounted read/write.
Click to expand...
OK, here is the check with the file system unmounted:

Code: 
[ko@hn ~]$ su
Password:
# fsck -y /dev/da0p2
** /dev/da0p2 (NO WRITE)
** Last Mounted on /media/FreeBSD
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
68695 files, 8352721 used, 48777877 free (6477 frags, 6096425 blocks, 0.0% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****

***** FILE SYSTEM WAS MODIFIED *****
#
 
SirDice said:
smartctl(8) (from sysutils/smartmontools) is a good indicator. But it may or may not work with external drives, some of these (cheap) controllers don't pass the SMART data.
Click to expand...
OK, I did that. The output is long, here is an excerpt:

Rich (BB code): 
# smartctl -x /dev/da0
smartctl 7.5 2025-04-30 r5714 [FreeBSD 15.0-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 2.5" HDD MK..59GSXP (AF)
Device Model:     TOSHIBA MK5059GSXP
Serial Number:    439QCC7YT
LU WWN Device Id: 5 000039 4b3f81842
Firmware Version: GN001A
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database 7.5/5706
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon May  4 17:48:50 2026 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM level is:     128 (minimum power consumption without standby)
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

. . .

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     PO-R--   100   100   050    -    0
  2 Throughput_Performance  P-S---   100   100   050    -    0
  3 Spin_Up_Time            POS--K   100   100   001    -    2168
  4 Start_Stop_Count        -O--CK   100   100   000    -    730
  5 Reallocated_Sector_Ct   PO--CK   100   100   050    -    0
  7 Seek_Error_Rate         PO-R--   100   100   050    -    0
  8 Seek_Time_Performance   P-S---   100   100   050    -    0
  9 Power_On_Hours          -O--CK   084   084   000    -    6452
 10 Spin_Retry_Count        PO--CK   114   100   030    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    305
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    0
192 Power-Off_Retract_Count -O--CK   100   100   000    -    90
193 Load_Cycle_Count        -O--CK   099   099   000    -    19861
194 Temperature_Celsius     -O---K   100   100   000    -    35 (Min/Max 15/42)
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O--CK   100   100   000    -    0
198 Offline_Uncorrectable   ----CK   100   100   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
220 Disk_Shift              -O----   100   100   000    -    85
222 Loaded_Hours            -O--CK   100   100   000    -    89
223 Load_Retry_Count        -O--CK   100   100   000    -    0
224 Load_Friction           -O---K   100   100   000    -    0
226 Load-in_Time            -OS--K   100   100   000    -    220
240 Head_Flying_Hours       P-----   100   100   001    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

The P's (prefailure warnings) seem worrisome.
 

cracauer@ said:
Anything related in `dmesg` when the chflags call is being rejected?
Click to expand...
No, I couldn't see anything.

On 2nd thought, not quite anything:

Rich (BB code): 
[ko@hn ~]$ dmesg | tail -25
umass0:  SCSI over Bulk-Only; quirks = 0x0
umass0:2:0: Attached to scbus2
da0 at umass-sim0 bus 0 scbus2 target 0 lun 0
da0: <StoreJet Transcend 0> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number 439QCC7YT
da0: 40.000MB/s transfers
da0: 476940MB (976773168 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
WARNING: /media/FreeBSD was not properly dismounted
hdacc1: Unexpected unsolicited response with tag 63: ffffffff

This is despite what fsck says, as I posted in #11, that everything is OK.
 
Can you delete the lost+found directory itself?

Or want to try and find out what caused the issue in the first place?

I'm not sure deleting the directory will necessarily help, but might be something to try that is less invasive than a re-format.
 
richardtoohey2 said:
Can you delete the lost+found directory itself?
Click to expand...
Rich (BB code): 
# mount /dev/da0p2 /media/FreeBSD/
# cd /media/FreeBSD/
# ls
hn-backup    lost+found
# rm -rf lost+found/
rm: lost+found/#19570070: Operation not supported
rm: lost+found/#19570070: Operation not permitted
rm: lost+found/#14337884: Operation not permitted
rm: lost+found/#14954722: Operation not permitted
rm: lost+found/#14999341: Operation not permitted
...
rm: lost+found/#19271229: Operation not supported
rm: lost+found/#19271498: Operation not supported
rm: lost+found/#19287424: Operation not supported
...
richardtoohey2 said:
Or want to try and find out what caused the issue in the first place?
Click to expand...
Not so much what caused the issue, but I want to know what's preventing me from deleting all this junk.
richardtoohey2 said:
I'm not sure deleting the directory will necessarily help, but might be something to try that is less invasive than a re-format.
Click to expand...

No, deleting it would accomplish what I want.
Assuming that the drive is healthy ... but smartctl seems to be casting some doubts on this.
 
I had a (probably wrong) recollection that deleting the parent directory had helped me at one point in the past with "funny files" but obviously no help in your situation.
 
bakul said:
show us the output of "ls -lo" to see what flags are set but looking at the output of ls -l it appears there are a lot of bad values, which means the inodes themselves are corrupted. I think you should do what kent_dorfman766 says and just copy all the data to a new device.
Click to expand...
Rich (BB code): 
# ls -lo | head
total 636
-rwSr-S---  1 root       91458984   arch,schg,sunlnk,snapshot,uarch,hidden,uunlnk,opaque,reparse,sparse,system                                                           0 Nov 24  1972 #14337884
-rw-------  1 65536      4294836269 sappnd,schg,sunlnk,snapshot,uappnd,uarch,hidden,uchg,nodump,uunlnk,offline,opaque,rdonly,reparse,sparse,system                       0 Dec 20  151635083 #14954722
l-wxrw---T  1 root       wheel      sappnd,arch,schg,sunlnk,snapshot,uappnd,uarch,hidden,uchg,nodump,uunlnk,offline,opaque,rdonly,reparse,sparse,system                  0 Apr  8  1991 #14999341 ->
s-wx-wx--T  1 daemon     676643492  sappnd,arch,schg,sunlnk,snapshot,uappnd,uarch,hidden,uchg,nodump,uunlnk,offline,opaque,rdonly,reparse,sparse,system                  0 bad date val #15000653
-rw-------  1 root       95614408   schg,sunlnk,snapshot,hidden,uunlnk,offline,opaque,rdonly,reparse                                                                     0 Jan 11  1973 #16406834
-rw-------  1 root       95616632   schg,sunlnk,snapshot,uarch,hidden,offline,opaque,rdonly,reparse,system                                                               0 Jan 11  1973 #16442418
-rw-------  1 root       3010720    sappnd,arch,snapshot,hidden,uunlnk,rdonly,sparse                                                                                  4096 Feb  4  1970 #18230170
-rw-------  1 root       3011282    sappnd,arch,snapshot,uappnd,hidden,uchg,uunlnk,offline,rdonly,sparse                                                              4096 Feb  4  1970 #18230171
-rw-------  1 root       3011828    sappnd,arch,snapshot,hidden,rdonly,reparse,sparse,system                                                                          4096 Feb  4  1970 #18230172
#
 
Seems quite messed up. Typically very few files have at most one or two flags (uarch, schg) set. I wouldn't trust anything on such a filesystem....
 
point is...you can take the quick heavyweight fix and have your problem solved in a short time, or continue to look for the magic bullet and spend far more resources with no payoff. I understand the value of understanding the cause of the problem and choosing the correct solution but given that you don't fully understand what caused this problem in the first place,wereyou working for me then I'd instruct you to take the sledgehammer approach and "git er done!"
 
Completely agree with cracauer@: This file system is messed up, and that must have been FreeBSD's and UFS's doing.

bakul said:
show us the output of "ls -lo" to see what flags are set but looking at the output of ls -l it appears there are a lot of bad values,
Click to expand...
Indeed, the output of both ls commands shows that much of the file metadata is likely corrupted. While theoretically it could be possible to create such flags, it makes no sense. And the "bad date" in the first ls output means that some other data on disk is being mis-interpreted. Which means that the combination of unclean shutdown and fsck messed it up.

ko56 said:
OK, I did that. The output is long, here is an excerpt:
...
The P's (prefailure warnings) seem worrisome.
Click to expand...
No, you're misinterpreting the output. It just means that this value CAN BE a prefailure warning. Not that it is one. For example, the accumulated run time of about 6K hours, and the zero error rates all favor this being a healthy drive.
 
ko56 said:
OK, here is the check with the file system unmounted:

Code: 
[ko@hn ~]$ su
Password:
# fsck -y /dev/da0p2
** /dev/da0p2 (NO WRITE)
** Last Mounted on /media/FreeBSD
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
68695 files, 8352721 used, 48777877 free (6477 frags, 6096425 blocks, 0.0% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****

***** FILE SYSTEM WAS MODIFIED *****
#
Click to expand...
It says "NO WRITE" so something still has the filesystem locked. (Yes, fsck has quite a few ... things)
 
You must log in or register to reply here.
Back
Top