Solved What is the Best Choice for *Non-Microsoft* External File System?

D

Dave-D

Problem:

I have been using GPT partitions with Fat32 file system, mainly because it is said to be the most commonly supported filesystem among different systems, which I do not doubt.

However, change of plans. I no longer trust the Fat32 file system.

I just had a bad experience with missing files, nothing like that to drive you crazy when you're trying to get work done and cannot find the files you just created yesterday.

So I pulled the files in from a backup. But they refused to show up in FreeBSD / zfs.

So I found out what was going on...

1.) Using a linux computer with ext4, I copied 12 files from ext4 (external hard drive) to fat 32 (external hard drive), fat32 shows 12 files copied.
So far, so good.
2,) Using a FreeBSD computer with zfs, I copied 12 files from fat 32 (external hard drive), zfs shows only 7 files copied.
Not good.
And even worse...
Not only did it refuse to copy 5 files over, but it then deleted those same 5 files on the source (fat32 file system on external hard drive).

Repeated this experiment twice, with the same exact result.
In case it matters, all permissions were the same, groups the same, all are libreoffice spreadsheet files. But since we're dealing with fat32, all owner/group/permissions are lost anyway.

I then repeated the same experiment using the ext2 file system on the external (transfer) device.

1.) Using a linux computer with ext4, I copied 12 files from ext4 (external hard drive) to ext2 (external thumb drive), ext2 shows 12 files copied.
2.) Using a FreeBSD computer with zfs, I copied 12 files from ext2 (external thumb drive), zfs shows all 12 files copied.
Works fine, as it should.

Solution:

So, here's what I'm wondering.

I would like to know, based on real experiences, what is the best file system for external use (thumb drives and external hard drive(s)/enclosures)?

Some possibilities at this point:

ext2​
​
ext3/4 - (I heard there can be incompatibilities between linux versions / ext3/4, not sure if true, but journaling is not good for thumb drives.)​
​
UFS​
​
ZFS (on non-hardware raid multi-drive enclosures) - but a pain to mount the device?​
​
Others ??​


Might also be wise to do away with thumb drives and use only hard drives?


Thanks for your help!
 
How did you copy the files?

ZFS is annoying when it comes to mounting, but it is the only real filesystem supported with good software on FreeBSD, Linux, macOS and now even Windows.

I use external M.2 to USB enclosures with small SSDs I pick up on ebay instead of 16 GB sticks and the like.
 
cracauer@ said:
So thunar failed to give error messages when copying failed?
Click to expand...

I also have the Thunar "Verify file checksums on copy" option enabled.

There was no error until I copied the files from FreeBSD using Thunar,
when I copied the original source (12 files), but it only (actually) copied 7, then deleted the 5 source files that it did not copy.
No error messages on FreeBSD/Thunar, whatsoever.
The only reason I noticed it was because those 5 missing files where the files I had intended to work on.
I could not believe what I was seeing.
So I repeated the steps again to verify.
All happened exactly as stated.
 
Dave-D said:
when I copied the original source (12 files), but it only (actually) copied 7, then deleted the 5 source files that it did not copy.
Click to expand...
This is not a file system problem. The target file system is (in practice) not capable of deleting 5 files on the source file system. The problem is either the copying tool (and I have no idea what "Thunar" is, probably an overly complex and badly constructed file manager tool), or it is PEBCAK, meaning user error.

In particular, the problem is not the use of a Microsoft file system on the target. There may be some valid reasons to dislike Microsoft products, but this is not one of them.

Obnoxious side remark: I'm always amazed how people let their paranoia cloud their judgement.
 
I've never had any issues with fat32 thumb drives... but make sure you sync before removing the drive after writing. umount *probably* does that, but I still type out sync from habit :-) I'd recommend trying again, but use any gui tools. do the copy / read testing from the command line. That'll eliminate several areas of uncertainty.

I've used ZFS on thumb drives, too. Seems to work ok for basic stuff, but zfs send/recv doesn't play nice with cheap thumb drives at all, so I don't bother anymore.

Mostly my sneakernet use of thumb drives is pc to android-tv. For that, fat/exfat is probably the only thing that'll even work.
 
ralphbsz said:
This is not a file system problem. The target file system is (in practice) not capable of deleting 5 files on the source file system. The problem is either the copying tool (and I have no idea what "Thunar" is, probably an overly complex and badly constructed file manager tool), or it is PEBCAK, meaning user error.

In particular, the problem is not the use of a Microsoft file system on the target. There may be some valid reasons to dislike Microsoft products, but this is not one of them.

Obnoxious side remark: I'm always amazed how people let their paranoia cloud their judgement.
Click to expand...

If you can't stick with the facts, and be nice, then don't bother.

Not interested.

*ignore*
 
Dave-D said:
Repeated this experiment twice, with the same exact result.
Click to expand...
Did you try a different thumb drive/USB stick? Even better a different brand?

I've had case sensitivity issues but never ever had a copy that deleted files.

Was it definitely a copy? Not a cut/move? Maybe the copying-to-destination part file failed but the delete-after-copy still went ahead (if it even works like behind the scenes, this is grasping at straws!).
 
richardtoohey2 said:
Did you try a different thumb drive/USB stick? Even better a different brand?

I've had case sensitivity issues but never ever had a copy that deleted files.

Was it definitely a copy? Not a cut/move? Maybe the copying-to-destination part file failed but the delete-after-copy still went ahead (if it even works like behind the scenes, this is grasping at straws!).
Click to expand...
It was definitely a copy, not a move. I paid very close attention to what I was doing when I attempted to duplicate the behavior.

Thumb drives are SanDisk.

However, I was NOT using a thumb drive when the problem occured.
I was using an external hard drive (on the first attempt) when the files were not moved, then deleted.
Drive was a new Western Digital 250GB Sata.
 
exFAT. Your bad experience with FAT32 has some explanation like power loss or 3rd party implementation.
 
+1. For my needs, exFAT is the only file system compatible with FreeBSD, Linux, macOS and Windows that supports files larger than 4GB.
 
First, FAT32 is a case-insensitive filesystem. Try to copy file1a, file1A and File1A to it and see what happens.

Second, files exist or don't exist irrespective of whether Thunar can show them as present. Bugs in Thunar are not bugs in FreeBSD, nor are they bugs in FAT32 implementations. Unless you verify from the command line with ls(1) what the state of the filesystem is, reports of missing files are not to be taken seriously.

Third, I found this link that helps put any statements made by the original poster in context: https://forums.freebsd.org/threads/is-freebsd-for-business-use-no-longer-an-option.90050/
 
JordanG said:
Third, I found this link that helps put any statements made by the original poster in context: https://forums.freebsd.org/threads/is-freebsd-for-business-use-no-longer-an-option.90050/
Click to expand...

Yes, I'm just a flawed human being, struggling to make sense of it all.

Just like all of you.

Does that in some way surprise you?

And yes, I have a personality, that I am not afraid to display.

And yes, I can sometimes go to extremes, that's my personality.

But in the end, it all comes full circle and I make sense of things.

At least I'm not boring.
 
  • Like
Reactions: mro
That's twice now, in one short thread.

It will be a very sad thing if this forum turns from one of the greatest,
most kind, and most helpful forums I have ever known,
into one of personal attacks.

As for me, I will think twice before posting here again.

Two years ago? Incredible. Now? Not so much.
 
With ZFS there is issues with version of pool, like: can use external hdd with ZFS with FreeBSD, Linux Mint, but not Debian stable.

What about NTFS? I use it for years, yes, on FreeBSD NTFS is slower than on Lin/Win, but it works good.
 
Dave-D said:
As for me, I will think twice before posting here again.

Two years ago? Incredible. Now? Not so much.
Click to expand...
That's fine. Some communities are more suitable than others for specific needs. The active members here are pretty consistent for (at least) the last two years so if you still aren't making progress here, perhaps try the official mailing lists or IRC.

As for your issue; I find exFat to be a good compromise. It is native to most operating systems and with FUSE's exfat, you can pretty much get it anywhere. I have normalized on it for most of my removable media.
 
CeXP1917 said:
With ZFS there is issues with version of pool, like: can use external hdd with ZFS with FreeBSD, Linux Mint, but not Debian stable.

What about NTFS? I use it for years, yes, on FreeBSD NTFS is slower than on Lin/Win, but it works good.
Click to expand...

Yes, you need to restrict features when creating the pool.
 
** More Issues **

I've been attempting to use ext2.

1. Used the native FreeBSD support for ext2, with no additional configuration.
External ext2 drive mounted great, used and unmounted 3 times.

2. 4th time ext2 external drive mounted with errors. Something about "illegal operation."

3. Could not cleanly unmount.

4. External ext2 drive is now stuck in read only mode. Could see all files, but everything marked with a padlock.

5. Installed the fusefs-ext2 package, added fusefs_load="YES" to /boot/loader.conf.
Reboot. No change. External ext2 now mounts without errors, all files visible, but drive and all files show read only.

6. Attempted to access same external ext2 drive from a linux computer, no problem.
Can read and write. No indication of any errors. All files visible.

7. Attempted to once again access the same external drive from FreeBSD, drive still shows read only,
only now, *ALL DATA IS GONE.* Verified from command line. All data is gone. (or, more likely, doesn't show.)


Then, attempted from FreeBSD to access ext3 external drive, drive shows read only.
Then, attempted from FreeBSD to access ext4 external drive, drive shows read only.

Apparently all ext* formatted drives are now unusable.

Not having much luck using FreeBSD with file systems on external drives.

I'm starting to think maybe they all have issues.

And that maybe it would be best to stick with FreeBSD's own file systems (UFS and/or ZFS)?

Thoughts and experiences, please...

Thank you.
 
Dave-D said:
** More Issues **

I've been attempting to use ext2.

1. Used the native FreeBSD support for ext2, with no additional configuration.
External ext2 drive mounted great, used and unmounted 3 times.

2. 4th time ext2 external drive mounted with errors. Something about "illegal operation."

3. Could not cleanly unmount.

4. External ext2 drive is now stuck in read only mode. Could see all files, but everything marked with a padlock.

5. Installed the fusefs-ext2 package, added fusefs_load="YES" to /boot/loader.conf.
Reboot. No change. External ext2 now mounts without errors, all files visible, but drive and all files show read only.

6. Attempted to access same external ext2 drive from a linux computer, no problem.
Can read and write. No indication of any errors. All files visible.

7. Attempted to once again access the same external drive from FreeBSD, drive still shows read only,
only now, *ALL DATA IS GONE.* Verified from command line. All data is gone. (or, more likely, doesn't show.)


Then, attempted from FreeBSD to access ext3 external drive, drive shows read only.
Then, attempted from FreeBSD to access ext4 external drive, drive shows read only.

Apparently all ext* formatted drives are now unusable.

Not having much luck using FreeBSD with file systems on external drives.

I'm starting to think maybe they all have issues.

And that maybe it would be best to stick with FreeBSD's own file systems (UFS and/or ZFS)?

Thoughts and experiences, please...

Thank you.
Click to expand...

Anything in `dmesg` during those mounts?
 
cracauer@ said:
Anything in `dmesg` during those mounts?
Click to expand...
Nothing special that might indicate a problem.

So, thought I would try again, this time with the UFS file system.

1. Formated external hard drive with UFS2 file system, from a FreeBSD computer, using bsdinstall.

2. Installed pkg fusefs-ufs, fusefs_load="YES" already in /boot/loader.conf.

3. Reboot

4. Connected external drive with UFS2 file system to FreeBSD computer, instantly recognized, instant (very fast) access,
copied about 2TB worth of files onto drive. Everything worked great.

5. Umounted and remounted external drive 3 or 4 times, same behavior. Instant mounts, read/write works, fast unmount.

6. On 5th attempt to remount, started throwing same type of errors as ext2/3/4, "invalid command," /etc.,
yet it showed the first level of folders. All other files previously on drive missing (or at least not showing.) command line shows all files missing (or at least not showing.). Drive does not say "read only mode" but it refuses attempted
write operations.

7. And, this is a new one. Now, after rebooting, the Thunar file manager refuses to run. Click to run it, nothing happens.
Now we've done some kind of damage to the operating system.


One other thought...
Could it be some kind of a hardware problem? The USB port? The drive enclosure?
All these tests and results came from the same client, running FreeBSD 14.2 amd64 w/XFCE.

???
 
A few more thoughts...

Could it have anything to do with:

1.) the computer automatically mounting the external drive, but the automount package is not installed,

2.) no dedicated mount point (other than /media), prior to the computer automatically mounting the drive,

3.) unmounting the drive by selecting either "unmount" or "safely remove" from Thunar,

Or

4.) the external drive enclosure having a "power on/off" button, which unmounts when turned off,
but I'm thinking you have no way of knowing if the drive is fully synced at the time of hitting the "off"
button, so I selected to unmount, then turn off the power only after the computer says "safe to remove."

??
 
You must log in or register to reply here.
Back
Top