Other Which filesystem for a portable 1TB USB drive?

hanzer

hanzer

I have a new 1TB portable USB hard-drive that will be used to share data among FreeBSD, CentOS, and Ubuntu machines (all OS's are kept up-to-date). The hard-drive is probably pre-formatted with NTFS but I've had problems with that filesystem in the past and I hope to find something that is more stable/reliable on the unix-like systems.

The options are:
  1. UFS
  2. EXT4
  3. XFS
  4. NTFS
How well does FreeBSD deal with XFS? It seems to be very reliable on my CentOS machine and I assume Ubuntu will use the same XFS software as CentOS.

Alternatively, did EXT4 support ever mature in FreeBSD?

Finally, are there any recommendations for a 5th or 6th way (an option I missed)?
 
Linux's UFS support doesn't really work on FreeBSD, I do believe it's actually Sun's UFS they're supporting, FreeBSD's UFS is quite different and not compatible. So that one's off the table. Not sure about XFS on FreeBSD, I'd guess that one's out too.

Ext2/3 should probably be fine, not sure if ext4 is best for external drives. Oddly enough the one filesystem that has good support by pretty much every OS is Microsoft's FAT16/FAT32.

In my view we really need a good portable filesystem (preferably BSD licensed) for external drives.
 
As you probably know, the main issue with FAT32 is that it has some pretty tight maximum file sizes and folder sizes, however it is the most portable.

I am always a little scared to use Ext2 or Ext3 because Linux is so volatile I can honestly imagine the developers removing these out of the kernel by default and saying "just use Ext4, its newer and better".

For large files, I always have best luck with NTFS (via fuse on FreeBSD). Most operating systems support this format some way or another these days because Microsoft is still so popular.

For one that could be perfect but I have not yet used for hard disks before is UDF (https://en.wikipedia.org/wiki/Universal_Disk_Format). Apparently it can be used as a standard filesystem and has some pretty large limits. I don't know if FreeBSD supports this or has the required tools to use yet though. Again, I have only used this on DVDs but apparently it has the potential to be used for so much more.
 
Personally I backup on cheap SATA drives using a trayless drive carrier.
I use ext2fs for that since years because it keeps the basic unix permission data and is supported by all OSes (even Windows, see here).
Ext3 and 4 are imho potentially problematic because of the journal stuff supported only by Linux (->potential data loss/corruption on other systems).
Using NTFS is not a good idea imho, as imho "NT" should generally be interpreted as "Not-There!".
 
SirDice said:
[snip] Oddly enough the one filesystem that has good support by pretty much every OS is Microsoft's FAT16/FAT32. [snip]
Click to expand...

I oddly forgot about FAT32. My other 1TB USB drive is FAT32 and it works just fine. This new drive came with FAT32 as the default. The 4GB file size limitation might become a problem someday.

kpedersen said:
[snip] For large files, I always have best luck with NTFS (via fuse on FreeBSD). [snip]
Click to expand...

I had an NTFS filesystem go wonky after writing a couple hundred gigabytes of data from an OpenBSD machine. The data was ultimately unrecoverable.

Snurg said:
[snip] I use ext2fs for that since years because it keeps the basic unix permission data and is supported by all OSes (even Windows, see here). [snip]
Click to expand...

Will FAT32 distort the permissions, timestamps, or other file metadata?

I had a similar decision to make (which FS for USB drive) earlier this year but I somehow forgot about the issues and the conclusions. The environment was a bit different - OpenBSD was in the mix. It's all posted here. In summary, there were significant problems with EXT2.

SirDice said:
[snip] In my view we really need a good portable filesystem (preferably BSD licensed) for external drives.
Click to expand...

Oh, what a brave new world to have such software in it! ;)
 
hanzer said:
Will FAT32 distort the permissions, timestamps, or other file metadata?
Click to expand...
It will. FAT32 doesn't have ownership for files/directories for example. But you could just tar(1) your files and save the resulting archive. That would preserve the meta-data.
 
Juha Nurmela said:
Do you have to choose just 1? How about partitions, maybe one for direct tarring past any fs ?
Click to expand...

Hey, that's an interesting idea. I suppose it could be worthwhile to experiment-with/assess the various options then set up multiple partitions, each configured and tuned for a particular workload. But for now, other projects have much higher priority and there is the possibility that this drive might someday need to be shipped-to/shared-with others. Maximizing plug-and-play compatibility is important. As it is, both CentOS and Ubuntu recognize and automount a FAT32 drive in a very straightforward and usable way. FreeBSD is a bit finicky, requiring mount_msdosfs [B][I]-o large[/I][/B] /dev/da0s1 /mnt but it hasn't shit the bed or corrupted the filesystem, yet.

Are there any suggestions for FreeBSD mount_msdosfs options to improve interoperability and/or stability?
 
You know emulators/mtools ?

Most of the time I just bring in stuff (pictures from a camera, and not much else). So I just
stick the cam into a port, type camg (in-house) and while making the coffee, any changed files get copied into /scratch/pics and stay there forever. I've never thought of making it bidirectional. There must be existing solutions like that...

Juha
 
kpedersen said:
For one that could be perfect but I have not yet used for hard disks before is UDF (https://en.wikipedia.org/wiki/Universal_Disk_Format). Apparently it can be used as a standard filesystem and has some pretty large limits. I don't know if FreeBSD supports this or has the required tools to use yet though. Again, I have only used this on DVDs but apparently it has the potential to be used for so much more.
Click to expand...

Out of casual curiosity, I've started looking into UDF. There seems to be some enthusiasm for it (in conversations like this thread), but the only software I could find is udfclient and linux-udf, available on *BSD and Linux, respectively. Both projects look a bit shaky and/or abandoned.

Putting aside UDF on a hard drive for the moment, how is it used on a DVD-R, what are the advantages, and how portable/accessible are disks with this filesystem?

For example, I found this blurb:

UDFclient basicly provides a userinterface not unlike ftp(1) but differs in that it allways fetches or writes files recursively from the disc(s). As from version 0.4.9 it supports the basic get, mget, put, mput, mv, rm, mkdir, cd, lcd, free, pwd, lls command set. get and put can now also retrieve or store using different names than the origional. ​

Is this what you've used with UDF DVDs? I'm not sure that I see any advantage to accessing data this way. Can you walk through a scenario or two where UDF on a DVD has some value?
 
FAT32 seems more compatible to me after going through similar exercises with iSCSI storage filesystems.
NTFS has alternative data streams (often used maliciously) and you need fuse to use it.
FAT32 on large drives is not supported by Windows though(IRONIC). FreeBSD/Linux has no problems with large FAT32 volumes. If your dealing with under 4GB files it works well. Not sure what the directory limitations are.
 
SirDice said:
FAT32 doesn't have ownership for files/directories for example
Click to expand...
I never thought about this problem.

So what would happen if you copied FreeBSD files with permissions to FAT32 and then back? Would all permissions be reset or set to a minimal level?
 
forquare said:
Does both Ubuntu and CentOS support ZFS? [snip]
Click to expand...

I found this -> http://zfsonlinux.org/ <- and that looks like a Yes. But I don't know how stable it is or how compatible it might be with FreeBSD's version of ZFS (what version of ZFS does FreeBSD-10.3 include?).

forquare said:
[snip] If they do is there any reason to not use ZFS on such a drive?
Click to expand...
  1. My FreeBSD machine is 32-bit, single core with 1GB RAM, and it needs to do more than just run ZFS.
  2. The Linux machines are also fairly wimpy.
  3. In my case, this drive needs reasonable plug-and-play portability.
 
ZFS is a major overkill for an external drive intended for just transferring data from one system to another. It's a very powerful filesystem when used right but has some idiosyncrasies such as the definition of a pool that relies solely on on-disk labels and not on pre-determined markers such as partitions that you can use to identify where the filesystem is located exactly and where not. ZFS is always suspect to an error condition where you plug in a disk from another system to your system and you get pool name conflicts or the attached disk contains a pool with mountpoints that overlap with your system. You don't want anything like that with a filesystem you use for just a mundane task of transferring data from one place to another.
 
Given my choice, I'd like a good FreeBSD-specific UFS driver for Linux. That is not likely to happen, so I recommend instead of a plain external drive, a netbook or the like running NAS software, NFS (or Samba if absolutely necessary) and a network connection. That avoids reverse-engineered or poorly-supported drivers writing to fragile filesystems.
 
wblock@ said:
[snip] so I recommend instead of a plain external drive, a netbook or the like running NAS software, NFS (or Samba if absolutely necessary) and a network connection. That avoids reverse-engineered or poorly-supported drivers writing to fragile filesystems.
Click to expand...

That's a lot of extra layers of software and hardware to get around the problem. Since the poverty is in the portable file-system software arena, it seems like that would be the ideal place to make an investment towards a solution.

However, given the current economic situation: hardware is cheap, existing FOSS software is free, engineering something new is extremely expensive, relatively; the Rube Goldberg approach might be sadly pragmatic ;)
 
There is nothing preventing a port of FreeBSD's UFS to Linux. Or even just fixing what already exists. There are even UFS drivers for Windows. A good first step would be an array of tests to throw at such things, which could also be used to find bugs and regressions on FreeBSD's UFS.
 
Another way to go about this is to run a VM set up to talk directly to the target disk. Essentially, the VM is a filesystem driver.
 
hanzer said:
That's a lot of extra layers of software and hardware to get around the problem. Since the poverty is in the portable file-system software arena, it seems like that would be the ideal place to make an investment towards a solution.

However, given the current economic situation: hardware is cheap, existing FOSS software is free, engineering something new is extremely expensive, relatively; the Rube Goldberg approach might be sadly pragmatic ;)
Click to expand...

I've been looking for a BFS (Be, not boot) solution to link Linux, FreeBSD, and Haiku together seamlessly. I don't want a fuse-oriented solution (tho that could work with lower performance). I think the licensing of OpenBFS is MIT.

Linux has a read-only driver for BFS IIRC.
 
wblock@ said:
There is nothing preventing a port of FreeBSD's UFS to Linux. Or even just fixing what already exists. There are even UFS drivers for Windows. A good first step would be an array of tests to throw at such things, which could also be used to find bugs and regressions on FreeBSD's UFS.
Click to expand...

It may seem subtle but there are significant constraints preventing a port of FreeBSD's UFS to Linux from happening: people with the will & skill[1] to execute such a thing need sufficient and secure life support during the project, and the type of work should line up with their current career tack and goals.

[1]: A formal, systematic education of Computer Science and Software Engineering. Provincial experience of folk programmer craft probably isn't sufficient for something like cross-platform file-system software development. This, of course, rules out someone like me ;)
 
Cross-platform stuff is difficult. Note the number of platforms and the how few inter-platform filesystems we have found.
 
hanzer said:
That's a lot of extra layers of software and hardware to get around the problem. Since the poverty is in the portable file-system software arena, it seems like that would be the ideal place to make an investment towards a solution.
Click to expand...

wblock@ said:
Cross-platform stuff is difficult. Note the number of platforms and the how few inter-platform filesystems we have found.
Click to expand...

Of those very few cross-platform filesystems, two are NFS and CIFS. wblock@'s earlier recommendation might require another machine and another OS, but it's going to be the least painless in the long run. The whole reason I built an NAS was to no longer have to worry about whether files I need could be quickly, safely, and easily moved between devices.

You might think that adding another, low-powered machine with some extra configured daemons is too complicated, but is it really? The alternative is to cobble together a bunch of work-around programs on each system you're moving data to and from, and learn whole new sets of commands for each system. Once you've properly configured an NAS you don't have to touch it anymore.
 
wblock@ said:
Another way to go about this is to run a VM set up to talk directly to the target disk. Essentially, the VM is a filesystem driver.
Click to expand...

I actually like this solution. I already use a VM as my "proprietary software compatability layer" ;)

I guess for a USB drive you will need to use the slightly heavier VirtualBox rather than Qemu because of the requirement of usb passthrough.
 
You must log in or register to reply here.
Back
Top