Other Format for a second disc

Spity

Member

Reaction score: 6
Messages: 48

I was planning to use a second hard disk as a backup for documents and files. I'm not sure what format to use ext4 / ntfs / ufs to be able to access it from FreeBSD and Win10 since I have dual boot. When I was using Linux I used a program from Windows to access ext4., But with ufs I don't know ... I would like to avoid problem of corrupted files, etc. When editing a document... Any ideas or suggestions?
 

jmos

Well-Known Member

Reaction score: 262
Messages: 331

Doing backups on a internal drive doesn't make sense. F.e. a simple hardware error of the disk controller resp. motherboard can shredder both disks data in one go.

So it has to be external. As you want access from different installations one of the most convenient solutions is to use a cheap NAS: As long you're using network file transfers like SSHFS, NFS, SFTP, FTP or Samba (the last I would choose) you're independent from the file system - that belongs to the NAS.

Costs initial more, but in the long run you've got more options. I also recommend not to go with the low budget nonames. And SSH access is something I would insist of.

Disadvantage for many users is the speed, as it's limited to their WLan. But if you're using ethernet… A small self made home server gives you the most options as you have full control over the servers OS (on a purchased NAS you have to deal with the manufactures tinker).
 

grahamperrin

Beastie's Twin

Reaction score: 1,695
Messages: 5,073

NTFS, I guess.

sysutils/linuxfdisk

Code:
% pkg info --list linuxfdisk | grep bin | sort
        /usr/local/sbin/cfdisk-linux
        /usr/local/sbin/fdisk-linux
        /usr/local/sbin/sfdisk-linux
%

sysutils/gdisk

Code:
% pkg info --list gdisk | grep bin | sort
        /usr/local/sbin/fixparts
        /usr/local/sbin/gdisk
        /usr/local/sbin/sgdisk
%

sysutils/fusefs-ntfs

Code:
% pkg info --list fusefs-ntfs | grep bin | sort
        /usr/local/bin/lowntfs-3g
/usr/local/bin/ntfs-3g
/usr/local/bin/ntfs-3g.probe
/usr/local/bin/ntfscat
/usr/local/bin/ntfscluster
/usr/local/bin/ntfscmp
/usr/local/bin/ntfsfix
/usr/local/bin/ntfsinfo
/usr/local/bin/ntfsls
/usr/local/sbin/mkntfs
/usr/local/sbin/ntfsclone
/usr/local/sbin/ntfscp
/usr/local/sbin/ntfslabel
/usr/local/sbin/ntfsrecover
/usr/local/sbin/ntfsresize
/usr/local/sbin/ntfsundelete
%

Experimenting here with a spare USB flash drive, after using cfdisk-linux

Code:
root@mowa219-gjp4-8570p:~ # gdisk /dev/da4
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************


Command (? for help): p
Disk /dev/da4: 30375936 sectors, 14.5 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): B7E19521-A2BE-414C-87F9-812E36265B4C
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 30375902
Partitions will be aligned on 1-sector boundaries
Total free space is 13082 sectors (6.4 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              63        30362849   14.5 GiB    0700  Microsoft basic data

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/da4.
Warning: The kernel may continue to use old or deleted partitions.
You should reboot or remove the drive.
The operation has completed successfully.
root@mowa219-gjp4-8570p:~ # mkntfs --label NTFS /dev/da4p1
The partition start sector was not specified for /dev/da4p1 and it could not be obtained automatically.  It has been set to 0.
The number of sectors per track was not specified for /dev/da4p1 and it could not be obtained automatically.  It has been set to 0.
The number of heads was not specified for /dev/da4p1 and it could not be obtained automatically.  It has been set to 0.
Cluster size has been automatically set to 4096 bytes.
To boot from a device, Windows needs the 'partition start sector', the 'sectors per track' and the 'number of heads' to be set.
Windows will not be able to boot from this device.
Initializing device with zeroes:   9%
 

PMc

Daemon

Reaction score: 747
Messages: 1,494

Doing backups on a internal drive doesn't make sense. F.e. a simple hardware error of the disk controller resp. motherboard can shredder both disks data in one go.
I wouldn't say that so absolutely. These are quite rare events. There is another risk a lot more likely which You didn't mention: lightning strike/power spike (might even result in fire). And then a machine-external backup that is still plugged in the same building is not helpful..
So, internal may well make sense if it is not the only scheme. Vital data might be copied to the cloud, while a local (and probably internal) backup would cover the by far most common faults in a quick fashion: operator mistake, damaged zpool, damaged disk.
 

grahamperrin

Beastie's Twin

Reaction score: 1,695
Messages: 5,073


Initializing device with zeroes: 9%

Outcome:

Code:
Initializing device with zeroes:  99%Failed to complete writing to /dev/da4p1 after three retries.
This should not happen.
root@mowa219-gjp4-8570p:~ #

I'll retry with MBR, without converting to GPT.

If failure recurs I'll write something other than NTFS then run stressdisk.


– maybe because a partition start sector was not specified.

Postscripts

A subsequent run of mkntfs(8) with --partition-start 63 failed in the same way.

I gave the drive to UFS then ran stressdisk, without error, for more than two hours.

Now there's an unreasonable amount of read activity for an unreasonably long time during or after mkntfs creating the backup boot sector:

1625937980425.png

Found: FreeBSD bug 206978 – sysutils/fusefs-ntfs: Disable UBLIO as it breaks mkntfs
 
Last edited:
OP
Spity

Spity

Member

Reaction score: 6
Messages: 48

I understand everything, but only one answered my question. For a hard drive backup, which file system is "best" ? ext4 ntfs or ufs. The problem is that I don't know how to see UFS from windows
 

grahamperrin

Beastie's Twin

Reaction score: 1,695
Messages: 5,073

Alain De Vos

Son of Beastie

Reaction score: 996
Messages: 3,076

Is the visibility from windows a requirement ?
If so you could also use the simple FAT32 format.
If visibility from windows is not needed i would personally use a zfs zpool.
 

DanDare

Member

Reaction score: 32
Messages: 61

I understand everything, but only one answered my question. For a hard drive backup, which file system is "best" ? ext4 ntfs or ufs. The problem is that I don't know how to see UFS from windows

It used to be very unreliable years ago. Today, fuse/ntfs-3g is said to be stable and reliable and it contains the most important features (like NTFS compression support). It can be a pain sharing NTFS because it's permission schemes are different/incompatible in other OS's. For example files/folders can be inaccessible from FreeBSD/linux if original file permissions are too restrictive under Windows.
But I don't use share NTFS in a daily basis so I'm not the best person to make a recommendation.

Suggestions:
1- Consider ext32 if you think it's features (and the partition size) are coherent with your use case.
2- Test NTFS for a while if you can, make your tests. It will give you preliminary conclusions.
 

grahamperrin

Beastie's Twin

Reaction score: 1,695
Messages: 5,073

… I assume that NTFS will be better suited to repair, if a file system inconsistency occurs.

… 2- Test NTFS for a while if you can, make your tests. It will give you preliminary conclusions.

Hint

When Windows says:

You don't need to scan this drive

We haven't found any errors on this drive. …

– do not believe that the file system is truly free from errors.

1625939905162.png

Do routinely perform scans.

Yesterday, for example, runs of CHKDSK revealed failure at stage 1 (basic file system structure) for a C: drive that was supposedly error-free.
 

DanDare

Member

Reaction score: 32
Messages: 61

Hint

When Windows says:



– do not believe that the file system is truly free from errors.

View attachment 10550

Do routinely perform scans.

Yesterday, for example, runs of CHKDSK revealed failure at stage 1 (basic file system structure) for a C: drive that was supposedly error-free.
That's true! I've hit false negatives like this a couple of times too.
 

grahamperrin

Beastie's Twin

Reaction score: 1,695
Messages: 5,073

… even better /s is when you schedule a drive repair at Windows startup time, then Windows suffers a silent unexpected restart as a result of a problem with the drive.

Blink, and you'll miss it; seeing Windows started, there's a logical assumption that checks and repairs are complete when truly, there was not even a check.

That's not to detract from the qualities of NTFS, just something to be aware of with Windows.
 
Last edited:

ralphbsz

Son of Beastie

Reaction score: 2,695
Messages: 3,557

I understand everything, but only one answered my question. For a hard drive backup, which file system is "best" ? ext4 ntfs or ufs. The problem is that I don't know how to see UFS from windows
The reason I haven't answered your question is that I don't know a good answer.

What you're trying to do is to have a single file system (ext4, ntfs, ufs, ...) that is accessed from two different operating systems, one at a time. The problem with that is that file systems are deeply integrated into the OS they are developed for: Ext4 is native on Linux, NTFS on Windows, and UFS on the BSDs. For using it on a different OS, you'll have to use a "compatible" file system implementation, which is always a second class citizen, not written by the original developers of the file system, and in the case of NTFS written without full access to documentation and a sample implementation. Modern file systems are heinously complex, and have very interesting internal metadata structures, which are hard to keep correct. Or to put it bluntly: those alternate implementations are junk, and likely to cause corruption and data loss.

Here are two versions my advice: If you can, use that second disk and file system primarily from one OS, with the native file system implementation in read-write mode, and the alternate one in read-only mode. So for example, if you decide that you use Windows more often, make the backup disk NTFS, and from Linux and BSD only read it, and be carefully to check that what you read is correct (like manually verify sizes and checksums). Even better: GIve the second disk its own computer, which runs continuously. You could for example get a very small, inexpensive and power-efficient server (Raspberry Pi?), and attach the backup disk (formatted as Ext4 to match the Pi's Raspbian OS) to the network, and mount it via NFS or Samba. Or you could keep the disk in your main computer, but whatever primary OS you are running, also set up a VM with a small OS kernel that is used only as a disk server, and again exports the disk via a network protocol.

But in reality, we're discussing an XY problem here. Let's analyze your basic problem: You want reliable backup, and you want that backup to be accessible from multiple OSes. Why are you trying to solve that problem yourself? Many other people have long solved it, and they are certainly more of them than you alone. And without undue criticism: they are probably much more experienced in storage than you are. Simply move your backup to a cloud provider. There are many potential solutions, such as Amazon, Backblaze, Dropbox, Google, and smaller one. Any of those solutions is going to be easier and more reliable than something that you cook up yourself.
 

bsduck

Well-Known Member

Reaction score: 435
Messages: 442

Using ext4 doesn't make sense, it's foreign to both operating systems involved.

I don't dual boot, neither run Windows, but if I had to choose a filesystem for a shared disk, I would go for NTFS, because the NTFS implementation for FreeBSD is for sure more widely used, and therefore better tested, than the UFS implementation for Windows.

Make sure to read ntfs-3g(8) before using, it has important information like this one:
Windows hibernation and fast restarting

On computers which can be dual-booted into Windows or Linux, Windows
has to be fully shut down before booting into Linux, otherwise the NTFS
file systems on internal disks may be left in an inconsistent state and
changes made by Linux may be ignored by Windows.

So, Windows may not be left in hibernation when starting Linux, in
order to avoid inconsistencies. Moreover, the fast restart feature
available on recent Windows systems has to be disabled. This can be
achieved by issuing as an Administrator the Windows command which
disables both hibernation and fast restarting :

powercfg /h off
(it says Linux but the same of course applies with FreeBSD)
 

ralphbsz

Son of Beastie

Reaction score: 2,695
Messages: 3,557

... I would go for NTFS, because the NTFS implementation for FreeBSD is for sure more widely used, and therefore better tested, than the UFS implementation for Windows.
Good argument.

Counter-argument: The people who wrote UFS for Windows had access to the UFS-for-FreeBSD source code, and if they had questions, they would probably get answers (on mailing lists, or from the original developers themselves. The people who wrote NTFS for Linux/BSD did not have access to the real NTFS source code, and unless they work for Microsoft or a few specially privileged employees of other companies have no way to have questions answered.

I have no idea where the balance of arguments ends up; but for my data, I would not run "foreign" file systems.

Make sure to read ntfs-3g(8) before using, it has important information like this one:
That's just one example of all the pitfalls. The sheer number of pitfalls tell me that this should be stayed away from.
 

mark_j

Daemon

Reaction score: 870
Messages: 1,449

I was planning to use a second hard disk as a backup for documents and files. I'm not sure what format to use ext4 / ntfs / ufs to be able to access it from FreeBSD and Win10 since I have dual boot. When I was using Linux I used a program from Windows to access ext4., But with ufs I don't know ... I would like to avoid problem of corrupted files, etc. When editing a document... Any ideas or suggestions?
Perhaps a Linux file system? My reasoning: doesn't win10 have that WSL? Can you access "external" disks with that?
ZFS will surely be an option in the future, but that's not much help for you now.
 

grahamperrin

Beastie's Twin

Reaction score: 1,695
Messages: 5,073


… The people who wrote UFS for Windows had access to the UFS-for-FreeBSD source code, and if they had questions, they would probably get answers (on mailing lists, or from the original developers themselves. …

True, however the last update to ufs2tools (for Windows) was more than fifteen years ago. Partly thanks to funding by Netflix, the FreeBSD source code relating to UFS has changed significantly since then.
 

a6h

Daemon

Reaction score: 1,610
Messages: 1,429

FAT32
If you have UPS, use FAT32L. Otherwise use NTFS. Sudden umount/eject causes unrecoverable file corruption in FAT32, both in Windows and FreeBSD, esp. when files are open.

NTFS
Problem with NTFS (beside fuse) -- a scenario:
1. mount NTFS -> use -> umount NTFS in FreeBSD.
2. Run Windows and perform CHKDSK on NTFS
3. Windows often complains that there's some problem/corruption/error.

=> Windows doesn't like they way fuse-fs handles NTFS.
## False positive? I don't know!

How I deal with shared HDD:

Internal HDD1 (4 MBR)
2x NTFS Windows + 1x 0xA6/FFS2 OpenBSD + 1x 0xA5/UFS2 FreeBSD

External HDD2 (backup)
* 1 UFS2
* 1 NTFS (formatted on Windows)
* FreeBSD and OpenBSD UFS and MBR implementations are not compatible, so I only use FreeBSD as an example.

Procedure:
1. Mount NTFS on FreeBSD, copy files from UFS2 to NTFS, go back to Windows and CHKDSK.
2. I only use UFS2 backup on FreeBSD and NTFS backup on Windows.
3. When I have to change/add/remove files in/to/from backup: GOTO STEP#1

Footnote
* When it comes to Windows (one shared drive, no server) then extX, ZFS, UFS/FFS... all are irrelevant. You have only two options: FAT32 or NTFS
* With default Windows tools you can use partition FAT32, up to 32GB and 256K cluster (max). on FreeBSD you have a limit of 32K -S sector-size. Keep that in mind when you format FAT32 in Windows and want to change default "Allocation Unit Size"
* For FAT32, there're 3rd-party tools to overcome Windows 32GB FAT32 limit. I don't trust them!
 

grahamperrin

Beastie's Twin

Reaction score: 1,695
Messages: 5,073

1. mount NTFS -> use -> umount NTFS in FreeBSD.
2. Run Windows and perform CHKDSK on NTFS
3. Windows often complains that there's some problem/corruption/error.

Are there sub-steps?

1.1 wait
1.2 physically detach from the FreeBSD computer

If so, how long do you wait at step 1.1?
 

grahamperrin

Beastie's Twin

Reaction score: 1,695
Messages: 5,073

Thanks.

I wondered whether there might be non-flushed data (and so, no ultimate mark of file system cleanliness) where a drive is physically detached immediately after umount(8) but from what I can tell, there's not the risk. At least, not at the FreeBSD level:

 

DanDare

Member

Reaction score: 32
Messages: 61

This is not very related to original question but can be a very good alternative IMO:
I use TrueCrypt in Windows what makes truecrypt volumes transparently accessible from anywhere able to mount truecrypt volumes.

Tried building truecrypt from quarterly but it's broken. I didn't test it but Veracrypt (truecrypt fork) is available in FreeBSD.
I use truecrypt for years and never had any problems.
 

mark_j

Daemon

Reaction score: 870
Messages: 1,449

Yes, Vera/TrueCrypt is a bit of lateral thinking.
I use Veracrypt on all platforms, it works seamlessly. (Though I've not experimented on accessing it from one OS to another on the same machine).
 
Top