Other writing data to NTFS partition in Unix and Linux to share with others can be destructive and risky.

R

rupeshforu3

Hi I am Rupesh from India and I brought a new desktop pc with ryzen 5500GT cpu and Asus prime b450 motherboard. I installed Linux freebsd along with windows 11 and everything is working fine except I can't copy files properly to NTFS partition and share those files with others or other computers.

I love using unix and Linux operating systems and I have the habit of sharing files present in Unix and Linux operating systems with other people and even I use those files outside Linux operating systems I mean I use them in windows 11 operating system.

In unix operating system I have the habit of downloading youtube videos and compress them to av1 codec and finally move the compressed files to NTFS partition or file system for backup or to view those videos in windows. I have the habit of downloading mp3 music files from websites and transfer them to NTFS partition or file system.

Suppose I have 1000 files which I want to transfer them to NTFS partition or file system and moved them to NTFS partition using Nautilus file manager and after booting windows 11 and try to open the moved folder I am getting error as folder or directory inaccessible. Now inorder to overcome this problem I may issue the following command

chkdsk d: /f /x

This command checks for file system errors and after that when I try to open the moved folder I am able to access the particular folder but unfortunately I am able to see only 800 files and so the remaining 200 files are lost.

The second issue is suppose now I have downloaded 1000 files and transferred those files to NTFS partition and after two to three months I have decided to install my present Linux operating system from the beginning and I have installed it from the beginning and after that I have mounted the same NTFS partition in file manager and succeeded and I am able to write new files but if I want to move the previously copied or moved 1000 files from NTFS partition to my home folder I can't.

If I continuously use the same Linux or Unix operating system for years without installing freshly from the beginning there's no problem but if I install it freshly then I can't move files from NTFS to my home folder.

Today I have installed freebsd operating system and in Linux operating system I have copied 100 files to NTFS partition and after booting into freebsd I can't move those particular 100 files from NTFS partition to my home folder.

I can't move means when I open any file manager and right click on the particular folder which is problematic I can't see option cut.

Copying or moving data between Linux or Unix operating systems of other computers is easy through Linux native partitions like ufs2 xfs or ext4. And even you or me can't delete a file or folder which was copied from other computers and inorder to delete it I must open a terminal emulator and gain super user permissions and navigate to the directory and finally issue the following commands to delete them permanently

rm -rf
rmdir

I think that lots of security information is added to the copied or moved files and directories when they are copied to mounted partitions.

Previously I lost 30 gb data by moving files from Linux local xfs file system to NTFS partition. Today I lost 2 gb data by moving data from freebsd ufs2 file system to NTFS partition.

I know that you are getting annoyed through my explanation and so I will illustrate what's going on as below

Today we can create a zip file through compressing a folder and add security to it by adding password to it and you have distributed to other people without providing password and finally they can't access it.

The same thing is happening in the present situation also I have some files in my unix system and I have copied them to NTFS partition or file system but unfortunately the operating system copied files along with user credentials like username or password uid etc.,.

I or other people have tried to mount the same NTFS partition in another unix system but can't move or delete those files because username or password uid are invalid.

Another issue is suppose in my NTFS partition or file system I have 1000 files and transferred those files to my unix or Linux system and opened a terminal emulator and after that I have navigated to the concerned directory on which transferred files are present and issued list command I am getting unknown ASCII characters in the terminal emulator which are not part of the file names.

Many people and all of you mostly work in Unix or Linux systems only but don't try to transfer files to and from NTFS partitions but I frequently use both systems. So you have not noticed any such errors.

You can suggest to turn off hibernation in windows 11 and I have already done even no use.

Please try to rectify this particular problem ie., moving files to NTFS partition without any security information. If you rectify this issue many people will be benefited.

Regards,
Rupesh.
 
rupeshforu3 said:
I installed Linux freebsd along with windows 11
Click to expand...
FreeBSD is NOT a Linux. It's Linux or FreeBSD.

rupeshforu3 said:
but unfortunately the operating system copied files along with user credentials like username or password uid etc.,.
Click to expand...
It doesn't work that way. While the UserID (or SID in Windows) does get stored along with the file (ownership) it's not the filesystem that enforces them, it's the OS that does the actual enforcing.

rupeshforu3 said:
I or other people have tried to mount the same NTFS partition in another unix system but can't move or delete those files because username or password uid are invalid.
Click to expand...
Has nothing to do with it. Just mount it with the proper permissions.
rupeshforu3 said:
I have navigated to the concerned directory on which transferred files are present and issued list command I am getting unknown ASCII characters in the terminal emulator which are not part of the file names.
Click to expand...
Sounds like a localization issue.

rupeshforu3 said:
Many people and all of you mostly work in Unix or Linux systems only
Click to expand...
Most of use actually use and/or maintain a lot of different systems. Or at least have to communicate/exchange information with a variety of systems, including Windows.

rupeshforu3 said:
Please try to rectify this particular problem ie., moving files to NTFS partition without any security information.
Click to expand...
Direct your grievances to the developers of NTFS-3G and/or Microsoft, it's not part of FreeBSD. It's a reverse engineered filesystem driver.
 
Windows's "fast startup" is going to cause basically all your problems. Seriously consider making a separate partition and either format it with FAT32 (which has a built-in driver for FreeBSD), or exFAT and mount with sysutils/fusefs-exfat (exFAT has no file size limit, and might be greatly beneficial over FAT32).
 
rupeshforu3 said:
Previously I lost 30 gb data by moving files from Linux local xfs file system to NTFS partition. Today I lost 2 gb data by moving data from freebsd ufs2 file system to NTFS partition.
Click to expand...
NTFS is complex file system and using 3rd party drivers is not good idea. To move files from Unix/Linux to Windows, use exFAT partitions or FAT32 when possible. Тhe question is why you did not learn your lesson from the first case? And why you move and not copy? Copy, verify destination files, then delete originals and will not lose data, only time. PS: Is there undelete tool for XFS?
 
Instead of dual-booting Windows 11 and FreeBSD on a single machine, try these
  • use Windows Subsystem for Linux (WSL) to run Linux inside Windows 11, which will allow you to access files on each side using \\wsl.localhost and /mnt/c
  • use VirtualBox on Windows 11 to run FreeBSD (or anything else) inside a VM, which will allow you to access files over the virtual network using Samba or WinSCP
 
FWIW, I tried to write to NTFS from Void Linux using the kernel driver, and the result was a disaster. I was using an NTFS volume for my Steam library, and there were so many problems I was figuring Steam on Linux just plain didn't work. I moved my library to an ext4 filesystem on a lark, and all the problems disappeared.

I'll echo what others have said on this thread: don't use NTFS from anything other than Windows.
 
I’m writing several TiB per week to NTFS formated portable hard drives using mostly Linux, but also FreeBSD, sometimes Windows 10/11. Those hard drives are exclusively accessed by Windows 7,8,10, and 11. Done this for more years then I remember.

No issue at all.

My recommendation at OP would be to reconsider his workflow.
 
How you format your NTFS portable drives? I guess it is not from Windows or it is from older version, not 10/11. Anyway, don't see reason to use NTFS instead of exFAT for portable drive. You need EFS or access rights to folders?
 
As you suggested to use exfat partition while sharing data between various operating systems and so I have created exfat partition and tried to share data between them using exfat.

At present everything thing is working fine and no user credentials are copied along with files or folders.

I can't understand what is the need of copying user credentials along with files or folders when NTFS partition is used as destination.

Again I am not going to use NTFS partition along with BSD or Linux because it is problematic.

If you want proofs of what I said previously I am ready to provide screenshots of this issue.
 
rupeshforu3 said:
no user credentials are copied along with files or folders.
Click to expand...
Credentials are never copied along with files or directories, with any filesystem. Please try to understand what file ownership, permissions and access control lists (ACL) mean and how they work. Because you seem to have some really weird misconceptions on how filesystems work.
 
SirDice said:
Credentials are never copied along with files or directories, with any filesystem. Please try to understand what file ownership, permissions and access control lists (ACL) mean and how they work. Because you seem to have some really weird misconceptions on how filesystems work.
Click to expand...

Most likely what was meant was the ownership and modes being copied. NTFS-3G by default tries to make everything as openly-accessible as possible, but it does have a utility to generate Windows-to-Unix user mappings and shove things into the Unix world that will map back to Windows's native user database and permissions.

Linux also has a new built-in kernel driver that uses the NTFS POSIX namespace to support full Unix semantics at all times on NTFS, and does not try to interoperate with Windows's native user database. If OP is actually using Linux and used this driver, it could spell trouble for interoperability.

All said and done, however, exFAT is just a stupidly simple file system without support for ownership or modes (other than the write bit). It's almost certainly what is actually wanted to share files between multiple operating systems.
 
chungy said:
Most likely what was meant was the ownership and modes being copied.
Click to expand...
That did cross my mind, that's why I mentioned those specifically. But it's not the filesystem that enforces those restrictions, a filesystem only records them.

NTFS on a USB stick, external drive, or other exchangeable media is even problematic if you need to share files between two or more windows machines.
 
I have several FreeBSD machines now and one Windows machine. I transfer files back and forth with rsync (under WSL on the Windows box), because I haven't felt like dealing with the hassle of getting different operating systems to share the same disk. It seems like rsync might work especially well for this use case, downloading and compressing a bunch of files, then moving them in bulk to the Windows machine.
 
Two general comments.

First: With the exception of very simple filesystems (such as FAT), and file systems explicitly designed to be used from multiple OSes (such as ZFS), avoid using non-native file system implementations. As examples: When on Windows, don't use UFS or ext2, when on Linux, don't use NTFS or UFS, and when on FreeBSD, don't use ext2 or NTFS. If you have to use them, then do so read-only, and don't modify the on-disk file system. Why? Because those non-native file systems are usually implemented by studying the source code (where available) and the on-disk data structures, and are usually coded by people who are not experts (the designers) of the native file system, leading to misinterpretation of data structures. When only reading, this can cause crashes and wrong results; when writing, it can corrupt data. If you have to share data between different OSes, either use a natively shared file system (such as ZFS between Linux and FreeBSD), or a very simple file system (such as FAT, perhaps enhanced by moving tar balls), or an outboard cluster file system server.

Second: When sharing disks on the same machine between different OSes, be super careful with caching. For example, if you have a shared disk or partition between Linux and Windows on a machine, and write to the shared partition, and then IMMEDIATELY hibernate/suspend the OS and start the other one from hibernation/suspend, then the other OS may not know that the first OS is still holding modified data in its buffer cache. If you have to operate this way, make sure the data has been flushed to disk, for example by shutting down both OSes completely, or unmounting/remounting the shared disk or partition.
 
6502 said:
How you format your NTFS portable drives? I guess it is not from Windows or it is from older version, not 10/11. Anyway, don't see reason to use NTFS instead of exFAT for portable drive. You need EFS or access rights to folders?
Click to expand...
Yes, I do for every single drive:


mkfs.ntfs -q -f /dev/sdc1


Nope, NTFS filesystems is a mandatory requirement, not my choice.
 
After reading all the posts here I’m quite puzzled about the conception Linux or FreeBSD aren’t able to write NTFS in a stable capacity.

My workflow is rather simple here in particular using Linux:

01. Create MD5 hash for all the files to be copied to portable USB drive.

02. Format the portable drive using Linux.

mkfs.ntfs -q -f /dev/sdc1

03. Mount drive.

04. Copy files to hard drive using:
cp -a src /media/usbdisk1/.

05. Unmount the drive to force all data to be written to disk.

06. Mount the drive read only.

07. Check MD5 checksums on the drive. If verification has been successful unmount and be done.

The hard drives are usually accessed from Windows within a week and data will be completely moved of the drive for further access.

I encouter on average two drives per year which report IO error along the process, which are then marked as defect and will be replaced by a new one. The defect hard drives will be destroyed (mechanical shredded).
 
You must log in or register to reply here.
Back
Top