While OpenZFS right now supports 1023 byte long file names, this feature cannot be utilized on FreeBSD because of FreeBSD's old-fashioned file name length limitations.
So anyone who would want to transfer large amounts of data from NTFS to ZFS on FreeBSD could face many files not being copied with cp/scp/rsync.
This guide provides commands to create a mapfile with which it would be possible to transfer files with long file names from NTFS to OpenZFS hosted on FreeBSD (this can be helpful for copying to many other Unix-like systems including Linux).
The mapfile will match currently existing files with long filenames and their new destinations along with shortened file names for copying, so file names will be shortened.
Only the minimal necessary shortening is done (to 255 bytes) which helps with avoiding duplicates and preserves accessibility of as much information as possible.
Unfortunately, only shortening the file names would allow full access to them while using FreeBSD even though OpenZFS right now supports longer file names.
An alternative is archiving, but you would not be able to extract an archive with long filenames while using FreeBSD.
If you are committed enough, you might want to recompile FreeBSD kernel to allow longer file names, but this was not tested and is not a part of this guide.
The described approach allows to copy files back to NTFS with restoring their full file names if needed and creates matching lists of short and long filenames for use and long file name preservation.
A Linux machine functions as an intermediary here and all commands are for Linux. They can probably be adapted to FreeBSD. FreeBSD-hosted OpenZFS dataset is mounted with a help of sshfs.
These commands might not be the most optimized, but they have been tested by me to solve my problem of transfering files with long file names from NTFS to OpenZFS hosted on FreeBSD.
Always recheck the commands and results in your case.
Original mountpoint:
/media/NTFS/
New mountpoint:
/media/FreeBSDZFS/
Folder with listing files:
/home/user/listingfiles/
Replace the mentioned 3 directories with those you would be using.
1. Create a list of files:
2. Create a list of files without paths:
3. Create a list of calculated file name byte lengths:
4. Create a list of long (longer than FreeBSD allows) and short (those which FreeBSD allows) file names based on length:
5. Create a list of files with 1 or 0 in front of every file path to remove the short names with grep in the next command:
6. Create a list of files with only long file names:
7. Create a list of files with only long file names without paths:
8. Create a list of long filenames' byte counts:
9. A prophylactic command, as the next command will not replace a listing file but will be adding data incrementally to it:
10. Create a list of bytes to remove to shorten the long file names to 255 bytes:
11. Combine lists of long filenames and bytes to remove for the next command:
12. Create a list of files with last bytes conflicing with FreeBSD removed while preserving only UTF-8 characters since UTF-16 characters cut in half may be unreadable:
13. Create a mapfile (here named 12copyfile) for copying:
14. Extra. If directory structure on the new mountpoint has not yet been created, copy what can be copied without errors with cp, scp or rsync. The mountpoint /media/FreeBSDZFS/ should exist for cp to work:
15. Copy files while shortening their names with the help of the mapfile:
16. Extra. Copy files back to the original mountpoint when needed and restore their original names (to copy to a different mountpoint, use sed or other method to replace the original mountpoint in the mapfile). This might overwrite your data:
17. Extra. To remove the files copied the following command should work:
Preparatory 13 commands without comments:
The 15th command which copies the files:
Some additional notes:
rsync copies a wider variety of files than cp (like links)
I've encountered a rumour that rsync can't copy files with 255 byte length or close to 255. In my experience rsync copies 255 byte length file names without issues. For archiving I often use
A nice way to mount a FreeBSD hosted OpenZFS dataset on Linux is with sshfs (for machines connected via network). No installed support for ZFS on Linux is required that way. If you copy on sshfs and get an error "Bad message", this usually means that the file name is too long for the system copied to.
If your file name structure is such that after shortening to 255 byte length you would be getting many duplicate names in same directories, try to shorten further and add a number to the end of the new shortened name. For example, a list of random numbers could be generated and combined with the main list using
If someday you would have a FreeBSD without outdated file name length limitation you could use the 15th command to rename files to longer file names using mv command instead of cp with some minor command and mapfile editing (replacing mountpoints with the relevant one).
So anyone who would want to transfer large amounts of data from NTFS to ZFS on FreeBSD could face many files not being copied with cp/scp/rsync.
This guide provides commands to create a mapfile with which it would be possible to transfer files with long file names from NTFS to OpenZFS hosted on FreeBSD (this can be helpful for copying to many other Unix-like systems including Linux).
The mapfile will match currently existing files with long filenames and their new destinations along with shortened file names for copying, so file names will be shortened.
Only the minimal necessary shortening is done (to 255 bytes) which helps with avoiding duplicates and preserves accessibility of as much information as possible.
Unfortunately, only shortening the file names would allow full access to them while using FreeBSD even though OpenZFS right now supports longer file names.
An alternative is archiving, but you would not be able to extract an archive with long filenames while using FreeBSD.
If you are committed enough, you might want to recompile FreeBSD kernel to allow longer file names, but this was not tested and is not a part of this guide.
The described approach allows to copy files back to NTFS with restoring their full file names if needed and creates matching lists of short and long filenames for use and long file name preservation.
A Linux machine functions as an intermediary here and all commands are for Linux. They can probably be adapted to FreeBSD. FreeBSD-hosted OpenZFS dataset is mounted with a help of sshfs.
These commands might not be the most optimized, but they have been tested by me to solve my problem of transfering files with long file names from NTFS to OpenZFS hosted on FreeBSD.
Always recheck the commands and results in your case.
Original mountpoint:
/media/NTFS/
New mountpoint:
/media/FreeBSDZFS/
Folder with listing files:
/home/user/listingfiles/
Replace the mentioned 3 directories with those you would be using.
1. Create a list of files:
find /media/NTFS/ -maxdepth 99 -type f -printf "%p\n" > /home/user/listingfiles/01listoffiles2. Create a list of files without paths:
awk '{print $NF}' FS=/ /home/user/listingfiles/01listoffiles > /home/user/listingfiles/02listoffilesnopaths3. Create a list of calculated file name byte lengths:
while IFS= read -r line; do printf '%s' "$line" | wc -c; done < /home/user/listingfiles/02listoffilesnopaths > /home/user/listingfiles/03filenamelengthbytecounts4. Create a list of long (longer than FreeBSD allows) and short (those which FreeBSD allows) file names based on length:
while IFS= read -r line; do if [ "$line" -gt 255 ]; then printf '1\n'; else printf '0\n'; fi ; done < /home/user/listingfiles/03filenamelengthbytecounts > /home/user/listingfiles/04filenamelengthcheck5. Create a list of files with 1 or 0 in front of every file path to remove the short names with grep in the next command:
paste -d'\0' /home/user/listingfiles/04filenamelengthcheck /home/user/listingfiles/01listoffiles > /home/user/listingfiles/05filenamelengthforsorting6. Create a list of files with only long file names:
grep ^1 /home/user/listingfiles/05filenamelengthforsorting | cut -c 2- > /home/user/listingfiles/06onlylongfilenames7. Create a list of files with only long file names without paths:
awk '{print $NF}' FS=/ /home/user/listingfiles/06onlylongfilenames > /home/user/listingfiles/07onlylongfilenamesnopaths8. Create a list of long filenames' byte counts:
while IFS= read -r line; do printf '%s' "$line" | wc -c; done < /home/user/listingfiles/07onlylongfilenamesnopaths > /home/user/listingfiles/08onlylongfilenamesbytecounts9. A prophylactic command, as the next command will not replace a listing file but will be adding data incrementally to it:
rm /home/user/listingfiles/09onlylongfilenameslongbytes10. Create a list of bytes to remove to shorten the long file names to 255 bytes:
while IFS= read -r line; do echo $((256-line)) | cut -c 2- >> /home/user/listingfiles/09onlylongfilenameslongbytes ; done < /home/user/listingfiles/08onlylongfilenamesbytecounts11. Combine lists of long filenames and bytes to remove for the next command:
paste -d'?' /home/user/listingfiles/09onlylongfilenameslongbytes /home/user/listingfiles/06onlylongfilenames > /home/user/listingfiles/10forbytelengthsetting12. Create a list of files with last bytes conflicing with FreeBSD removed while preserving only UTF-8 characters since UTF-16 characters cut in half may be unreadable:
awk -F? '{ print substr( $0, $2, length($2)-$1 ) }' < /home/user/listingfiles/10forbytelengthsetting | iconv -f utf-8 -t utf-8 -c | sed 's@^.*/media/NTFS/@/media/FreeBSDZFS/@' > /home/user/listingfiles/11newlongnames13. Create a mapfile (here named 12copyfile) for copying:
paste -d'?' /home/user/listingfiles/06onlylongfilenames /home/user/listingfiles/11newlongnames > /home/user/listingfiles/12copyfile14. Extra. If directory structure on the new mountpoint has not yet been created, copy what can be copied without errors with cp, scp or rsync. The mountpoint /media/FreeBSDZFS/ should exist for cp to work:
cp -a /media/NTFS/* /media/FreeBSDZFS/15. Copy files while shortening their names with the help of the mapfile:
while IFS=? read ntfs freebsdzfs; do cp -a "$ntfs" "$freebsdzfs"; done < /home/user/listingfiles/12copyfile16. Extra. Copy files back to the original mountpoint when needed and restore their original names (to copy to a different mountpoint, use sed or other method to replace the original mountpoint in the mapfile). This might overwrite your data:
#while IFS=? read ntfs freebsdzfs; do cp -a "$freebsdzfs" "$ntfs"; done < /home/user/listingfiles/12copyfile17. Extra. To remove the files copied the following command should work:
#while IFS=? read ntfs freebsdzfs; do rm "$freebsdzfs"; done < /home/user/listingfiles/12copyfilePreparatory 13 commands without comments:
find /media/NTFS/ -maxdepth 99 -type f -printf "%p\n" > /home/user/listingfiles/01listoffiles
awk '{print $NF}' FS=/ /home/user/listingfiles/01listoffiles > /home/user/listingfiles/02listoffilesnopaths
while IFS= read -r line; do printf '%s' "$line" | wc -c; done < /home/user/listingfiles/02listoffilesnopaths > /home/user/listingfiles/03filenamelengthbytecounts
while IFS= read -r line; do if [ "$line" -gt 255 ]; then printf '1\n'; else printf '0\n'; fi ; done < /home/user/listingfiles/03filenamelengthbytecounts > /home/user/listingfiles/04filenamelengthcheck
paste -d'\0' /home/user/listingfiles/04filenamelengthcheck /home/user/listingfiles/01listoffiles > /home/user/listingfiles/05filenamelengthforsorting
grep ^1 /home/user/listingfiles/05filenamelengthforsorting | cut -c 2- > /home/user/listingfiles/06onlylongfilenames
awk '{print $NF}' FS=/ /home/user/listingfiles/06onlylongfilenames > /home/user/listingfiles/07onlylongfilenamesnopaths
while IFS= read -r line; do printf '%s' "$line" | wc -c; done < /home/user/listingfiles/07onlylongfilenamesnopaths > /home/user/listingfiles/08onlylongfilenamesbytecounts
rm /home/user/listingfiles/09onlylongfilenameslongbytes
while IFS= read -r line; do echo $((259-line)) | cut -c 2- >> /home/user/listingfiles/09onlylongfilenameslongbytes ; done < /home/user/listingfiles/08onlylongfilenamesbytecounts
paste -d'?' /home/user/listingfiles/09onlylongfilenameslongbytes /home/user/listingfiles/06onlylongfilenames > /home/user/listingfiles/10forbytelengthsetting
awk -F? '{ print substr( $0, $2, length($2)-$1 ) }' < /home/user/listingfiles/10forbytelengthsetting | sed 's@^.*/media/NTFS/@/media/FreeBSDZFS/@' > /home/user/listingfiles/11newlongnames
paste -d'?' /home/user/listingfiles/06onlylongfilenames /home/user/listingfiles/11newlongnames > /home/user/listingfiles/12copyfileThe 15th command which copies the files:
while IFS=? read ntfs freebsdzfs; do cp -a "$ntfs" "$freebsdzfs"; done < /home/user/listingfiles/12copyfileSome additional notes:
rsync copies a wider variety of files than cp (like links)
I've encountered a rumour that rsync can't copy files with 255 byte length or close to 255. In my experience rsync copies 255 byte length file names without issues. For archiving I often use
rsync -aPAXUH (since -N doesn't work on my Linux)A nice way to mount a FreeBSD hosted OpenZFS dataset on Linux is with sshfs (for machines connected via network). No installed support for ZFS on Linux is required that way. If you copy on sshfs and get an error "Bad message", this usually means that the file name is too long for the system copied to.
If your file name structure is such that after shortening to 255 byte length you would be getting many duplicate names in same directories, try to shorten further and add a number to the end of the new shortened name. For example, a list of random numbers could be generated and combined with the main list using
paste command. That's just one of the possible solutions.If someday you would have a FreeBSD without outdated file name length limitation you could use the 15th command to rename files to longer file names using mv command instead of cp with some minor command and mapfile editing (replacing mountpoints with the relevant one).