Other dosfs screwed when copying files from ZFS to a micro SD

I'm going crazy with my FreeBSD machine when trying to copy files to a micro SD card using fat 32 filesystem.
The micro SD is a snadisk 256 GB, I've already tried two of them, thru two different USB adaptors (one being Sandisk, if that maters) and on different USB ports.

Let's start with the end result:

Code:
mammoth% doas mount_msdosfs /dev/da0s1 /mnt
mammoth% ls /mnt/ELEMENTARY/01
Elementary - 01x01 - Arma impropria.mkv                 Elementary - 01x12 - Il mandante.mkv
Elementary - 01x02 - Mentre dormivi.mkv                 Elementary - 01x13 - La squadra rossa.mkv
Elementary - 01x03 - L'uomo dei palloncini.mkv          Elementary - 01x14 - Il deduttore.mkv
Elementary - 01x04 - Corsa al successo.mkv              Elementary - 01x15 - Caricata a droga.mkv
Elementary - 01x05 - L'angelo della morte.mkv           Elementary - 01x16 - Dettagli.mkv
Elementary - 01x06 - Sabotaggio.mkv                     Elementary - 01x18 - Deja Vu.mkv
Elementary - 01x07 - Conflitto di coscienza.mkv         Elementary - 01x19 - Angeli nella neve.mkv
Elementary - 01x08 - Miccia lunga.mkv                   Elementary - 01x20 - Ricatto col morto.mkv
Elementary - 01x09 - Fare da sé.mkv                     Elementary - 01x21 - Un fatto storico.mkv
Elementary - 01x10 - La cassaforte inespugnabile.mkv    Elementary - 01x22 - Gestione del rischio.mkv
Elementary - 01x11 - Panni sporchi.mkv                  Elementary - 01x23 - Peonie.mkv


mammoth% ls /mnt/ELEMENTARY/03
!g+??r\f.ǰ?     ?.g??;?f.}??    ??d?"???.?3?    R_n`lJ??.???    s?XSM?/?.?r?    ?h?A?}??.p??    ??C&??4?.&2?
#@<g?UK?.???    ?4?c4!??.??V    ????????.?4v    Ra{?lt??.???    s??wM???.?qg    ?k6e?~A?.p??    ??VQ??!?.&??
#C?C?V??.???    ?:???/?7.???    ??i0????.?7?    Rb?3lw??.??d    s???M???.?or    ??q???&u.\#?    ???"????.)3-
#}?4?h??.???    ?;x?0.?3.???    ??;7??L?.d(?    U?\Sk?+?.?'?    s?M$M?:?.?l?    ??d???3?.\??    ???u????.)?8
#~)??k^?.???    ????)???.???    ???T<?}?.?M?    U?I$k?>?.???    t*j`J???.???    ?[c??n?f.[??    ????????.?Dx
$??T??y?.?8?    ??d???3?.~z?    ????8?l.?A?     VP??hE?0.??@    t???J???.???    ?ev??p?1.[??    ??RQ??%?.?y?
$??#??l?.???    ??j?7?=?.?C?    ?Ҳ?8??s.?|1     VS??hF??.???    w???I???.?]v    ?f???s??.[?i    ??G&??0?.?G?
%V?A?C??.???    ??m?0??d.???    ????????.??r    Vn??h{?G.??u    w?I?I?>'.?^?    ?x???m?b.[?\    ???u????.?zm
'L-??YZ?.???    ??r????1.}??    ??m????3.???    ]]?@chh?.???    w?\?I?+P.?`?    ???s??g?.?n?    ??????a?.-??
'O???Z?7.???    ??u???"u.zv?    ???7??s.?~`     ]^?dck??.???    w???I??t.?cc    ?@???U?S.??u    ??????tr.-<?
'q???d?@.???    ??x???/'.N??    ??ub&???.???    ]`??cu??.???    x???f??t.?a2    ?~???k?$.??`    ?!???4??.y?i
'r8??gOd.?ݶ     ??{㭼?c.o&?     ????????.s?*    ^???`?k?.?)?    x???f?o?.?\?    Ʀm0????.3b?     ???a????.y?\
(m???x??.??C    ???????@.???    ???3????.k)U    ^???`?~p.???    x???f??#.?_'    ??xG????.3|?    ?d1??qfa.*??
(n|??[?3.???    ????????.N?g    ???%????.~y{    `??c^???.?4Z    x???f?zp.?b?    ?Jq??_??.!??    ?z$??os6.*??
(pi??e?d.???    ????4???.??C    ???????0.o?Q    aZDv_O3?.???    yl??gy?g.???    ?tdb?a??.!ٿ     ???V??w?.qf?
(s???f?`.??V    ????<???.?p     ????????.`??    adQ?_q&?.???    yo;?gzlc.???    ??????sU.ut?    ???r????.qe?
*ik2?|??.???    ?E?q.Pa?.???    ??q?"??a.???    b?g?\??1.?.?    yq.?gdy4.???    ????ۦ?q.uw;     ???!??b?.qx?
+??p????.?9`    ?J??*_?!.??8    ????"??e.??n    b???\???.?-X    z??wd???.?&2    ????????.ui.    ?:_??/(?.4??


I copied the files witht he command:
Code:
mammoth% cp -Rf /tank/VIDEO/SOAP/ELEMENTARY /mnt
and no errors where thrown.

The same files, copied on a fat32 hard disk work as expected, there is no filesystem corruption.

I prepared the micro SD as follows:
Code:
root@mammoth:~ # gpart add -t mbr da0

root@mammoth:~ # gpart list da0
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 524290047
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: da0s1
   Mediasize: 268436472320 (250G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 32256
   Mode: r1w1e1
   efimedia: HD(1,MBR,00000000,0x3f,0x1f4007c1)
   rawtype: 11
   length: 268436472320
   offset: 32256
   type: fat32
   index: 1
   end: 524290047
   start: 63
Consumers:
1. Name: da0
   Mediasize: 268436504576 (250G)
   Sectorsize: 512
   Mode: r1w1e2

root@mammoth:~ # newfs_msdos -F 32 -L FLUCA_MINI da0s1

My operating system is:
Code:
mammoth% uname -a
FreeBSD mammoth 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64

What am I missing here?
 
Looks like you're trying to store movies on that card, but I cannot get their size. You are aware that on common FAT32 the maximal size of a file is 4G?
 
Try using exfat instead of plain fat32 - it's still crap, but at least it won't suffer from 4G limitation and IIRC handles filename conversion automatically (i.e. replaces any non-ascii/DOS character with a placeholder)

If no windows host is involved it is usually best to avoid FAT altogether and use a real filesystem. For file transfers I even use zfs on flash media - yes, it's slower, but I have guaranteed file integrity, which IMHO is crucial on unreliable media like USB flash drives.
 
and use a real filesystem.
😂 I love such ones.
Nah, on sdcards I never had any real issues with fat32 (exfat); but I also had the issue:"...wtf?!... - uh, yeah, file is 6G."

But I agree: If there is no other system involved but FreeBSD, use ufs, or even zfs. Plus you don't have to mess with filenames UTF-8 vs. UTF-16, as T-Daemon mentioned.
 
Looks like you're trying to store movies on that card, but I cannot get their size. You are aware that on common FAT32 the maximal size of a file is 4G?

Yes I know, and all the files are around 1.2GB and a folder does not contain more than 25 files. Filenames do not contain any utf-8 char, so there should be no problem with that. Filenames are shorter than 256 chars.
 
😂 I love such ones.
Nah, on sdcards I never had any real issues with fat32 (exfat); but I also had the issue:"...wtf?!... - uh, yeah, file is 6G."

But I agree: If there is no other system involved but FreeBSD, use ufs, or even zfs. Plus you don't have to mess with filenames UTF-8 vs. UTF-16, as T-Daemon mentioned.

I need the micro sd to be plugged into android devices, so neither zfs nor ufs are options. NTFS could be, but I don't like using this on FreeBSD, so what can I choose?
 
just to understand that error a bit better:
the garbled filenames are shown immediately after you copied them over? or after you removed and re-inserted the SD card? did you sync before removal?
 
just to understand that error a bit better:
the garbled filenames are shown immediately after you copied them over? or after you removed and re-inserted the SD card? did you sync before removal?

They are shown immediatly after the cp finishes, and before I umount the media.
 
They are shown immediatly after the cp finishes, and before I umount the media.
Perhaps there is a corruption happening. Try fsck_msdosfs(8), and sysutils/smartmontools to check SD card health.

I need the micro sd to be plugged into android devices, so neither zfs nor ufs are options. NTFS could be, but I don't like using this on FreeBSD, so what can I choose?
Assuming the SD card is healthy, why not let Android device format the card (not sure what format it chooses, perhaps exFAT [2] ), or format the card with a file system Android understands (ext4 [1] or exFAT [2]) and can be mounted on FreeBSD.

ext4 mounted on FreeBSD by ext2fs(5) (after loading the kernel module).

[1]
e2fsprogs-core-1.47.2_1 Utilities & library to manipulate ext2/3/4 filesystems

[2] exFat:
exfat-utils-1.4.0_1 Utilities to create, check, label and dump exFAT filesystem
fusefs-exfat-1.4.0_1 Full-featured exFAT FS implementation as a FUSE module
 
What are the filenames in the source for the 03/ subdirectory?

Code:
mammoth% ls /tank/VIDEO/SOAP/ELEMENTARY/03
Elementary - 03x01 - Delitto perfetto.mkv
Elementary - 03x02 - Cinque perline arancioni.mkv
Elementary - 03x03 - Indovinelli mortali.mkv
Elementary - 03x04 - Bella.mkv
Elementary - 03x05 - Imbroglio.mkv
Elementary - 03x06 - Terra pericolosa.mkv
Elementary - 03x07 - Omicidi alla noce moscata.mkv
Elementary - 03x08 - Fine turno.mkv
Elementary - 03x09 - Vita eterna.mkv
Elementary - 03x10 - L'erba del vicino.mkv
Elementary - 03x11 - L'illustre cliente (prima parte).mkv
Elementary - 03x12 - L'illustre cliente (seconda parte).mkv
Elementary - 03x13 - Cicuta.mkv
Elementary - 03x14 - L'esemplare femmina.mkv
Elementary - 03x15 - Il prezzo della vita.mkv
Elementary - 03x16 - Amnesia.mkv
Elementary - 03x17 - Criogenesi.mkv
Elementary - 03x18 - Onniscienza.mkv
Elementary - 03x19 - Amicizia.mkv
Elementary - 03x20 - Un'azione tempestiva.mkv
Elementary - 03x21 - Sottopelle.mkv
Elementary - 03x22 - Interessi privati.mkv
Elementary - 03x23 - Latitante.mkv
Elementary - 03x24 - Verso l'abisso.mkv

The only strange thing I see is for those files with the single quote, but that would lead to (in the worst case) a file problem, not a full corruption.
 
Perhaps there is a corruption happening. Try fsck_msdosfs(8), and sysutils/smartmontools to check SD card health.


Assuming the SD card is healthy, why not let Android device format the card (not sure what format it chooses, perhaps exFAT [2] ), or format the card with a file system Android understands (ext4 [1] or exFAT [2]) and can be mounted on FreeBSD.

Well, because I trust FreeBSD to do the job, I only need to understand how to perform it.
Besides, I've tried by formatting it on the android side and mounting it with
mount_msdosfs and that lead to the very same result.

ext4 mounted on FreeBSD by ext2fs(5) (after loading the kernel module).

[1]
e2fsprogs-core-1.47.2_1 Utilities & library to manipulate ext2/3/4 filesystems

[2] exFat:
exfat-utils-1.4.0_1 Utilities to create, check, label and dump exFAT filesystem
fusefs-exfat-1.4.0_1 Full-featured exFAT FS implementation as a FUSE module


So far I'm testing exfat:

Code:
gpart create -s gpt da0
gpart add -t ms-basic-data -a 1M -l FLUCA_MICRO da0
mkfs.exfat -n FLUCA_MICRO da0p1
mount.exfat /dev/da0p1 /mnt

I'm copying files, and I will report back what it happens.
 
Code:
mammoth% ls /tank/VIDEO/SOAP/ELEMENTARY/03
Elementary - 03x01 - Delitto perfetto.mkv
Elementary - 03x02 - Cinque perline arancioni.mkv
Elementary - 03x03 - Indovinelli mortali.mkv
Elementary - 03x04 - Bella.mkv
Elementary - 03x05 - Imbroglio.mkv
Elementary - 03x06 - Terra pericolosa.mkv
Elementary - 03x07 - Omicidi alla noce moscata.mkv
Elementary - 03x08 - Fine turno.mkv
Elementary - 03x09 - Vita eterna.mkv
Elementary - 03x10 - L'erba del vicino.mkv
Elementary - 03x11 - L'illustre cliente (prima parte).mkv
Elementary - 03x12 - L'illustre cliente (seconda parte).mkv
Elementary - 03x13 - Cicuta.mkv
Elementary - 03x14 - L'esemplare femmina.mkv
Elementary - 03x15 - Il prezzo della vita.mkv
Elementary - 03x16 - Amnesia.mkv
Elementary - 03x17 - Criogenesi.mkv
Elementary - 03x18 - Onniscienza.mkv
Elementary - 03x19 - Amicizia.mkv
Elementary - 03x20 - Un'azione tempestiva.mkv
Elementary - 03x21 - Sottopelle.mkv
Elementary - 03x22 - Interessi privati.mkv
Elementary - 03x23 - Latitante.mkv
Elementary - 03x24 - Verso l'abisso.mkv

The only strange thing I see is for those files with the single quote, but that would lead to (in the worst case) a file problem, not a full corruption.

I would be worth it to copy these one by one to find out which one triggers the corruption of the filenames in the directory.
 
Well, because I trust FreeBSD to do the job, I only need to understand how to perform it.
Besides, I've tried by formatting it on the android side and mounting it with
mount_msdosfs and that lead to the very same result.




So far I'm testing exfat:

Code:
gpart create -s gpt da0
gpart add -t ms-basic-data -a 1M -l FLUCA_MICRO da0
mkfs.exfat -n FLUCA_MICRO da0p1
mount.exfat /dev/da0p1 /mnt

I'm copying files, and I will report back what it happens.

Gosh, errors also using exfat:

Code:
cp: /mnt/ELEMENTARY/03/Elementary - 03x16 - Amensia.mkv: Input/output error
mount.exfat[1368]: unexpected entry type 0x57 after 0xffffffff at 0/5

the error repeats for a lot of files with a different type but same offset.
I'm going to do as suggested by cracauer and test one file at a time...
 
You should try copying to a virtual disk managed by md(4). That will allow you to separate the limitations of the filesystems and the bugs in their implementations from hardware problems (or driver problems).

You can also prepare an image on a virtual disk and then copy it to an SD card with dd(1).
 
Code:
mammoth% ls /tank/VIDEO/SOAP/ELEMENTARY/03
Elementary - 03x01 - Delitto perfetto.mkv
Elementary - 03x02 - Cinque perline arancioni.mkv
Elementary - 03x03 - Indovinelli mortali.mkv
Elementary - 03x04 - Bella.mkv
Elementary - 03x05 - Imbroglio.mkv
Elementary - 03x06 - Terra pericolosa.mkv
Elementary - 03x07 - Omicidi alla noce moscata.mkv
Elementary - 03x08 - Fine turno.mkv
Elementary - 03x09 - Vita eterna.mkv
Elementary - 03x10 - L'erba del vicino.mkv
Elementary - 03x11 - L'illustre cliente (prima parte).mkv
Elementary - 03x12 - L'illustre cliente (seconda parte).mkv
Elementary - 03x13 - Cicuta.mkv
Elementary - 03x14 - L'esemplare femmina.mkv
Elementary - 03x15 - Il prezzo della vita.mkv
Elementary - 03x16 - Amnesia.mkv
Elementary - 03x17 - Criogenesi.mkv
Elementary - 03x18 - Onniscienza.mkv
Elementary - 03x19 - Amicizia.mkv
Elementary - 03x20 - Un'azione tempestiva.mkv
Elementary - 03x21 - Sottopelle.mkv
Elementary - 03x22 - Interessi privati.mkv
Elementary - 03x23 - Latitante.mkv
Elementary - 03x24 - Verso l'abisso.mkv

The only strange thing I see is for those files with the single quote, but that would lead to (in the worst case) a file problem, not a full corruption.

fun fact about UTF8: there are over 10 different "minus/hyphen/dash" characters that usually all look the same in most fonts... so those could also throw off everything that still can't speak unicode (e.g. FAT).

I remember that audio/deadbeef also choked on some files from my collection - renaming them with (allegedly) the same name fixed it, because the original filename contained some 'exotic' variants of characters that looked like the 'common' ones (IIRC some apostrophes were the culprit; of which there are also dozens of variants in UTF8...)
 
Doesn't shell you're using trying to process the single quotes as "quoted"?
What happenes if you try to copy using GUI filers like x11-fm/caja or something you prefer?
Not 100% sure it works or not, as I've recently encountered filename truncation problem on downloading purchased flac files from Qobuz using www/firefox, which contained single/double quotes and any charachers disallowed for filenames.
 
How is the SD card inserted into the machine? SD card slot, SD card USB adapter?

I would also check the cards health (sysutils/smartmontools). The card could be worn out.

I already wrote that is inserted via USB SndDisk adaptor.
The cards, as well as the adaptor, are new. I'm going to test it with a real filesystem (e.g., UFS) just to see if the problem is an hardware prbolem or a filesystem problem.
 
fun fact about UTF8: there are over 10 different "minus/hyphen/dash" characters that usually all look the same in most fonts... so those could also throw off everything that still can't speak unicode (e.g. FAT).


I'm going to do in turn these experiments:
  • test the micro SD card with a FreeBSD native filesystem
  • copy one file at the time onto exfat to see if there is a particular file that triggers the first and subsequent errors
  • rename all the files to dumb names without quotes, dashes, and alike and see if it works.

What puzzles me is that the same files copied onto an USB mechanical hard drive work as expected (filesystem fat32).
 
Back
Top