FBSD7.1R Serving Windows clients - Filename issues

[FestusHagen]

Hi All,

Wasn't sure where to post this...
I've tried searching, but don't know what to search for....

FBSD7.1R
Samba 3.0?

Windows filenames that have characters from the higher half of the Windows code page (cp1252) ("é,Ö,á") are getting munged... Is there anyway configure fbsd so it maintains the use of the same characters as M$lop???

The real problem is with my music library, Artists like Mötley Crüe and Blue Öyster Cult are getting munged.

I've tried copying with Explorer from M$lop via Samba.
I've tried cp via mount_smbfs with and without -E codepages.

Any tips, pointers to info leading to a resolution would be greatly appreciated.

Thanks

-Enjoy
fh : )_~

 

[FestusHagen]

I take it there is no resolution to the incompatibility of FreeBSD in a M$lop environment (world)...

fh : )_~

 

[mickey]

Windows filenames that have characters from the higher half of the Windows code page (cp1252) ("é,Ö,á") are getting munged... Is there anyway configure fbsd so it maintains the use of the same characters as M$lop???

Check the setting of "unix charset" in your smb.conf configuration file.

From the smb.conf(5) manpage:

unix charset (G)

   Specifies the charset the unix machine Samba runs on uses. Samba
   needs to know this in order to be able to convert text to the
   charsets other SMB clients use.

   This is also the charset Samba will use when specifying arguments
   to scripts that it invokes.

   Default: unix charset = UTF8

   Example: unix charset = ASCII

 

[FestusHagen]

Check the setting of "unix charset" in your smb.conf configuration file.

From the smb.conf(5) manpage:

unix charset (G)

   Specifies the charset the unix machine Samba runs on uses. Samba
   needs to know this in order to be able to convert text to the
   charsets other SMB clients use.

   This is also the charset Samba will use when specifying arguments
   to scripts that it invokes.

   Default: unix charset = UTF8

   Example: unix charset = ASCII

Thanks for your suggestion.

In my complete reading of the Samba man pages multiple times (Including that section), I find it states by default Samba uses 'LOCALE', Locale is UTF8! (Default en.FBSD)
Not to mention the other places that it refers to Samba's defaults matching Windows.

However, This is not a Samba issue, In my OP I state: "I've tried cp via mount_smbfs with and without -E codepages.", smbfs does not use Samba.

Another thing...
On a smbfs mouted Windows share, using 'ls' and 'find' produce two different results with these characters.

# find /TestMusic
Displays the characters (filenames) correctly...

However!
# ls -R /TestMusic
Does not!

-Enjoy
fh : )_~

 

[mickey]

I find it still unclear, what exactly you are trying to accomplish. Use FreeBSD/Samba to serve files to Windows clients, or use mount_smbfs to mount an SMB filesystem from a Windows machine?

 

[FestusHagen]

I find it still unclear, what exactly you are trying to accomplish. Use FreeBSD/Samba to serve files to Windows clients, or use mount_smbfs to mount an SMB filesystem from a Windows machine?

Simply put, File naming compatibility between FreeBSD and Windows!

So valid Windows files can be stored on a FreeBSD Server WITHOUT MANGLING filenames!

Via All possible methods! Mainly Samba and SMBFS.

For example: The Album title:

The Best of Blue Öyster Cult​

Gets mangled when trying to store on FreeBSD.
And depending on if it's copied via Samba or SMBFS it gets mangled differently.

At the very least consistency would be a half step forward.

Thanks

-Enjoy
fh : )_~

 

[mickey]

Simply put, File naming compatibility between FreeBSD and Windows!

So valid Windows files can be stored on a FreeBSD Server WITHOUT MANGLING filenames!

Well, my best guess is, file name mangling was put into place, to achieve compatibility between platforms.

Personally I have been using Samba for quite some time now, to serve files to Windows clients. On the Unix side, these are stored using ISO-8859-1. This works in both directions and has never given me any problems or headaches.

I can not tell much about mount_smbfs, as I have never used it. When I wanted to give it a try, I couldn't even get past authentication. Maybe it lacks NTLMv2 auth? For Unix<->Unix file sharing, I prefer NFS anyways.

 

[FestusHagen]

Well, my best guess is, file name mangling was put into place, to achieve compatibility between platforms.

That would not be compatibility ... The definition of Compatibility is: (Take note of #5)

Main Entry: com·pat·i·ble
Pronunciation: \kəm-ˈpa-tə-bəl\
Function: adjective
Etymology: Middle English, from Medieval Latin compatibilis, literally, sympathetic, from Late Latin compati
Date: 15th century

1 : capable of existing together in harmony <compatible theories> <compatible people>
2 : capable of cross-fertilizing freely or uniting vegetatively
3 : capable of forming a homogeneous mixture that neither separates nor is altered by chemical interaction
4 : capable of being used in transfusion or grafting without immunological reaction (as agglutination or tissue rejection)
5 : designed to work with another device or system without modification;
especially : being a computer designed to operate in the same manner and use the same software as another computer

— com·pat·i·bil·i·ty \-ˌpa-tə-ˈbi-lə-tē\ noun
— compatible noun
— com·pat·i·ble·ness \-ˈpa-tə-bəl-nəs\ noun
— com·pat·i·bly \-blē\ adverb
​

Personally I have been using Samba for quite some time now, to serve files to Windows clients. On the Unix side, these are stored using ISO-8859-1. This works in both directions and has never given me any problems or headaches.

I have been using Samba since before it was called Samba, It was called something like DOS Pathworks.
And I Love it!

However, I have never worried about this because I have never used it for my main server, I've always used M$lop! (or Novell in the older days) I am now trying to get completely rid of my M$lop Server and this is a HUGE issue to me!
I have a very large audio collection (>33k titles) that I will NOT allow to get mangled.

If you don't mind doing a simple test for me to confirm that it can actually work I would sure appreciate it...

On a Windows box, Create path/filenames like:

The Best of Blue Öyster Cult
Mötly Crüe
Beyoncé
Tiësto
​

Copy to a FreeBSD box, then do a 'ls' or 'find' (from the console, not in X) of that filename on the FreeBSD box does it look correctly??????

Now copy that path/file back to a Windows box, Is it still correct?????

If you have X, Check it out to, Is everything at least consistent???
​

I cannot do that! The names will get mangled, Depending how it's done/looked at are all different ... No consistency.

If it works for you I would be EXTREMELY interested in what you have done to accomplish this.

I can not tell much about mount_smbfs, as I have never used it. When I wanted to give it a try, I couldn't even get past authentication. Maybe it lacks NTLMv2 auth? For Unix<->Unix file sharing, I prefer NFS anyways.

SMBFS works great, Never had an issue except for the filename mangling and I truly believe that can be fixed with the right tweeks...

What those tweeks are is what I am trying to find out, for both Samba and SMBFS.

Thanks for your input.

-Enjoy
fh : )_~

 

[mickey]

If you don't mind doing a simple test for me to confirm that it can actually work I would sure appreciate it...

On a Windows box, Create path/filenames like:

The Best of Blue Öyster Cult
Mötly Crüe
Beyoncé
Tiësto
​

Copy to a FreeBSD box, then do a 'ls' or 'find' (from the console, not in X) of that filename on the FreeBSD box does it look correctly??????

Now copy that path/file back to a Windows box, Is it still correct?????

If you have X, Check it out to, Is everything at least consistent???
​

I assume by saying "Not in X", you mean within a shell, not some freaky filemanager...

So here's what I got...

I can create directories with those names from a windows machine either on the local harddrive or a Samba network volume. Windows sees all those names correctly as i created them.

I can move those directories from the Samba volume to Windows' local harddrive, and back, the names stay intact.

Logged into the Samba server via SSH, using ls -al from the shell, here's the output:

total 24
drwxr-x---  6 mickey  music  512 14 Nov 01:17 .
drwxr-x---  9 mickey  music  512 14 Nov 01:11 ..
drwxr-x---  2 mickey  music  512 14 Nov 01:17 Beyoncé
drwxr-x---  2 mickey  music  512 14 Nov 01:17 Mötly Crüe
drwxr-x---  2 mickey  music  512 14 Nov 01:17 The Best of Blue Öyster Cult
drwxr-x---  2 mickey  music  512 14 Nov 01:17 Tiësto

Looking at those directories from yet another machine, which has the same volume mounted via NFS, the output is exactly the same.

Using find to look for a name like "*Mötly*" or "*Crüe*" yields the expected result, locally on the server, as well as on the NFS mounted volume.

Gnome's file-manager Nautilus displays the directories exactly, as they are supposed to, no matter, if i view them on the NFS mount, or via smb://...

And even the FreeBSD console displays them correctly. So I cannot say that there's much of a problem here.

 

[FestusHagen]

And even the FreeBSD console displays them correctly. So I cannot say that there's much of a problem here.

Awesome! Thank you VERY much!
It's possible, I just couldn't believe it wasn't.

What I meant by "Not in X", Was from the FreeBSD console, Not a Shell within X.

Now I wonder where the breakage is in my system...

smb.conf

[global]
unix charset = ISO8859-1

login.conf
(Under standard: (my user class is standard, yes, I ran cap_mkdb))
(I followed the lead of the Russian Example in login.conf)

:charset=ISO8859-1: \
:lang=en_US.ISO8859-1: \

locale

LANG=en_US.ISO8859-1
LC_CTYPE="en_US.ISO8859-1"
LC_COLLATE="en_US.ISO8859-1"
LC_TIME="en_US.ISO8859-1"
LC_NUMERIC="en_US.ISO8859-1"
LC_MONETARY="en_US.ISO8859-1"
LC_MESSAGES="en_US.ISO8859-1"
LC_ALL=

Terminal

TERM=cons25l1

What's missing?
Is any of that wrong?
Any other info needed?
Is that everything that would affect this ?

The anxiety has ramped 10 fold, I see light at the end of the tunnel, But how long is that tunnel!

-Enjoy
fh : )_~

 

[mickey]

What I meant by "Not in X", Was from the FreeBSD console, Not a Shell within X.

That should not be much of a difference, as both shells undergo the same initialization procedure, i.e. read /etc/csh.cshrc ...

The only real difference could be the font that is in use.

Now I wonder where the breakage is in my system...

smb.conf

[global]
unix charset = ISO8859-1

I have this:

        unix charset = iso-8859-15
        display charset = iso-8859-15

But I suppose that makes no difference. -15 is -1 plus Euro sign, and according to the man page, the display charset is only used for messages to stdout/stderr.
I wonder though, whether samba expects it to be "ISO8859-XX" or "iso-8859-XX", or if it doesn't really care.

login.conf
(Under standard: (my user class is standard, yes, I ran cap_mkdb))
(I followed the lead of the Russian Example in login.conf)

:charset=ISO8859-1: \
:lang=en_US.ISO8859-1: \

locale

LANG=en_US.ISO8859-1
LC_CTYPE="en_US.ISO8859-1"
LC_COLLATE="en_US.ISO8859-1"
LC_TIME="en_US.ISO8859-1"
LC_NUMERIC="en_US.ISO8859-1"
LC_MONETARY="en_US.ISO8859-1"
LC_MESSAGES="en_US.ISO8859-1"
LC_ALL=

My login.conf file is the standard one, that comes installed with FreeBSD. No charset or other relevant settings defined there. Would not make much sense either, as GDM since 2.26.X ignores settings from login.conf anyways.

My locale settings are a bit more esoteric

LANG=en_US.ISO8859-15
LC_TIME=en_GB.ISO8859-15
LC_MONETARY=de_DE.ISO8859-15
LC_CTYPE=de_DE.ISO8859-15
LC_COLLATE=de_DE.ISO8859-15
LC_MESSAGES=en_US.ISO8859-15
LC_NUMERIC=de_DE.ISO8859-15

Terminal

TERM=cons25l1

Same here for the console.

What's missing?
Is any of that wrong?
Any other info needed?
Is that everything that would affect this?

What font are you using on the console?
In my /etc/rc.conf, I have the following:

font8x8="iso15-8x8.fnt"
font8x14="iso15-8x14.fnt"
font8x16="iso15-8x16.fnt"

Maybe the characters are correct, but the current font is not able to display them correctly?

 

[FestusHagen]

Awesome! 1 step forward!

I have this:

        unix charset = iso-8859-15
        display charset = iso-8859-15

But I suppose that makes no difference. -15 is -1 plus Euro sign, and according to the man page, the display charset is only used for messages to stdout/stderr.
I wonder though, whether samba expects it to be "ISO8859-XX" or "iso-8859-XX", or if it doesn't really care.

I have tried both ways, without success!
Still cannot get Samba to quit mangling the filenames....

However, I can now create and display filenames with those characters locally on FreeBSD.

My login.conf file is the standard one, that comes installed with FreeBSD. No charset or other relevant settings defined there. Would not make much sense either, as GDM since 2.26.X ignores settings from login.conf anyways.

This is where the success starts!
I have to have the locale set to ISO8859-1 ... How to do it other than the login.conf is unknown to me, from what I gather from the man pages, this is the correct method...

Correct or Incorrect???
Hopefully someone in the know will respond.

My locale settings are a bit more esoteric

LANG=en_US.ISO8859-15
LC_TIME=en_GB.ISO8859-15
LC_MONETARY=de_DE.ISO8859-15
LC_CTYPE=de_DE.ISO8859-15
LC_COLLATE=de_DE.ISO8859-15
LC_MESSAGES=en_US.ISO8859-15
LC_NUMERIC=de_DE.ISO8859-15

Ya think ... Wow!
Seems to me that would be confusing to the system.

What font are you using on the console?
In my /etc/rc.conf, I have the following:

font8x8="iso15-8x8.fnt"
font8x14="iso15-8x14.fnt"
font8x16="iso15-8x16.fnt"

Maybe the characters are correct, but the current font is not able to display them correctly?

THAT'S IT!!!!!!
After adding the following to rc.conf

[INDENT]font8x8="iso-8x8.fnt"
font8x14="iso-8x14.fnt"
font8x16="iso-8x16.fnt"
[/INDENT]

NOTE the difference! (standard 8859-1 (iso-), not 15 (iso15-) as you have)

I can now do 'ls' or 'find' and the correct characters are displayed.
Why this all is not set during installation when language is selected is beyond me. IMNSHO it should be!

So this is what I had to change to properly handle the proper display of high characters. ("é,Ö,á" etc...)

login.conf
(I wound up adding it to default (this is a private system), It really should not be, It should be added to the appropriate section. don't forget cap_mkdb)

[INDENT]:charset=ISO8859-1: \
:lang=en_US.ISO8859-1: \
[/INDENT]

rc.conf

[INDENT]font8x8="iso-8x8.fnt"
font8x14="iso-8x14.fnt"
font8x16="iso-8x16.fnt"
[/INDENT]

With those two changes I can now create and display those high characters from the FreeBSD console.

Now I can totally focus on Samba ...
Any other ideas???

Mickey, Thank you very very much for your input.
It has been very helpful.

-Enjoy
fh : )_~

 

[mickey]

How to do it other than the login.conf is unknown to me, from what I gather from the man pages, this is the correct method...

Correct or Incorrect???

Correct.
I already had all my settings in login.conf once, but since the version bump from GDM 2.20.X to >= 2.24.X, GDM completely ignores login.conf, so I had to revert to the old way of putting my settings into /etc/profile and /etc/csh.cshrc.

Otherwise login.conf is absolutely the preferred way, as it is independant of the shell you use, so you do not have to put the same settings into multiple shell init files.

Ya think ... Wow!
Seems to me that would be confusing to the system.

Only to software, that thinks it can break down all locale settings, to a single $LANG *sigh*

I can now do 'ls' or 'find' and the correct characters are displayed.
Why this all is not set during installation when language is selected is beyond me. IMNSHO it should be!

Well, the last time I used sysinstall, must have been back in the days of FreeBSD 4.X... But I'm pretty sure that i have seen a menu entry for changing the console screen font somewhere deep down in the post-installation tuning options. :e

Anyway, I'm glad I could help.