Help with Missing Charsets and Tor Timming Out

Hello, I am new to FreeBSD and BSD in general. I am loving FreeBSD a lot so far. I installed my system and have been tinkering with it a little bit, and have a few things I haven't been able to reslove myself. First thing, my system langauge is english as that is my native language however I have a lot of music in Chinese, japanese etc. the files and metadata for said files are written in thei respective lanaguges. These files don't show their characters however, but rather question marks (??? those). I am confident this is due to missing charsets as I had the same issue on Arch Linux. I followed this: https://cooltrainer.org/a-freebsd-desktop-howto/#install-fonts, however after rebooting after installing all those fonts the issue wasn't fixed. How would I reslove this issue? Second issue, I use tor for some things and have it also running on my other laptop so I know this issue isn't due to it not working on my network. I have installed tor and set it to run at boot as well as did the following:

then vi /etc/sysctl.conf:

# prevent traffic analysis that exploits sequential IP IDs
net.inet.ip.random_id=1

After that was all done I setup my many applications to run through it, they all seem to time out when they try to connect to the network through tor. How do I fix this? Thank you for your time.
 
The Howto section is not for asking how to do things.

Select the right forum for new threads

And I suggest splitting your question, it's usually not very effective if you ask for help on two completely different subjects, the thread will get confusing with different answers intermingled.
 
The first place you need to start is with localization for your language.
https://www.freebsd.org/doc/handbook/using-localization.html
I have been going through that and apologize, but I am having a hard time understanding it. So for example, say I want my main langauge to be US english, but I have MP3s that are in Chinese. These MP3s show ? marks where the Chinese should be, how do I have it show the Chinese? In case it helps, here is everything I have done to my system. Every command and every package installed: https://pastebin.com/eRuDeGU2.
 
The tags showing "?" doesn't really have anything to do with *language* or using correct fonts (if it was fonts problem, you would see boxes with numbers inside showing the character code instead), those are (most likely) in different *codeset* (encoding), so you need a tool translating those tags from say Big5 or GB18030 to UTF-8 (that's what you use, right?). One obvious example is iconv(1), but I can't give any more advice here as I don't know what or how you use those exactly.

One thing you *could* try is run the program that is showing "?" with an encoding frequently used with Chinese (you need to guess here), e.g. LC_CTYPE=zh_CN.GB18030 program or LC_CTYPE=zh_CN.Big5 program (there's no guarantee it will help, or if it's useful).
 
The tags showing "?" doesn't really have anything to do with *language* or using correct fonts (if it was fonts problem, you would see boxes with numbers inside showing the character code instead), those are (most likely) in different *codeset* (encoding), so you need a tool translating those tags from say Big5 or GB18030 to UTF-8 (that's what you use, right?). One obvious example is iconv(1), but I can't give any more advice here as I don't know what or how you use those exactly.

One thing you *could* try is run the program that is showing "?" with an encoding frequently used with Chinese (you need to guess here), e.g. LC_CTYPE=zh_CN.GB18030 program or LC_CTYPE=zh_CN.Big5 program (there's no guarantee it will help, or if it's useful).
Is there a way to transfer the encoding to utf8 on those files?
 
At least you can see yours... I have 60 .mp3's with Russian character set that I cannot see in the folder, cannot delete the folder on the USB stick as root, or move it to the trashcan as root. I do all that kind of work by invoking x11-fm/xfe as root through the terminal and should be done with it.

I have installed russian/p5-Lingua-RU-Charset and still cannot even see the files in the folder on the USB stick. I do have x11-fonts/font-screen-cyrillic installed, the prior is supposed to detect and convert it. When I initially transferred the folder to the HDD it transferred all the English characters worded files but gave an error on these.

kuku4kokopuffs.png
kuku4kokopuffs2.png

I will eventually remove everything else I have stored on the stick and dd it to solve the issue, as I'm more interested in that than the mp3's but have not been able to fix it myself to this point.
 
To answer this question you need to specify where exactly you are seeing the "?" stuff -- is it filenames, or mp3 tags, or...? :)
I posted this comment already, but think I posted it to the wrong person. If this now shows twice I apologize, I am not use to how the FreeBSD forum works just yet. When I get home, I'll upload an image of my issue. It is hard to describe at the moment and I feel a visual will help :)
 
At least you can see yours... I have 60 .mp3's with Russian character set that I cannot see in the folder, cannot delete the folder on the USB stick as root, or move it to the trashcan as root. I do all that kind of work by invoking x11-fm/xfe as root through the terminal and should be done with it.

I have installed russian/p5-Lingua-RU-Charset and still cannot even see the files in the folder on the USB stick. I do have x11-fonts/font-screen-cyrillic installed, the prior is supposed to detect and convert it. When I initially transferred the folder to the HDD it transferred all the English characters worded files but gave an error on these.

View attachment 5528
View attachment 5529

I will eventually remove everything else I have stored on the stick and dd it to solve the issue, as I'm more interested in that than the mp3's but have not been able to fix it myself to this point.
I haven't actually been able to move my music over to my machine from the USB it's on either. It keeps giving an error about the ? marks. I will upload an image of the issue in hopes it helps. Definitely feel your pain though.
 
Meanwhile, just to show what I mean, here's a simple example:

1. Create some files with cyrillic in names (my current locale is C.UTF-8):
thor:yuripv:~/tst$ touch тест1
thor:yuripv:~/tst$ touch файл2
thor:yuripv:~/tst$ touch хлам3
thor:yuripv:~/tst$ ll
total 2
13619 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 тест1
13620 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 файл2
13621 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 хлам3

They show up correctly in the listing, good.

2. Now convert the filenames to KOI8-R (was widely used for Russian previously):
thor:yuripv:~/tst$ for i in *; do mv $i $(echo $i | iconv -f utf8 -t koi8-r); done
thor:yuripv:~/tst$ ll
total 2
13620 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 ????2
13621 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 ????3
13619 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 ????1

Not so good, byte sequences that are perfectly legal in KOI8-R are illegal in UTF-8, so ls can NOT display the filenames now.

You guys could use the same method, but in reverse direction, translating from the encodings the filenames are now in to UTF-8, just need to know which one that is :) i.e.:

$ for i in *; do mv $i $(echo $i | iconv -f <CURRENT-ENCODING> -t utf8); done
 
Meanwhile, just to show what I mean, here's a simple example:

1. Create some files with cyrillic in names (my current locale is C.UTF-8):
thor:yuripv:~/tst$ touch тест1
thor:yuripv:~/tst$ touch файл2
thor:yuripv:~/tst$ touch хлам3
thor:yuripv:~/tst$ ll
total 2
13619 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 тест1
13620 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 файл2
13621 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 хлам3

They show up correctly in the listing, good.

2. Now convert the filenames to KOI8-R (was widely used for Russian previously):
thor:yuripv:~/tst$ for i in *; do mv $i $(echo $i | iconv -f utf8 -t koi8-r); done
thor:yuripv:~/tst$ ll
total 2
13620 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 ????2
13621 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 ????3
13619 -rw-r--r-- 1 yuripv staff uarch 0 Nov 14 20:49 ????1

Not so good, byte sequences that are perfectly legal in KOI8-R are illegal in UTF-8, so ls can NOT display the filenames now.

You guys could use the same method, but in reverse direction, translating from the encodings the filenames are now in to UTF-8, just need to know which one that is :) i.e.:

$ for i in *; do mv $i $(echo $i | iconv -f <CURRENT-ENCODING> -t utf8); done
I am also gonna post a file I have been making with every command and edit I have made on my system in it, I think there are some things I possibly missed. I am starting to feel like this is fixable though :D
 
yuripv I got a readout using russian/rux but it is a mess. They're all The Sensational Alex Harvey Band songs:

rosettastoned.png

I don't know how I would go about entering any of that output to chardetect. security/wipe won't wipe a directory, I'm sure dd will.
 
Sorry, I misunderstood you. That's a more readable format:

rosettastone.png

But it's history now. I moved the 6GB of data backup I had back onto the HDD and dd made short work of it. I'm in the process of transferring it back onto the clean USB stick.

I do appreciate your help but am glad to be done with it now.
 
To answer this question you need to specify where exactly you are seeing the "?" stuff -- is it filenames, or mp3 tags, or...? :)
Here is a screenshot and a list of ei
OK, looks like we have a winner: textproc/py-chardet. Try feeding the output from ls to chardetect command that the port installs.
I guess I can't upload an image. Everytime I got to do so via imgur FireFox quits. So I guess I can't on my FreeBSD laptop.
 
Here is a screenshot and a list of ei

I guess I can't upload an image. Everytime I got to do so via imgur FireFox quits. So I guess I can't on my FreeBSD laptop.

If you've already got a screenshot use the Attach files option to your lower left and upload it directly.
 
Trihexagonal I didn't say anything about ports in russian/, just do a echo /media/da0s1/Alex/SAHB/* | chardetect.
I haven't been able to fix the charset issue, I wanted to upload an image to show the issue I am having. Whenever I am trying to upload an image on here, imgur etc. FireFox justs quits instead of opening Thunar. How do I fix this? I am so sorry for all the questions, I have been trying my best and seem to keep hitting problems :(
 
This version of www/firefox has crashed on me more than any I've used.

If you're using pkg and not ports (don't mix them) I would try www/palemoon. It will only take a minute to install and may be more stable for you. It's what I'm using now.

I'd build it anyway. That way you're not stuck on one if a vulnerability is found. I have www/seamonkey installed too but it's vulnerable and has been for a while.
 
This version of www/firefox has crashed on me more than any I've used.

If you're using pkg and not ports (don't mix them) I would try www/palemoon. It will only take a minute to install and may be more stable for you. It's what I'm using now.

I'd build it anyway. That way you're not stuck on one if a vulnerability is found. I have www/seamonkey installed too but it's vulnerable and has been for a while.
They haven't been fixing the vulnerabilities and other issues? :(
 
They haven't been fixing the vulnerabilities and other issues? :(

Patches for 3rd party programs like www/seamonkey have to come from upstream, Mozilla in this case, and is not a FreeBSD shortcoming.

FreeBSD is the base system and the terminal you are presented with after your first reboot. Everything else is 3rd party programs. No Xorg, DE or WM, multimedia players, etc. and where most vulnerabilities come into play.

You stay current with FreeBSD vulnerabilities by using # freebsd-update fetch and # freebsd-update install if there is a return. Run # pkg audit -F to check your 3rd party programs and go from there.
 
Back
Top