Do FreeBSD Command Line Text Utilities Support Wide Characters and Unicode?

[Pinetree]

Do command line (CLI) text utilities in the FreeBSD "World" (like wc, grep, sed, awk and so on) support wide characters (through wchar_t representation and wc character and string functions or by some other means of the C Language) and UTF encodings? Or is that not true and these terminal programs understand only ASCII and numerous 8-bit encodings, based on ASCII, till now?

I understand, this is an elementary question, but I'm not a permanent FreeBSD user and I'm not aware of the situation.

 

[Erichans]

Welcome to FreeBSD.

Generally speaking: yes.

Your questions is part of the internationalisation/localisation domain, you'll find more about that in:
Handbook - Chapter 25. Localization - i18n/L10n Usage and Setup.

When dealing with command line utilities, you can have:

1. your shell & terminal & keyboard and keyboards language settings
2. the utility itself

For all holds that whether they can appropriately function with "wide characters" is dependent on the locale(1); this is not different then with any other *nix based OS; or most likely any other modern OS. FreeBSD supports UTF-8. "Wide characters" is an imprecise term and relates to character encodings and character sets. UTF-8 is a character encoding (one of many) of Unicode. Character encodings and characters sets are quite broad and extensive subjects.

If you want to work on FreeBSD with UFT-8 encoded characters, the above items of the list must be suitable for handling them; as for the software you have to set your locale to UTF-8. From all of the standard utilities you mention, awk(1) is, AFAIK, not too long ago "extended" to fully support UTF-8; more information and documentation you can get at onetrueawk/awk, that is the awk(1) that FreeBSD has in its base-install. There are however exceptions, in FreeBSD's base ee(1) does not fully support UTF-8 for example.

For programs and utilities outside the base-install, i.e. from ports (that is also where your packages come from), you'll have to look for yourself in the documentation. Most support UTF-8, but I have no knowledge of specific percentages.