Help me debug xterm

[schweikh]

I'm in contact with the xterm maintainer (Thomas Dickey) since I observed that modern xterm (version 402 and current 403; see echo $XTERM_VERSION) has a problem with some double width characters on my end of the net, but not his. Maybe we can gather some statistics to get to the bottom of this.

In particular, among others, the trademark symbol is displayed as a double width character when it should be single width. You can try this with a shell that supports the \uXXXX format for Unicode codepoints (bash, zsh).

printf '<\u2122>\n'
<™ >    # What I see: with space
<™>     # What Thomas sees: no space

If I understand the code in wcwidth.c correctly, xterm is told to make TM a double width by its own unicode table, which differs from what libc's wcwidth returns (width = 1).

Please let me know what you see with either 402 or 403 (and if you have 401 still around, why not try it and see if the behavior is different). Thanks!

 

[cracauer@]

Hmm...

My FreeBSD-16-current xterm:

% /usr/bin/printf '<\u2122>\n'
<u2122>

 

[karel]

> echo $XTERM_VERSION
XTerm(403)
> bash -c "printf '<\u2122>\n'"
<™>

 

[schweikh]

Hmm...

My FreeBSD-16-current xterm:

% /usr/bin/printf '<\u2122>\n'
<u2122>

You should use the built-in printf in bash or zsh, /usr/bin/printf does not understand \uXXXX.

 

[cracauer@]

You should use the built-in printf in bash or zsh, /usr/bin/printf does not understand \uXXXX.

bash:

% printf '<\u2122>\n'
<?>

 

[hruodr]

$ printf '<\u2122>\n'
<™>
$ echo $XTERM_VERSION
XTerm(402)

 

[bakul]

zsh% printf '<\u2122>\n'
<™>
zsh% echo $XTERM_VERSION
XTerm(397)
zsh% uname -a
FreeBSD xxx 14.3-RELEASE-p5 FreeBSD 14.3-RELEASE-p5 GENERIC amd64
zsh% appres XTerm xterm|grep utf8Fonts.font:
*VT100.utf8Fonts.font: -misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1

xterm on freebsd, xserver (XQuartz) on macos.

 

[schweikh]

bash:

% printf '<\u2122>\n'
<?>

This will probably print a TM in some way when you "export LC_CTYPE=C.UTF-8".

 

[Charlie_]

401 is single width.

 

[PMc]

This here says XTerm(402) and produces a rectangle between <>
The rectangle is one char width, so together three chars (and a linefeed) are printed

I don't think this depends on which end of the net something happens, because I would have a hard time explaining on which end of the net I am (this depends almost entirely on who is asking ).

 

[schweikh]

For those of you with a C compiler and not afraid to use it, maybe we can get some hints from the xterm-provided test programs.
All you need to do is extract the port, run configure, then make and run the test program:

$ cd /usr/ports/x11/xterm
$ make extract
$ cd work/xterm-403
$ ./configure
$ make check
[...snippety...]
** executing test_wcwidth
.. range 32-126
0/95 mismatches (0%)
0/95 mismatches (0%)
.. range 160-0xff00            ## This is
732/65121 mismatches (1%)      ## the interesting
732/65121 mismatches (1%)      ## range containing u2122
.. range 0x10000-0x11000
166/4097 mismatches (4%)
166/4097 mismatches (4%)

Also this:
$ ./test_wcwidth 0x2000-0x2200
U+2028  -1      0
U+2029  -1      0
U+203C  2       1
U+2049  2       1
U+20C1  1       -1
U+2122  2       1          <-- 2 means double width (thus mismatch)
U+2139  2       1
U+2194  2       1
U+21A9  2       1
9/513 mismatches (2%)

Does the last command say "U+2122 1 1" for you, or as shown above?

 

[hruodr]

** executing test_wcwidth
.. range 32-126
0/95 mismatches (0%)
0/95 mismatches (0%)
.. range 160-0xff00
554/65121 mismatches (1%)
554/65121 mismatches (1%)
.. range 0x10000-0x11000
126/4097 mismatches (3%)
126/4097 mismatches (3%)
...

Then:

% ./test_wcwidth 0x2000-0x2200
U+2028 -1 0
U+2029 -1 0
U+203C 2 1
U+2049 2 1
U+2122 2 1
U+2139 2 1
U+2194 2 1
U+21A9 2 1
8/513 mismatches (2%)

 

[zzfs6]

% export LC_CTYPE=C.UTF-8
% echo $XTERM_VERSION
XTerm(403)
% printf '<\u2122>\n'
<u2122>

 

[cracauer@]

This will probably print a TM in some way when you "export LC_CTYPE=C.UTF-8".

Something's still wedged:

(bash)

% LC_CTYPE=C.UTF-8 printf '<\u2122>\n'
<â
  ¢>

xterm 403.

 

[freethread]

I use tcsh shell so...

~> node -e "console.log ('<\u2122>');console.log ('XTERM_VERSION = ' + process.env['XTERM_VERSION']);"
<™>
XTERM_VERSION = XTerm(402)

Edit: LANG=LC_ALL=it_IT.UTF-8

 

[zzfs6]

It will print from a C program, but not printf(1).

#include <stdio.h>
#include <stddef.h>
#include <locale.h>

int main(void)
{
    setlocale(LC_CTYPE, "");
    wchar_t c = 0x2122;
    printf("%lc\n", c);
}

 

[schweikh]

$ printf '<\u2122>\n'
<™>
$ echo $XTERM_VERSION
XTerm(402)

So no space after TM? It's hard to tell since the forum's font may render the codepoint differently (in my first post I had to manually edit the code).

 

[hruodr]

So no space after TM?

No space. Under the xterm of 14.3

 

[blackbird9]

Running 14.3R updated to the latest level, I get xterm version 402, 'TM' and definitely no space.
Running the bash shell gives me:-

$ echo $XTERM_VERSION
XTerm(402)
$ printf '<\u2122>\n'
<™>

Locale:
$ locale
LANG=C.UTF-8
LC_CTYPE="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_TIME="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_ALL=

 

[SirDice]

dice@williscorto:~ % bash
[dice@williscorto ~]$ printf '<\u2122>\n'
<™>
[dice@williscorto ~]$ echo $XTERM_VERSION
XTerm(403)
[dice@williscorto ~]$ echo $LANG
C.UTF-8

Seems to do what it's supposed to do.

 

[schweikh]

It seems my end of the net is the problem. Thanks everybody for your help. I'll chase a little bit more what's funny here. I suspect the cause to be in the way my X11 is configured, or my xterm, or my environment, or my libc, or my Unicode, or my system, which is a FrankensteinBSD made from ports and packages starting at 14.2 and annoying me with warnings that no ports are guaranteed to work anymore.