Is emacs broken?

[PMc]

,,, or am I doing it wrong?

Please try the following:

• start emacs with default settings in an xterm (textonly terminal mode) with a new file
• set to conf-mode ("ESC-X conf-mode CR")
• enter four lines of text:

one
two
#three                                                                          
#four

The two lines with hash-marks should now be colored differently.

Mark all four lines with the mouse (left-click and mark), and paste (middle-click) them in again below.
What I am getting is then this (note the empty line):

one
two
#three                                                                           
#four
one
two
#three                                                                          \
                                                                                
#four

The problem gets obvious when pasting the selection into hexdump:

00000000  6f 6e 65 0a 74 77 6f 0a  23 74 68 72 65 65 20 20  |one.two.#three  |
00000010  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
*
00000050  20 20 20 20 20 20 20 20  0a 23 66 6f 75 72 20 20  |        .#four  |
00000060  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
*
000000a0  20 20 20 20 20 20 20 20  20 0a                    |         .|

 

[Crivens]

More likely the text snap is the problem here - I would guess that VI does the same under the same circumstances.

 

[PMc]

More likely the text snap is the problem here - I would guess that VI does the same under the same circumstances.

It definitely doesn't. This is specific to emacs and happens in all lines where emacs did change the color.
I can do "set-background-color" and it will then happen at every line.

It may well be a problem with the text snap or the X or our compilation of X or whatever, but it appears specifically when emacs changes color.

Please try it out. I need to know if it is reproducible, I can't really imagine this going undetected.

 

[Crivens]

simple test - does it happen in VT (not X11)?

 

[PMc]

simple test - does it happen in VT (not X11)?

How could it? There is no cutbuffer, and mouse support (if active at all) is entirely different.

But it happens on Linux all the same.

 

[Crivens]

You have no mouse support under VT? I habe, and the spaces at the end of a MMB press are a well known thing for me.

 

[PMc]

You have no mouse support under VT?

I don't know - probably only on physical graphic stations where a mouse is attached. I don't know how to make it working in vncviewer - and I'm not fond of using vncviewer on console when I can login via ssh.

This is not the point. I hav dozens of nodes (machines, jails, virtuals), and all can be opened in xterms, and then config files etc. can be compared side by side, and edited, and snippets copied around. But this is not practical if there is trailing spaces garbage coming along.

I habe, and the spaces at the end of a MMB press are a well known thing for me.

That never happened here until emacs. The X cutbuffer was the reason I started to use X, some 30+ years ago, and it always worked.

I had editors/jove as the systems editor, from even before Linux/FreeBSD existed, but that cannot do UTF. I can use vi, in emergencies or over a modem line, but not for everyday editing. So, since jove already used emacs keybindings, I decided to roll out emacs as the systems editor everywhere, make it work for system users without a homedir, etc., and to behave somehow decently for that usecase.

 

[Zonoia]

Have you tried M-x xterm-mouse-mode?
Which version of Emacs are you running?
Have you tried reproducing the issue using the graphical Emacs?
Have you tried reproducing this issue in other terminals like st and alacritty?

 

[PMc]

Have you tried M-x xterm-mouse-mode?

Yes. That one doesn't work at all. Apparently emacs grabs the mouse events and keeps them to itself. So another emacs in another xterm doesn't get the copied data and just reports "No selection is available."
Also it is not even clear how this should be used. Simply marking the text with the mouse seems to define a region, and it is unclear if this should already do the job or if a "Copy" keystroke is also necessary.Anyway there seems to be no functionality.

Which version of Emacs are you running?

2024Q3, i.e. 29.4.3

Have you tried reproducing the issue using the graphical Emacs?

It doesn't reproduce - but this seems a pointless question: in graphical mode we do not use terminal escape sequences to set the color.

Have you tried reproducing this issue in other terminals like st and alacritty?

Never heard of these. Are these terminal programs to use in X?
Then that might be worth a try.

However, further analysis leads to the idea that this might be a lack of communication with the curses library:
As shown above, the problem does appear with syntax highlighting, and only selectively with certain lines. These lines have in common that they are comment lines. This gets illustrative with a C sourcefile:

/* bla bla              <line is filled with spaces>
 *                      <line is filled with spaces>
 */                     <line is correctly terminated!>
main()                  <line is correctly terminated!>
{                       <line is correctly terminated!>
}                       <line is correctly terminated!>

So what is special with comments? They do not end anywhere, but continue to the end of the line.

The third line here illustrates this: it is also a comment, but it does end. And it does not show the error!

This looks like somebody has forgotten to tell curses that the line ends somewhere before the right window border (since that necessity isn't very obvious), and consequentially the X cutbuffer collects it up to the right window border.

 

[Charlie_]

alacritty does not add spaces.

 

[Zonoia]

So another emacs in another xterm doesn't get the copied data and just reports "No selection is available."

That's fair enough. I'd like to point to you being able to save your 'copied' data not just in the application pastebuffer but also into your X pastebuffer (I have this set up for Vim and Emacs). This has the added benefit of using more advanced copying features, most of which mouseless. https://www.emacswiki.org/emacs/CopyAndPaste

This of course won't help you when copying from a terminal that is a portal to another system however (through SSH and what-have-you).
So I understand if this isn't enough, but to be frank, I nearly always use this method instead as I centrally manage my machines.


As for extra data regarding this:

When I use Emacs in the Gnome Wayland terminal, and I select a few lines, copy with ctrl-shift-c and paste it elsewhere with ctrl-shift-v, I get spaces on all lines, regardless of syntax highlighting.

And so I installed xterm. After being befuddled how backspace didn't work properly for some reason, I carried on and I found that on each line, spaces are added making all lines equal to terminal width without runover. Regardless of highlighting. So the same as the Gnome terminal.

Next, I tried Alacritty, and just like Charlie found, it does not add spaces.

Lastly, I tried st, which is a nice, lighter weight alternative to xterm. It also does not add spaces and thus works perfectly fine.

So my conclusion:
It seems to depend on the terminal emulator's behaviour. So you might want to consider trying an alternative terminal emulator. But I haven't been able to reproduce your problem, and all of the terminal emulators I tried have mostly reasonable behaviour. This might very well be a bug somewhere in your software stack / configuration.

 

[PMc]

alacritty does not add spaces.

That is interesting. And there are apparently lots of terminal programs - not really surprising, given the many desktop users interested in improving desktop experience. None of them seem really impressing beyond that, and among Qterminal, rxvt, lilyterm, Eterm - none works.
alacritty seems to be the rare exception - but too limited for my purposes.

 

[Charlie_]

No spaces were added in x11/rio either.
Does the GPU-accelerated terminal emulator not add whitespace?

 

[RussellASC]

I had no issues in urxvt. Emacs 29.4 (Xaw not GTK).

 

[Charlie_]

On my 14.1R urxvt, blanks were added.

 

[PMc]

I had no issues in urxvt. Emacs 29.4 (Xaw not GTK).

But I have. Same problem as above.
And I couldn't find an option for either Xaw or GTK there.

 

[PMc]

That's fair enough. I'd like to point to you being able to save your 'copied' data not just in the application pastebuffer but also into your X pastebuffer (I have this set up for Vim and Emacs). This has the added benefit of using more advanced copying features, most of which mouseless. https://www.emacswiki.org/emacs/CopyAndPastehttps://www.emacswiki.org/emacs/CopyAndPaste

Not yet getting the point here, need to read this.
But the main thing that got me hooked onto X is that copy/paste works keyless. Click once to select characters, twice for words, thrice for lines, pull along and then throw that selection into whatever you like - and that's all and (usually) works everywhere.
This is how was the unix way of old, on the workstations, and it was fine.
Then windows came along, and they had to do it somehow differently, because they had no X. And they decided to do it with keys. So now we got keys, and they are different everywhere, and it is all a mess.
And things like Firefox want to force us to use keys and do it Windows-like - some singlepage javascript applications found a means to disable the unix way.

Then, emacs has it's own understanding of deleting/copying and reinserting text, which is part of the editiing functions and happens inside a single editor instance; so that should be independent.

As for extra data regarding this:

When I use Emacs in the Gnome Wayland terminal, and I select a few lines, copy with ctrl-shift-c and paste it elsewhere with ctrl-shift-v, I get spaces on all lines, regardless of syntax highlighting.

And so I installed xterm. After being befuddled how backspace didn't work properly for some reason, I carried on and I found that on each line, spaces are added making all lines equal to terminal width without runover. Regardless of highlighting. So the same as the Gnome terminal.

Did you set-background-color or similar?
This is where I first noticed the issue; then it appears on every line. Only later I found syntax highlighting does also do it.

Next, I tried Alacritty, and just like Charlie found, it does not add spaces.

Lastly, I tried st, which is a nice, lighter weight alternative to xterm. It also does not add spaces and thus works perfectly fine.

So my conclusion:
It seems to depend on the terminal emulator's behaviour. So you might want to consider trying an alternative terminal emulator. But I haven't been able to reproduce your problem, and all of the terminal emulators I tried have mostly reasonable behaviour. This might very well be a bug somewhere in your software stack / configuration.

I tried alacritty and rio, and both do indeed not have the problem. But most of the well-known and long-standing alternatives have it. And since Charlie_ did reproduce it with urxvt, I don't think it is a local issue. What I would like to do is tap into the virtual terminal /dev/pts and see what emacs sends there (and probably tap into the X server and see what is sent there). But that appears to be a bit of work.

 

[PMc]

That's fair enough. I'd like to point to you being able to save your 'copied' data not just in the application pastebuffer but also into your X pastebuffer (I have this set up for Vim and Emacs). This has the added benefit of using more advanced copying features, most of which mouseless. https://www.emacswiki.org/emacs/CopyAndPaste

This of course won't help you when copying from a terminal that is a portal to another system however (through SSH and what-have-you).
So I understand if this isn't enough, but to be frank, I nearly always use this method instead as I centrally manage my machines.

I got it. In terminal mode, emacs has no way ever to reach the X pastebuffer, because it has no knowledge of an X server.
And there are no X libaries installed on the node where the editor runs, neither would it be feasible to have X network communications with some server in Singapore or Mexico.
And the linked article is only discussing functions that apply to graphical operation, i.e. emacs as an X application.

So apparently the issue here seems completely independent of emacs, as emacs just writes characters to a terminal. Then the xterm application grabs these characters, produces graphical output and sends that to an X server. And on receipt of mouse events the xterm collects what it assumes to be text and sends it to the X server's cutbuffer. Emacs is not involved in this.

The interesting question, which points back to emacs, then is: what different kinds of characters does emacs send to the terminal, depending on the textual line being a comment or something else? Because only there can be the reason why the xterm would behave differently in these cases.

 

[Charlie_]

If a line is coloured only in the middle of a line, no spaces are added.
I thik if the text is coloured, emacs also assigns an attribute after the end of the line, so it will probably be spaces when copied.

 

[Crivens]

The interesting question, which points back to emacs, then is: what different kinds of characters does emacs send to the terminal, depending on the textual line being a comment or something else?

It sends chars with a background color, not empty space (not spaces w.o. Color). Maybe it has in it's text buffer all the ANSI control sequences for each character position to set the background, and thus it can not see that as empty space. What Charlie_ found must be a special case.

 

[PMc]

There are two functions in term.c that appear to switch on and off the decoration, including foreground and background color.

emacs/src/term.c at 658715c977c78c312685cd6c594a2cbc0d31fc7d · emacs-mirror/emacs

Mirror of GNU Emacs. Contribute to emacs-mirror/emacs development by creating an account on GitHub.

github.com

emacs/src/term.c at 658715c977c78c312685cd6c594a2cbc0d31fc7d · emacs-mirror/emacs

Mirror of GNU Emacs. Contribute to emacs-mirror/emacs development by creating an account on GitHub.

github.com

It is quite obvious what these do, and I do not see from this where the problem might be.

 

[RussellASC]

This sounds like a terminal issue to me, not Emacs.

What TERM are you using? If you default to something like vt100, does the paste behavior work? Try "export TERM=vt100" and repeat your experiment.

I too value basic xterm style copy and paste, but if you're using Emacs in the terminal then the issue is likely at the terminal layer.

 

[PMc]

This sounds like a terminal issue to me, not Emacs.

Do You have a rationale for that assumption?

What TERM are you using? If you default to something like vt100, does the paste behavior work? Try "export TERM=vt100" and repeat your experiment.

Normally xterm-256color, but it is irrelevant as it doesn't make a difference. The TERM valus is just an environment variable that is read by the application, and depending on that value the application will send the respective escape-codes to the terminal. The terminal itself will continue to be an xterm, and the escape-codes will then either work (if the TERM value is sufficiently compatible to xterm) or produce crap on the screen.

The problem itself is probably not to be solved by assumptions, but by actually analyzing the data sent along and understanding what is going on. However, I do not think that I am oblidged to do that; I would rather see the emacs coders look into this, since it is not difficult to reproduce. And anyway, even if I would figure out what is going wrong, I would hardly be able to fix the emacs, as that heavily lispified C tends to rather create knots in my brain than anything working.

I too value basic xterm style copy and paste, but if you're using Emacs in the terminal then the issue is likely at the terminal layer.

Yeah, enjoy the wondrous world of interdependent effects. The issue is apparently with the X pastebuffer, which resides in the X server - because there the trailing spaces do appear. The issue is also with emacs, because there are apparently different ways for emacs to print a line on the screen and some of them get trailing spaces while others do not. The terminal, then again, is right in between these two components.

Interestingly, just sending colored text to the terminal, by itself does not create the trailing spaces.

In earlier times this would have been a simple matter of doing emacs > /some/file and then looking at the hexdump of that. Nowadays emacs does no longer honor stdin or stdout, neither the -t option as described in the manpage. Instead it insists on using /dev/tty, and /dev/tty appears not so easily to be faked:

# ln /dev/pts/0 /dev/tty
ln: /dev/tty: File exists
# ln -f /dev/pts/0 /dev/tty
ln: /dev/tty: No such file or directory

So yes, indeed, I got pissed.

 

[RussellASC]

Normally xterm-256color, but it is irrelevant as it doesn't make a difference

It absolutely makes a difference. The terminal type can change how Emacs and other TUI applications send data to the screen, and impact your paste. It can also impact how the terminal program interprets what it receives.

I'd suggest trying vt100, vt102, xterm and your xterm-256color and see if the behavior changes.

Some terminals vary in how they handle spaces to edge of screen. Ever had issues with line wrapping, or used alt-drag to select a box of text? This could be Emacs exposing a bug in the terminal program just as easily as it could be an Emacs bug.

Have you tried different terminal emulators, or only xterm?

 

[RussellASC]

xterm(1)

xterm has a toggle to trim whitespace off the end of a copy...