Cross rendering in presence of surrogate pairs

[alabuzhev]

Some codepoints in Unicode are represented by two wchar_ts, but logically occupy only one screen cell. Example.
It looks like Colorer processes them correctly internally (thanks to ICU?), but there's at least one issue with rendering:
EditorColor.StartPos and EditorColor.EndPos are physical string indices, not screen coordinates. It works fine with the text, because Far is able to project physical indices into screen coordinates itself. However, these fields are also used for the cross, which is a visual element. The EndPos of the horisontal cross line is reported as if all wchar_ts occupy exactly one cell, e.g. 120 if the editor width is 120, which doesn't look right if that's not the case:

Coincidentally, this is not an issue for neither fullwidth wchar_ts (e.g. Katakana on the screenshot), because in that case the visual position is greater than the real and beyond the edge of the screen, nor for fullwidth surrogates (e.g. ideographs on the screenshot), because their fullwidthness compensates the collapse of their wchar_ts.

The easiest way to fix this is probably reporting the right end of the cross as some ridiculously high number.

A better one, especially if there are other "visual" elements possible, would probably be introducing a new flag in EDITORCOLORFLAGS to let Far know that a particular EditorColor contains visual coordinates instead of real.

[ctapmex]

Yes, ICU hide from me string structure. But when I working with whitespace and cross I using simple work with string

FarColorer/src/FarEditor.cpp

Lines 631 to 633 in d557d88

	if (egs.StringText[j] == L' ' || egs.StringText[j] == L'\t') {
	while ((j <= llen) && (j < lend) && (egs.StringText[j] == L' ' || egs.StringText[j] == L'\t')) {
	j++;

. I think this is my mistake.
how to reproduce this picture ?
if enabled fullwidth-aware rendering , and using windows terminal 1.8.1521.0 I have many visual artifacts on rendering without cross

after enable cross I have many visual artifacts , but not like you

I can make the cross very long, but I cannot verify.

--edit--
oh, I think in your case the problem is in the parameter fullback. Setting it on true;

[alabuzhev]

how to reproduce this picture ?
after enable cross I have many visual artifacts , but not like you

We're limited by the technology of our time, but one day it will work.
Until then, the workaround is to replace them with something else:
https://github.com/FarGroup/FarManager/blob/956e9deb51627996b6cf7e115ecd7a0914b505af/far/interf.cpp#L790-L794
to at least be able to verify that everything is calculated properly.

oh, I think in your case the problem is in the parameter fullback

Nope, it's already enabled. Without it the the cross isn't really functional at all :)

[alabuzhev]

Still not fixed

[ctapmex]

I thought we were waiting for a fix in terminal. Or turn it on https://github.com/FarGroup/FarManager/blob/956e9deb51627996b6cf7e115ecd7a0914b505af/far/interf.cpp#L790-L794 ?

[alabuzhev]

The latest conhost/wt can render these things more or less acceptably (as can be seen on the screenshot), and there is now a workaround in Far anyway (as the absence of the aforementioned comment in the current master implies), so no extra steps are needed.

[ctapmex]

we need Interface.VirtualTerminalRendering = true.
I've been trying to play up the situation for a long time. the console was twitching terribly )

[alabuzhev]

Ah yes, in non-VT mode we can't do much.
Fullwidth-aware rendering should be on too.

[ctapmex]

I did a hack - doubled the length of the cross by default. I didn't find any other options, for example, to determine the size of the symbol when displayed.
The behavior of the console is interesting when line 1561 of encoding.cpp you extend it with the same symbols. They "disappear". Only moving the cursor to the left shows them
https://github.com/colorer/FarColorer/releases/tag/v1.4.24

[alabuzhev]

doubled the length of the cross

That should do, thanks. Far checks all the sizes and draws only what is needed anyway.

I didn't find any other options, for example, to determine the size of the symbol when displayed.

Determining the size of the symbol is a hard problem.
As I mentioned in the description, we could add a new flag to indicate that StartPos & EndPos are visual coordinates and let Far do the rest, but perhaps some other time when/if this issue becomes more relevant.