semanticTokenScopes produces wrgon mappin

[matklad]

Version: 1.44.0-insider
Commit: 20b88fa
Date: 2020-04-01T08:29:34.507Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Linux x64 4.19.113

Problem: I am developing an extension which takes advantage of semantic highlighting, and looks like "semanticTokenScopes" selector matching is broken. Specifically, my pluging defines the mapping like this:

        "semanticTokenScopes": [
            {
                "language": "rust",
                "scopes": {
                    "attribute": [
                        "meta.attribute"
                    ],
                    "builtinType": [
                        "support.type.primitive"
                    ],
                    "lifetime": [
                        "entity.name.lifetime.rust"
                    ],
                    "typeAlias": [
                        "entity.name.typeAlias"
                    ],
                    "union": [
                        "entity.name.union"
                    ],
                    "keyword.unsafe": [
                        "keyword.other.unsafe"
                    ],
                    "keyword.control": [ // <- the interesting bit
                        "keyword.control"
                    ],
                    "variable.constant": [
                        "entity.name.constant"
                    ]
                }
            }
        ]

but somehow control styling is applied to all keywords, not only to control ones:

Note that I also use a "zero" textmate grammar for rust, to make sure that there's no interference wit tm, but the same bug is observable with default rust grammar as well.

Steps to Reproduce:

1. Install rust-analyzer from source
2. Enable semantic highlighitng with { "rust-analyzer.highlighting.semanticTokens": true, "editor.semanticHighlighting.enabled": true, }
3. Observe all keywords colored as control keywords

[aeschli]

We have a set of default rules configured, and keyword currently goes to TextMate scope keyword.control: see

vscode/src/vs/platform/theme/common/tokenClassificationRegistry.ts

Line 499 in 9ed52e3

registerTokenType('keyword', nls.localize('keyword', "Style for keywords."), [['keyword.control']]);

You can overwrite that for rust, if you prefer:

"semanticTokenScopes": [
            {
                "language": "rust",
                "scopes": {
                    "keyword": [
                        "keyword"
                    ]
            }
]

[matklad]

Ah... Yeah, that explains the observed behavior. I wonder though, why default maps to keyword.control and not just to keyword?

[aeschli]

It comes from the built-in default themes where we differentiate between keywords ('if', 'function', ...) ('keyword.control') and operators ('and', 'not') ('keyword.operator'). So we basically interested in the color that a theme gives to the the first set.

[matklad]

Hm, right, that makes sense. Textmate scopes are confusing,I hope we’ll transition to more consistent semantic selectors eventually. Thanks for the explanation!