"comment" semantic token type is overly broad vs the syntax types that it overrides when themeing

[woody77]

Issue Type: Bug

There are multiple TextMate scopes for comments:

Rust:
comment.line.double-slash.rust
comment.line.documentation.rust

Java:
comment.line.double-slash.java
comment.block.javadoc.java

However, the semantic tokens only allow for comment, which breaks the ability to theme doc comments (comment.block.javadoc.java, comment.line.documentation.rust) separately from standard code comments.

When semantic highlighting is enabled, all comments become comment to the theme.

VS Code version: Code 1.44.2 (ff91584, 2020-04-16T17:07:18.473Z)
OS version: Darwin x64 19.3.0
Remote OS version: Linux x64 5.2.17-1rodete3-amd64

Extensions (16)

Extension	Author (truncated)	Version
remote-containers	ms-	0.112.0
remote-ssh	ms-	0.51.0
remote-ssh-edit	ms-	0.51.0
remote-wsl	ms-	0.44.2
vscode-remote-extensionpack	ms-	0.20.0
vscode-gn	npc	1.1.0
vscode-icons	vsc	10.1.1
gitstash	art	3.2.0
gitlens	eam	10.2.1
language-fidl	fuc	0.2.0
rust-analyzer	mat	0.2.151
git-graph	mhu	1.22.0
python	ms-	2020.4.76186
cpptools	ms-	0.27.1
Go	ms-	0.14.1
git-rebase-shortcuts	tre	1.0.5

(3 theme extensions excluded)

[vscodebot]

(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

• Commenting doesn't work if child elements are commented in HTML (#92777)
• Search only code or comments (#92781)

[woody77]

I see that there's a *.documentation semantic token modifier. It doesn't seem to be in use for any examples, and the documentation I've found for it says "Style to use for references in documentation"), which implies that it's a modifier that would be applied to a token types such as struct or enum which are found within documentation comments, not to comment lines or blocks as a whole.

If the intention is for the latter, I'll close this and open a bug against the LSP I'm using (rust-analyzer) to map /// comments to comment.documentation semantic type+modifier.

[aeschli]

Before adding a new type to the standard I'd suggest you to pioneer this for Rust first.

It should be done with a new token type that extends the comment token type.

{
  "contributes": {
    "semanticTokenTypes": [
      {
        "id": "docComment",
        "superType": "comment",
        "description": "A doc comment."
      }
    ]
}

Then for themes to support this, add a semanticTokenScopes contribution, with a rust specific rule (unless TextMate has a well established scope for doc comments):

{
  "contributes": {
    "semanticTokenScopes": [
      {
        "language": "rust",
        "scopes": {
          "docComment": ["comment.line.documentation.rust"]
        }
      }
    ]
  }
}

[aeschli]

I created #97063 to collect proposals for new standard token types/modifiers.