Repository navigation
switch to a different tool for python syntax highlighting #5824
Description
MagicPython doesn't seem to have been updated in 2 years, meaning it doesn't support any of the new syntax in python 3.12:
- Support for PEP 695 (Type parameters / generics) MagicStack/MagicPython#262
- support python 3.12 type alias syntax MagicStack/MagicPython#265
it seems very odd that these new features came out 6 months ago and are still not supported properly in vscode.
there's also the issue with raw strings being incorrectly treated as regex which has been raised many times:
- Wrong highlight for raw string MagicStack/MagicPython#259
- Incorrect syntax coloring for formatted-raw string literals (example: fr"{some_path_here}\some_file_here.txt") MagicStack/MagicPython#149
- Python highlighting - raw string literals incorrectly colour escape sequences in some cases MagicStack/MagicPython#155
- Raw string highlighting MagicStack/MagicPython#114
- Github highlights escape sequences in raw strings. MagicStack/MagicPython#82
i think this is a big source of confusion for users (as you can see from how many times it's been raised), and should be addressed. i think Gilberto Agostinho (@gilbertohasnofb) makes a good argument in MagicStack/MagicPython#259 (comment):
AFAIK this is not a standard defined in any PEP, and popular code formatter such as black will convert any
R''strings intor''by default.I get the reasoning behind this decision, and there were some arguments for it since TextMate and Sublime Text both highlight Python syntax this way by default, but with the widespread use of VSCode these days (which does not highlight raw strings in Python this way by default) this argument seems outdated. Ideally this idiosyncratic highlighting could be made optional, preferably off by default.
Activity
rchiodo commentedon Apr 29, 2024
Thanks for the issue. We actually have an internal issue already tracking this: https://github.com/microsoft/pyrx/issues/4026, but we can use this issue as the external facing problem.
JelleZijlstra commentedon Apr 30, 2024
Just noting that this part isn't true; Black does not convert between
R''andr'', specifically because MagicPython syntax highlights them differently. I don't have a strong opinion on whether Pylance should continue to treat the two prefixes differently: on the one hand it's a surprising and non-obvious behavior, on the other hand it is nice to get a way to syntax-highlight regexes.DetachHead commentedon Apr 30, 2024
i see this issue was moved to the pylance repo. from my understanding the syntax highlighting provided by magicpython is bundled with vscode and separate to the semantic highlighting provided by pylance. is the plan to remove that basic syntax highlighting entirely and instead have it handled entirely by pylance? if so, it would be a downgrade for users who are not using pylance (eg. vscodium/basedpyright users)
rchiodo commentedon Apr 30, 2024
The plan is to fix the colors of text in Python code. We haven't really started investigating how or where we'd implement such a thing, but we realize the textmate solution that uses MagicPython's tmlanguage isn't really working out.
debonte commentedon Apr 30, 2024
VS Code is exploring using Tree-sitter for syntax highlighting. If they move forward with that, the current thinking is that each language could choose either TextMate or Tree-sitter depending on which implementation is more up-to-date / better maintained -- microsoft/vscode#210475
For internal people, there was a discussion about this work at the beginning of the 4/30 "Language Teams and LSP Weekly Sync" meeting.
heejaechang commentedon Apr 30, 2024
did we drop the idea of supplying textmate rule in pylance rather than using one supplied by vscode?
3 remaining items
JackismyShephard commentedon Oct 19, 2024
What is the ETA on getting these issues fixed? The no syntax highlighting on type definitions and generics is really a thorn in my side 😋
debonte commentedon Nov 5, 2024
I checked in with vscode on microsoft/vscode#210475. There is no extension contribution story yet.
Our hope is that using tree-sitter will greatly simplify our syntax highlighting approach (vs TextMate), and therefore we are postponing work on syntax highlighting until vscode's tree-sitter experiment is further along.
clambert-jt commentedon Jul 2, 2025
For the time being, manually patching in MagicStack/MagicPython#270 into my vscode's MagicPython.tmLanguage.json works for PEP695 highlighting. Maybe worth doing in vscode while waiting for tree sitter support (it's been nearly 2 years since python 3.12).
debonte commentedon Jul 2, 2025
clambert-jt, MagicStack/MagicPython#270 is a draft PR that has been open for more than 3 months with no comments from the repo owners. That, combined with the PR author's comments that, "I have only a tiny bit of an idea of what I'm doing. I didn't bother writing tests," don't make me particularly interested in taking this change into vscode.
It's good to know that users who are particularly interested in PEP 695 highlighting have the option of manually patching the
.tmLanguage.jsonfile though. Thanks.heejaechang commentedon Jul 9, 2026
Using this as the tracking issue for Python syntax-highlighting limitations that stem from TextMate/regex not being able to model Python's grammar (nesting, balanced delimiters, multiline constructs, verbose-regex semantics). We're postponing this class of fix until VS Code's tree-sitter-based highlighting matures (vscode#210475), which addresses them structurally.
Related reports folded in here: #7813, #5791, #4912.
rchiodo commentedon Sep 17, 2026
We're currently trying out an experiment that will replace the MagicPython tmlanguage colors with something new.
It works by using the
textDocument/semanticTokenrequest.Right now, semantic tokens are only returned for "interesting" tokens. This experiment instead returns semantic tokens for EVERY token in a file, therefore superseding the MagicPython textmate based tokens.
Internally it's using a tree-sitter parser (like VS code was looking at) and seems to be pretty fast.
It will be behind a flag to start.
{ "python.analysis.enableTreeSitterColorization": true }DetachHead commentedon Sep 17, 2026
nice. will this eventually make it to vscode or will it stay as a pylance exclusive?
rchiodo commentedon Sep 17, 2026
No, this is a pylance only implementation as we're doing it in the
textDocument/semanticTokenrequest handler. I'm not sure why VS code has not done more work on the tree-sitter stuff. Seems promising as a replacement for tmlanguage files.I'm hoping if this works it prompts them to pick that back up again. So maybe this is a proof of concept?
bschnurr commentedon Oct 9, 2026
This issue has been fixed in prerelease version 2026.4.100, which we've just released. You can find the changelog here: CHANGELOG.md