Repository navigation
3.12+: TokenError no longer raised for some invalid source (mismatched braces?) #105714
Description
Activity
Unfortunately, this seems like one of those where we won't be able to preserve the previous behavior.
Tokenizing invalid pieces of code has been undefined (although undocumented) since forever, but we now have a warning explicitly mentioning just that since 3.11. There's also many different stakeholders pushing in opposite directions (code quality tools towards this being preserved, inspect & IPython/sympy profit from this being able to tokenize correctly).
Sorry we can't be more helpful than this.
@lysnikolaou reverting the patch above restores the behaviour -- I definitely think this is doable and easy
This is doable, yes. However, without this patch, multiple other things break, one of which is
inspect, on which a lot of other things depend likehypothesisin the case of the issue that triggered this.Pleasing both sets of stakeholders would require the C tokenizer to recover from errors and then raise at the end, which would be a much bigger undertaking and would burden us with more maintenance than we're comfortable with.
Reacted by Zachary WareReacted by anthony sottile
this comes from the pycodestyle testsuite -- I expect the tokenizer to raise an error for this however it seems to silently accept it now.
Bug report
yep, this is just a single curly brace
Your environment