Enable token-based rules on source with syntax errors #11950
+1,028
−148
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dhruvmanila
force-pushed
the
dhruv/token-rules-with-syntax-errors
branch
from
b7134c9
to
7db979b
Compare
dhruvmanila
commented
Jun 28, 2024
Comment on lines
-96
to
97
| for token in tokens.up_to_first_unknown() { | ||
| for token in tokens { | ||
| pylint::rules::invalid_string_characters( |
There was a problem hiding this comment.
This is looking at string tokens and the lexer doesn't emit them if it's unterminated. So, we might get away with not doing anything in this case for now.
dhruvmanila
commented
Jun 28, 2024
Comment on lines
24
to
29
| impl<'a> DocLines<'a> { | ||
| fn new(tokens: &'a Tokens) -> Self { | ||
| Self { | ||
| inner: tokens.up_to_first_unknown().iter(), | ||
| inner: tokens.iter(), | ||
| prev: TextSize::default(), | ||
| } |
There was a problem hiding this comment.
This extracts a specific set of comments so it doesn't require any specific update.
dhruvmanila
commented
Jun 28, 2024
Comment on lines
256
to
260
| _ => { | ||
| kind => { | ||
| if matches!(kind, TokenKind::Newline if fstrings > 0) { | ||
| // The parser recovered from an unterminated f-string. | ||
| fstrings = 0; | ||
| } |
There was a problem hiding this comment.
I think this should work as the newline tokens within f-strings are actually NonLogicalNewline, I'll move this into TokenIterWithContext. I'll test this a lot because f-strings are complex.
dhruvmanila
force-pushed
the
dhruv/token-rules-with-syntax-errors
branch
from
7db979b
to
1961406
Compare
dhruvmanila
commented
Jun 28, 2024
Comment on lines
+110
to
-115
| for token in tokens { | ||
| match token.kind() { | ||
| TokenKind::EndOfFile => { | ||
| break; | ||
| } | ||
|
|
There was a problem hiding this comment.
The token stream doesn't contain the EndOfFile token.
|
dhruvmanila
force-pushed
the
dhruv/token-rules-with-syntax-errors
branch
from
6e96839
to
f3bbacd
Compare
dhruvmanila
force-pushed
the
dhruv/token-rules-with-syntax-errors
branch
from
f3bbacd
to
7b42997
Compare
dhruvmanila
added a commit
that referenced
this pull request
Jun 28, 2024
This PR reverts #12016 with a small change where the error location points to the continuation character only. Earlier, it would also highlight the whitespace that came before it. The motivation for this change is to avoid panic in #11950. For example: ```py \) ``` Playground: https://play.ruff.rs/87711071-1b54-45a3-b45a-81a336a1ea61 The range of `Unknown` token and `Rpar` is the same. Once #11950 is enabled, the indexer would panic. It won't panic in the stable version because we stop at the first `Unknown` token.
dhruvmanila
force-pushed
the
dhruv/token-rules-with-syntax-errors
branch
from
7b42997
to
eeb24b1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR updates the linter, specifically the token-based rules, to work on the tokens that come after a syntax error.
For context, the token-based rules only diagnose the tokens up to the first lexical error. This PR builds up an error resilience by introducing a
TokenIterWithContextwhich updates thenestinglevel and tries to reflect it with what the lexer is seeing. This isn't 100% accurate because if the parser recovered from an unclosed parenthesis in the middle of the line, the context won't reduce the nesting level until it sees the newline token at the end of the line.resolves: #11915
Test Plan