Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New line characters treated wrong in Lexer #70

Open
lonely-lockley opened this issue May 12, 2024 · 0 comments
Open

New line characters treated wrong in Lexer #70

lonely-lockley opened this issue May 12, 2024 · 0 comments
Labels
possible-bug Could be a bug

Comments

@lonely-lockley
Copy link

lonely-lockley commented May 12, 2024

Hi! Unfortunately, I didn't save the exact output to help you reproduce the bug, but briefly: in the grammar with predicates checking char position in line (currentTokenColumn property of Lexer) calculated wrong.
In grammar like this:

fragment Nl           : ('\r'?'\n' | '\n')  ;
fragment Ws           : (' ' | '\t' | '\u000C') ;

EOL              : { this.currentTokenColumn > 0 }? Nl -> channel(HIDDEN) ;
EMPTY_LINE       : { this.currentTokenColumn == 0 }? Nl -> skip ;
BLANK            : { this.currentTokenColumn > 0 }? Ws+ -> channel(HIDDEN) ;
INDENTATION      : { this.currentTokenColumn == 0 }? Ws+ -> channel(HIDDEN) ;

NEWLINE_INDENT          : EOL BLANK* INDENTATION ;

and input like '\n\n ' the BLANK and INDENTATION rules never trigger because currentTokenColumn is calculated as if all those tokens are on the same line. I expected that each EOL would increase line number and reset currentTokenColumn to zero. In Java ANTLR4 implementation it works this way.

antlr4ng version 3.0.4
antlr4ng-cli version 2.0.0

@lonely-lockley lonely-lockley changed the title New line characters treated wrong New line characters treated wrong in Lexer May 12, 2024
@mike-lischke mike-lischke added the possible-bug Could be a bug label May 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
possible-bug Could be a bug
Projects
None yet
Development

No branches or pull requests

2 participants