Skip to content

[antlr4, maven testing] Fix for #4569: predefined.tokens interfering with testing. #4570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 6, 2025
Merged

[antlr4, maven testing] Fix for #4569: predefined.tokens interfering with testing. #4570

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9984caf
Fix #4569
kaby76 Jul 30, 2025
a473a06
Remove dup for TOKEN_REF. Choose first entry;
kaby76 Jul 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 32 additions & 3 deletions antlr/antlr4/ANTLRv4Lexer.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,45 @@ options {
// Using a predefined list of tokens here to ensure the same order of the tokens as they were defined
// in the old ANTLR3 tree parsers (to avoid having to change the tree parsers code).
// The actual values of the tokens doesn't matter, but the order does.
tokenVocab = predefined;
// tokenVocab = predefined;
//
// Instead of declaring predefined.tokens, which messes up the Maven tester for grammars-v4,
// we define the total order of token type values. This should work because it was stated
// that the "actual values of the tokens doesn't matter, but the order does." Declaring them
// in the tokensSpec section preserves the total order.
}

// Insert here @header for lexer.

// Standard set of fragments
tokens {
TOKEN_REF,
ACTION,
ARG_ACTION,
ARG_OR_CHARSET,
ASSIGN,
LEXER_CHAR_SET,
RULE_REF,
LEXER_CHAR_SET
SEMPRED,
STRING_LITERAL,
TOKEN_REF,
TOKEN_REF,
UNICODE_ESC,
UNICODE_EXTENDED_ESC,
WS,
ALT,
BLOCK,
CLOSURE,
ELEMENT_OPTIONS,
EPSILON,
LEXER_ACTION_CALL,
LEXER_ALT_ACTION,
OPTIONAL,
POSITIVE_CLOSURE,
RULE,
RULEMODIFIERS,
RULES,
SET,
WILDCARD
}

channels {
Expand Down
1 change: 0 additions & 1 deletion antlr/antlr4/predefined.tokens
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ RULE_REF = 9
SEMPRED = 10
STRING_LITERAL = 11
TOKEN_REF = 12
TOKEN_REF = 13
UNICODE_ESC = 14
UNICODE_EXTENDED_ESC = 15
WS = 16
Expand Down
Loading