Overlapping definitions for tokens

[ivanperez-keera]

I'm writing a parser for a subset of mermaid diagrams using BNFC.

One of the features of this language is that the lexer is context-dependent. See, for example, the following three diagrams:

Diagram 1:

stateDiagram-v2
    s2 : This is a state description

Diagram 2:

stateDiagram-v2
    s1 --> s2

Diagram 3:

stateDiagram-v2
    s1 --> s2: A transition

The choice of whether s1 and s2 are an identifier or free text in this language is based on other tokens. For example, once you are in a state description (determined by seeing a state, and then seeing a ":" right after a state identifier), anything until the end of the line is a state description.

The mermaid parser handles this by using states in the grammar (it's a bit confusing, because they have nothing to do with the states in the state diagram as such). See for example https://github.com/mermaid-js/mermaid/blob/df636c6d0a127a5ce3270672499dd7e230ce33fc/packages/mermaid/src/diagrams/state/parser/stateDiagram.jison#L15-L142.

Can this kind of situation be handled in BNFC?