RFC: Make white-space handling less confusing / more consistent with the introduction of an "adjacent selector": `,`

[Kroc]

Summary

Make white-space handling less confusing / more consistent with the introduction of an "adjacent selector": ,

Motivation

• The rules soi, eoi, white-space & comment look like any other rule, and with whitespace and comment being names that users are extremely likely to use themselves, the hidden behaviour can be confusing, unexpected and even impractical for the user's desire

• The implicit behaviour of the ~ selector only when a whitespace rule is defined, is sometimes not what the user wants and they have to resort to the @ modifier even though this effects the entire rule and not just one selector. This can make it very difficult to achieve certain effects without resorting to nested rules

Guide-level explanation

Adjacent elements in a rule can be separated by a ",":

rule = { a, b, c }

Where rule a is followed by rule b which is followed by rule c. No white-space is assumed between the rules, though note that any rule can itself contain a whitespace rule.

Contrast this with:

rule = { a ~ b ~ c }

Which will check for optional white-space and comments between rules a, b & c.
The adjacent (,) and white-space selector (~) can be combined in a rule; this can be used to carefully control the automatic assumption of white-space or comments.

rule = { a, b ~ c }

The behaviour of the adjacent selector remains consistent within nested rules making use of modifiers.
In the below example, the behaviour of rule a is unaffected by the parent rules utilising different white-space moderators.

rule1 = @{ a }
rule2 = ${ a }
rule3 = !{ c }

a = { b, c }

Reference-level explanation

I can't comment on the inner workings of Pest, as I haven't even completed my first ever Rust program yet, but based on writing parsers in other languages (VB6, Go, Perl6), the adjacent selector should provide minimum difficulty. It doesn't add functionality, and expresses a natural state of one token following another.

Drawbacks

• Even with the addition of such a simple feature there is still the cost of implementation, testing, documentation and compatibility

• There could be unforeseen consequences with parsing behaviour with complex combinations of features; adding another feature increases the available complexity

Rationale and alternatives

This is the simplest possible design to resolve a need of adjacent selection without changing the behaviour of existing selectors (for compatibility). The choice of actual character used (,), the name, the terminology etc, can be debated.

In some languages (e.g. PHP, some functional languages) a dot is used for concatenation:

alternative = { a . b . c }

This may be desirable over a comma as it is more explicitly a "operator" between words, where as a comma could be confused for something more general and peppered by the user in places where it shouldn't be.

Adjacency could be communicated without the use of a character where one rule separated from another rule by white-space implies adjacency; e.g.:

alternate = { a b c }

Whilst this form exists in BNF and some derivatives, the use of an explicit separator avoids potential parse-errors or unintentional behaviour from the user and also provides visual balance as every rule is always separated in all cases, regardless of separator.

Alternative options include changing the functionality of existing features, such as how the "~" selector or "@" modifier operates.

The automagic behaviour of rules with specific names is a documentation / support / learning hindrance, but I believe that that can be resolved separately outside of this RFC as they will have much more drastic implications.

The proposed feature adds to the project, provides benefits, whilst also not taking away from existing features. At the cost of additional documentation, it may help users avoid issues starting out.

Prior art

The use of a comma for adjacency is present in EBNF, which Pest roughly follows.

Unresolved questions

?

[Kroc]

Added . alternative. In some ways I actually prefer this over the comma, but additional symbols may want to be reserved for future functionality.

[wirelyre]

👍

The context-dependent meaning of ~ is troublesome. In some cases, it can even mean different things in a single rule:

double_meaning = { "a" ~ "b" } // when used directly, matches whitespace
indirect = @{ double_meaning } // when used indirectly, doesn't match whitespace

whitespace = { " " }

As an extension, I would consider changing ~ to always mean "match whitespace", and introducing a second operator to always mean "don't match whitespace". Then atomic modifiers would relate only to which rules appear in the output.

[dragostis]

Unfortunately, this doesn't take into consideration the other combinators like + or * which would also need a way to deal with the WHITESPACE issue. A solution would need to be proposed for these as well.

Apart from the drawbacks already mentioned, there's also the fact that there is some duplication of functionality when WHITESPACE is not defined, i.e. ~ and , do the same thing. How would one explain this? Of course, it might make sense to have ~ fail if WHITESPACE is not defined.

For the particular case of ~, maybe it makes to have it defined as a rule ~ = { " "* }?

Another alternative we could take into consideration is to separate non-whitespace from the other modifiers and have it work inside of the rules.

rule = { @(a ~ b*) ~ c }

Above, only the last ~ will accept WHITESPACE. It also probably makes sense to keep the cascading effect as well so that one doesn't have to write @ too often.

[Kroc]

For the purposes of this RFC, and to avoid going off on a bike-shedding tournament, we should avoid discussion of how best to change / replace the implicit whitespace rule / ~ here; that belongs in its own RFC, and I will probably do that in about a weeks time unless anybody else wants to go ahead and do that.

This RFC is for the discussion of an "adjacent selector" (I don't know if "selector", "operator" or "combinator" is the correct term for these things, it's never stated in the documentation). If we get into discussing alternate strategies for the whitespace rule, we'll literally never get this simple feature request in.

As Pest currently stands for this RFC, the ~ will operate differently depending on the rule modifier of that rule or any parent rule. This is still useful for some purposes. Again, alternate designs for ~ / whitespace should be discussed on a different RFC.

(sorry to be nagging, but I just want a simple adjacent operator as soon as possible, and worry about the whitespace weirdness after that)

[dragostis]

I agree that this could lead to a really long and possibly not as productive discussion, but the issue here is that adding an "adjacent selector" would only add complexity to the grammar without removing whitespace-dependent behaviour of atomic rules, since ~ operators would still function the same complicated way.

While I do agree that the current design may not be the most intuitive, I think that the best way forward would be to only change the grammar when absolutely needed. Maybe a good idea would be to implement this as a prototype, or simply argue a bit more about the case where adding this features plays well with all the other features and is a good addition by itself.

And don't get discouraged! The issue your brought up very much exists and I'm excited to find a solution together.

[wirelyre]

I have to admit that I'm very excited about this RFC, because I think the discussion could potentially make pest syntax more consistent and easier to learn.

In that spirit: another option would be to introduce a modifier to the operators themselves to indicate that they take optional whitespace. Something like a ~& b and ( expr )+&.

That looks a bit ridiculous, now that I see it written out. Clearly I've been using too much J.

[dragostis]

@wirelyre, I'm really excited about this too. I feel like this and pest_deconstruct will make pest an amazing parser. The issue here is that it's hard to find a syntax that is simple enough.

[CAD97]

This is sort of co-opting the RFC, but bringing this up in led to a discussion around Pest's whitespace handling and I came up with a proposal overhaul of how Pest handles whitespace that streamlines behavior.

To summarize:

• a b is the strict sequence of a and b
• a | b is the ordered choice of a then b
• a ~ b is the whitespace/comment-tolerant sequence of a and b
• &a is the positive predicate
• !a is the negative predicate
• a? is the repetition of a zero-or-one times
• a* is the whitespace/comment-tolerant repetition of a zero-or-more times
• a+ is the whitespace/comment-tolerant repetition of a one-or-more times
• $(a*) is the strict repetition of a zero-or-more times
• $(a+) is the strict repetition of a one-or-more times
• _(a) omits a but produces a's children
• @(a) produces a but omits a's children (transitively)

Those twelve rules are enough to define all of the matching behavior (excluding discussion of builtins). You can @ me on Gitter if you want to discuss this idea further, but I'm documenting it here.

I think that draft is both actionable and covers all of the expressiveness desired. (Interestingly, ~ behaves essentially as a normal sequenced rule here.) The only issue I really see rising up might be associativity definitions. Instead of using lack of a separator for sequence as in "pure" PEG, we could also use any of ,.:#%^- and retain the simplicity of this model, just with an explicit "strict sequence" operation and "loose sequence" operation instead of adjacency sequencing and ~ just being another adjacent production.

This is just speculative by me at 3 AM, though, so take it just as that -- an idea put onto the floor for discussion.

[CAD97]

I can't find the discussion, but I'd like to record what I do remember from discussion here:

Consensus was that having a "pure adjacency" available is very useful, and should definitely be added. - was suggested and no better alternatives were proposed.

, can't be used, as parameterized rules want to use , to separate parameters. (They could use ; instead, but having , allowed but not separate parameters in this position would be confusing.)

There are no specific plans on using . that would preclude its use for this purpose, but - is beneficial in that it appears more "binary operator"-like, and a visual similarity to ~ suggests a similar purpose. This also brings in a connotation that ~ is - but "roundabout" or "loose".

a - b ~ c would then expand to the functional equivalent of a - b - (WHITESPACE|COMMENT)* - c.

In order to allow repetitions to be strict, I believe the winning proposal was to treat it somewhat like how the repetitions are handled in Rust macros: rule-* would be strict repetition, rule~* would be loose repetition, and rule* would continue to be loose repetition.

---

Changes not part of this RFC's motivated changes, but on the table for consideration for a 3.0:

• Changing @ away from the current definition and instead just suppressing child rule emission
• Changing ~'s behavior dependence on @
• Linting use of ~ when neither WHITESPACE or COMMENT are defined
• Removing ~'s implicit behavior based on WHITESPACE and COMMENT, instead defining the trivia by assigning to it directly
• Changing WHITESPACE and COMMENT's intrinsic atomicity

---

A very vague outline of how to go about adding this - "strict adjacency" operator for 2.1:

• Adjust the grammar to allow for strict versus loose adjacency and repetitions
• Adjust the AST to allow for strict versus loose adjacency and repetitions
• In the processing passes done on the grammar, loose adjacency/repetition transforms into strict adjacency/repetition with whitespace/comment rules added as required

[dragostis]

I very much like introducing these rules as "experimental" for 2.1 with possibly an opt-in flag. That should make the transition to 3.0 much leaner.