feat: implement Pratt parsing

[39555]

Closes: #131

This is the initial implementation of the operator precedence parser based on the Pratt algorithm https://en.wikipedia.org/wiki/Operator-precedence_parser#Pratt_parsing.

The interface:

precedence(
    digit1.try_map(|d: &str| d.parse::<i32>()),
    (
        "-".value(2).prefix(|x| -1 * x),
        "+".value(2).prefix(|x| x),
        "!".value(2).postfix(|x| factorial(x)),
        "+".value(0).infix(|a, b| a + b),
        "-".value(0).infix(|a, b| a + b),
        "*".value(1).infix(|a, b| a * b),
        "/".value(1).infix(|a, b| a / b),
    ),
)
.parse_next(i)

Any parser that returns a number can be considered as operator with its binding power. To convert the parser into prefix, infix or postfix, we use the extension trait on top of the impl Parser.

Implementation details

Binding power, associativity, Pratt and Dijkstra

The Pratt algorithm only uses the binding power. Left and right associativity are used in Dijkstra's Shunting Yard algorithm. (It seems likechumsky incorrectly names its implementation as pratt while using shunting yard internally). EDIT: I am wrong here. We need to use associativity for infix operators to determine the order of the operators with equal binding power.

I named the combinator precedence with aliases to pratt, shunting yard, precedence climbing, separated. If any of the names are confusing, we may consider better alternatives.

The implementation is partially inspired by https://gist.github.com/ilonachan/3d92577265846e5327a3011f7aa30770.
But it doesn't use any Rc<...> allocations for closures; instead, it uses uses & RefCell<& dyn ...> references.

Operators from the tuple are statically dispatched by the requested affix so prefix.as_postfix() will return fail while prefix.as_prefix() with return the actual parser. This may have a performance issue because it iterates through the whole tuple each time the parser is requested for any affix:

See the trace:

> postfix                                                                                      | "+4"∅
 > opt                                                                                         | "+4"∅
  > alt                                                                                        | "+4"∅
   > fail                                                                                      | "+4"∅
   < fail                                                                                      | backtrack
   > fail                                                                                      | "+4"∅
   < fail                                                                                      | backtrack
   > "!"                                                                                       | "+4"∅
   < "!"                                                                                       | backtrack
   > fail                                                                                      | "+4"∅
   < fail                                                                                      | backtrack
   > fail                                                                                      | "+4"∅
   < fail                                                                                      | backtrack
   > fail                                                                                      | "+4"∅
   < fail                                                                                      | backtrack
   > fail                                                                                      | "+4"∅
   < fail                                                                                      | backtrack
  < alt                                                                                        | backtrack
 < opt                                                                                         | +0
< postfix

(I tried to make a some sort of a linked list where each parser holds an index to the next one with the same affix but, it quickly became a nightmare of lifetimes, complexity, and compilation errors so it was rejected).

Performance

A quick criterion benchmark with a simple input &str 1-2*4+12-561-5-6*6/9-3+1*-2*4-758*3 shows:

winnow   time: [191.25 ns 191.56 ns 191.94 ns]
chumsky time: [335.63 ns 335.93 ns 336.42 ns]

TODO

• Add more tests from here https://gist.github.com/ilonachan/3d92577265846e5327a3011f7aa30770 and here https://github.com/rust-bakery/nom/pull/1362/files
• Implement a full example with ternary and other fun operators
• Performance benchmarks for the example
• Documentation with examples
• Implement the api for Vec<..> and &[...]
• Streaming support
• Hide behind a feature flagunstable-pratt

EDIT:

Things to consider

• Precedence uses i64 but may overflow due to operations like precedence + 1 and precedence - 1. Consider replacing + and - with checks against the Assoc enum instead.

[epage]

Wow, this looks great. You put a lot of work into this and I see care is taken in cases like the handling of infinite loops and such

[epage]

I like names to make their role in the grammar clear. I'm trying to decide how this does and if there is anything better. We use separated for this kind of thing but unsure how to tie it in without it being overly verbose

[39555]

Maybe expression? Similar to how combine names this parser https://docs.rs/combine-language/latest/combine_language/fn.expression_parser.html

[epage]

Oh interesting, I wasn't aware of that crate. We should at least add expression_parser to our aliases.

We likely should pick one and deal with it and rename it if a better name comes up. expression seems just as good as any other. It fits with how this would be used in a language grammar.

[epage]

Our rule of thumb is grammar-level concepts are standalone functions and value processing are trait functions. These seem grammar-level.

[39555]

Thanks for reviewing! Do you suggest making prefix, infix.. standalone functions? I followed the concern in the attached issue to make <parser>.prefix(...) work:

So I assume we'd go with

let calc = pratt(
    digits1.map(Expr::Int),
    (
        '-'.prefix(Right(1), |r| unary(r, Op::Neg));
        '+'.infix(Left(0), |l, r| binary(l, r, Op::Add));
        '!'.prefix(Right(3), |r| unary(r, Op::Fact));
    )
);

These prefix, infix, postfix are strange beasts in this implementation. They are both grammar-level constructs and value processing functions because the data processing is required by the parser to advance further.

I though about the interface like prefix(0, "-").map(|(a, b)| a - b). It looks clean but what should be the default operator when we don't call map. For a unary it is easy -- just return the argument, but for a binary it is unclear how to merge operands.

[epage]

Ha! I previously looked at this and viewed the associativity as not grammar level and now looked at it and viewed it as grammar level. Normally trait methods are reserved for purely operating on the data, like map. This is actually changing how we parse the results. The one weird middle ground we have is cut_err. Unsure if this should be treated like that or not.

I also feel like a bunch of free functions floating around might not be the easiest for discovery and use.

I also suspect the ultimate answer will be dependent on the answer to other questions as I alluded to the interplay of my comments in #614 (comment)

[epage]

The list of parsers will need to be updated

[epage]

Don't know if aliases with spaces works

[epage]

I tend to prefer that impls be closely organized with their trait

[epage]

We are dumping a lot of stray types into combinator. The single-line summaries should make it very easy to tell they are related to precedence (maybe be the first word) and somehow help the user know what, if any, they should care about

[epage]

So we find the right "mode" by falling through an alt of parsers for that mode and then get the weight and fold function from the first success.

Looking at the example

precedence(
    digit1.try_map(|d: &str| d.parse::<i32>()),
    (
        "-".value(2).prefix(|x| -1 * x),
        "+".value(2).prefix(|x| x),
        "!".value(2).postfix(|x| factorial(x)),
        "+".value(0).infix(|a, b| a + b),
        "-".value(0).infix(|a, b| a + b),
        "*".value(1).infix(|a, b| a * b),
        "/".value(1).infix(|a, b| a / b),
    ),
)
.parse_next(i)

the alt seems unfortunate for performance reasons. dispatch can be used in this case and I suspect a lot of other cases. It would be a big help to find a way to set this up so dispatch can be used as well, e.g.

precedence(
    digit1.try_map(|d: &str| d.parse::<i32>()),
    dispatch!{any;
        '-' => prefix(2, |x| -1 * x),
        '+' => prefix(2, |x| x),
        '!' => postfix(2, |x| factorial(x)),
        '+' => infix(0, |a, b| a + b),
        '-' => infix(0, |a, b| a + b),
        '*' => infix(1, |a, b| a * b),
        '/' => infix(1, |a, b| a / b),
        _ => fail,
    ),
)
.parse_next(i)

(note: I might give other suggestions that run counter to this, we'll need to weigh out each one and figure out how we want to balance the different interests)

[39555]

Another idea. The implementation in nom rust-bakery/nom#1362 uses 3 parsers one for each affix in the function signature. It has an advantage that you can use dispatch and there is no prefix, postfix.. functions. The downside is that from just the function signature it's hard to remember which parser is which. And also fail is used if the parser is not provided.

Maybe abuilder pattern would be ideal e.g:

precedence(digit1.parse_to::<i32>())
    .prefix(dispatch!{any; 
         "+" => empty.map(|_| move |a, b| a + b)
         "-" => empty.map(|_| move |a, b| a + b)
   })
   .postfix(dispatch!{any; ...})
   .infix(dispatch!{any; ...})
   .build()
.parse_next(i)

Methods are similar to repeat().fold. This approach would eliminate stray functions, reduce the performance impact of iterating over each parser for each affix, and allow users to use dispatch or any custom logic they want.

I’m not sure yet how this would interact with binding power. I'm going to research whether we can drop the explicit index argument and rely on array indexing, how binding power interact between affixes .etc.

[39555]

I came across another idea. Instead of dispathing by the parser type maybe we could dispatch by the input type:

enum Input<'i, I>{
   Prefix(&'i mut I),
   Postfix(&'i mut I),
   Infix(&'i mut I)
}

then it would be possible to (I'm not sure yet if Deref would work or not)

dispatch!{any;
        '-' => prefix(2, |x| -1 * x),
        '+' => prefix(2, |x| x),
        '!' => postfix(2, |x| factorial(x)),
        '+' => infix(0, |a, b| a + b),
        '-' => infix(0, |a, b| a + b),
        '*' => infix(1, |a, b| a * b),
        '/' => infix(1, |a, b| a / b),
        _ => fail,
    )

then prefix parsers would do

if let Prefix() = input {
   ...
} else {
  fail
}

[epage]

This is nice because we only run the needed parsers (less alt, no dispatch just to get an error).

I agree about the opacity of parameter order. Its also not great if a parameter is unused (e.g. no postfix).

If we go with the "builder" approach, we are again violating the API design guidelines as discussed in #614 (comment). However, it does make it nice for naming the parameters and only requiring what is needed.

Maybe one way of looking at this is the error reporting. What happens if I use an operator in the wrong location?

[epage]

Not quite getting the input type and dealing with custom input types in the middle of a parser adds a lot of complexity.

[epage]

Organizationally, O prefer the "top level" thing going first and then branching out from there. In this case, precedence is core.

[epage]

Streaming support

This looks to be agnostic of streaming support like separated is

[epage]

Hide behind a feature flagunstable-pratt

If this is going to start off unstable, then its fine noting most of my feedback in the "tracking" issue and not resolving all of it here

[epage]

Is there a way we can do this without recursion? Parser authors are generally cautious around recursion due to blowing up the stack.

Hmm, seems all of the other ones use recursion

[epage]

From #614 (comment)

While looking at it I found a potential error. My current implementation is unsound. Consider the 2 + 4 + 7. In the current parser it evaluates as (4 + 7) + 2 instead of (2 + 4) + 7 when operators have equal binding power . That is why the algorithm uses associativity. While + or * are commutative someone might want to parse function calls for example

[39555]

Another difference from the #618 algorithm is that this recursive one backtracks when the left binding power of an operand is less than the current right binding power. It could affect the performance if parsing the operators is expensive.

 1     +     2     +     3
    1     2     1     2
          -     -
-> "1 + 2 +"   ~^ 1 < 2
-> "1 + 2"
-> "3"             +

The #618 always stacks all parsed operators into a vector.

[39555]

Operator callbacks should be able to return impl Parser or receive the input in order to provide ability to kick additional parsing. For example to parse an index operator, a function call, or a ternary operator. How this is done here https://gist.github.com/ilonachan/3d92577265846e5327a3011f7aa30770#file-nom-pratt-parser-rs-L198

fn ternary(cond: Expr) -> ... {
  move |i: &mut Input| {
        let (a, b) = separated_pair(pratt_parser , ':' , pratt_parser).parse_next(i)?;
        Ok(Expr::Ternary(cond, a, b)
  }
}
fn pratt_parser() -> ... {
    precedence(
        ...,
        trace("postfix", '?'.value((5, ternary))),
        ...
    )
}

[39555]

Updated with the new API.

• Without _: &mut _
• Without as _

use Infix::*;
expression(0, digit1.parse_to::<i32>())
    .prefix(dispatch! {any;
        '+' => Prefix(12, |_, a| Ok(a)),
        '-' => Prefix(12, |_, a: i32| Ok(-a)),
        _ => fail
    })
    .infix(dispatch! {any;
       '+' => Left(5, |_, a, b| Ok(a + b)),
       '-' => Left(5, |_, a, b| Ok(a - b)),
       '*' => Left(7, |_, a, b| Ok(a * b)),
       '/' => Left(7, |_, a, b| Ok(a / b)),
       '%' => Left(7, |_, a, b| Ok(a % b)),
       '^' => Right(9, |_, a, b| Ok(a ^ b)),
       _ => fail
    })
    .parse_next(i)

[epage]

Should start_power also be a "builder" method?

• Only used in very specific cases
• Can better focus documentation on it

[epage]

Also, would referring to it as precedence or precedence_level be a more approachable user-facing term?

[39555]

start_precedence_level or min_precedence_level, allowed_precedence ? This is needed for if the binding power of the operator < start_precedence the parser stops parsing. This is exposed to allow parsing a subset of an expression such as a ? b : c, 1 + 2 (parsing stops at ,)

[epage]

When a user sets it, is it the same precedence level they set for their operator? If so, I'd use current_ as the prefix

[epage]

For these, how come we don't do

Suggested change

	#[inline(always)]
	pub fn prefix<NewParsePrefix>(
	self,
	parser: NewParsePrefix,
	) -> Expression<I, O, ParseOperand, NewParsePrefix, Post, Pix, E>
	where
	NewParsePrefix: Parser<I, Prefix<I, O, E>, E>,
	{
	Expression {
	start_power: self.start_power,
	parse_operand: self.parse_operand,
	parse_prefix: parser,
	parse_postfix: self.parse_postfix,
	parse_infix: self.parse_infix,
	i: Default::default(),
	o: Default::default(),
	e: Default::default(),
	}
	}
	#[inline(always)]
	pub fn prefix<NewParsePrefix>(
	mut self,
	parser: NewParsePrefix,
	) -> Expression<I, O, ParseOperand, NewParsePrefix, Post, Pix, E>
	where
	NewParsePrefix: Parser<I, Prefix<I, O, E>, E>,
	{
	self.parse_prefix = parser;
	self
	}

[39555]

Different types of the Expression due to the different type of the new parser