Parallelize parsing. #25
Description
This can be an interesting idea to keep in mind for the future. I have already experimented with single-consumer-single-producer queues in order to have a token-producing and a token-consuming queue, but the performance is lack-luster, even when using state-of-the-art data structures and algorithms.
A possible approach here would be to have a 2-layered parsing approach: one would divide the input with a really simple grammar, and the other would parallelly parse these divisions.