read performance

[alandipert]

read is currently very slow, and likely the slowest part of program loading. Currently, reading core.gk results in 100+ subshells - probably because we subshell in readline and perform various tricks in order not to lose whitespace.

Any boost in read performance while retaining the ability to read whitespace characters is killer.

[abrooks]

How dow you feel about a dependency on awk? Perl?

[joelmccracken]

Awk is a posix standard.

Sent from my iPhone

On Nov 28, 2013, at 5:06 PM, Aaron Brooks [email protected] wrote:

How dow you feel about a dependency on awk? Perl?

—
Reply to this email directly or view it on GitHub.

[quoll]

Perl may not always be available. One of the reasons for Gherkin is for those environments where you don't know what you may have (Bash 4 is a bit of an ask, but ultimately every new system should have it by default).

Gherkin is already based on AwkLisp. It seems to me that if we're happy being dependent on Awk, then we would be better served by simply extending that project (which would be MUCH easier to do).

The readLine function is currently calling out to external programs (grep, tail, tr), which is NOT what we want... and it makes the reader very slow. However, no one has quite figured out how to avoid this yet (one attempt was close, but couldn't deal with * characters, IIRC). Other than that one line, I don't think we rely on external programs anywhere.

[alandipert]

@kanaka has made incredible progress in his experimentation with tokenization instead of character pushback. While our "string-mapped" file reader is about 40% faster than the interactive read-based one, Joel's current approach could bring us another 2x and also scales linearly with input. For latest progress, see his tokenizer.sh