Escape-handling in string literals

[burakemir]

Once string literals are parsed (by the ANTLR generated parser,) we need to process the string a bit before turning it into a string constant. This happens in the parse/parse.go VisitStr

This currently only does some basic processing of escaped quote characters.

When we turn strings constants into a string representation again in ast/at.go String(), it would be good if there was an String() method that produced output which could be parsed again.

String support was originally added for basic strings (think: words), but for a stable release we'd want to have proper support for escapes, and also support for multi-line strings (already in grammar).

[cpendery]

@burakemir Happy to try to contribute support for this

[burakemir]

Here is a proposal for a spec, taken from bigquery https://cloud.google.com/bigquery/docs/reference/standard-sql/lexical
Here is the CEL implementation https://github.com/google/cel-go/blob/master/parser/unescape.go

In a first step, one could leave out octal, hex and unicode escapes, and ignore the String() debug output function.

\a	Bell
\b	Backspace
\f	Formfeed
\n	Newline
\r	Carriage Return
\t	Tab
\v	Vertical Tab
\\	Backslash (\)
\"	Double Quote (")
\'	Single Quote (')
\`	Backtick (`)
\ooo	octal escape, with exactly 3 digits (in the range 0–7).
\xhh	Hex escape, with exactly 2 hex digits
\uhhhh	Unicode escape, with lowercase 'u' and exactly 4 hex digits. range D800-DFFF is not allowed
\Uhhhhhhhh	Unicode escape, with uppercase 'U' and exactly 8 hex digits.range D800-DFFF and values > 10FFFF not allowed

[rkrishnasanka]

I'm wondering there's a reason why this project is using the Vistor vs the Listener pattern. I've mainly used the listener pattern to date, and it was very easy to process inner-loops.

[burakemir]

Thanks for engaging on our first issue! There is a spec part and an implementation part.

I think we'd want escapes for the whitespace codes and unicode escapes, as in the CEL spec
https://github.com/google/cel-spec/blob/master/doc/langdef.md#string-and-bytes-values

Multi-line strings come with their own twist. The Bigquery (GoogleSQL) spec does offer multi-line literals https://cloud.google.com/bigquery/docs/reference/standard-sql/lexical

With proper support for \n and unicode escapes we already have enough to turn multi-line strings back into literals.

Regarding the use of visitors in ANTLR, I had replaced my hand-written parser with ANTLR in a hurry and the visitor way of using ANTLR was what I was familiar with. It may be worth to revisit, but I believe it would not make a difference for this particular issue, as we would want string literals to be tokens.

[burakemir]

Let me try convert this issue to a (freshly enabled) GitHub discussion.