Soft Parser: Initial implementation of SoftParser

[simerplaha]

Overview

• Towards Parser errors prevent CodeProvider execution (completion, goto definition etc) #104.
• Run Demo.scala to test the following properties of SoftParser.

Syntax error-insensitive

SoftParser always generates an AST, even if the code contains syntax errors. This ensures that LSP APIs (jump-to-definition, code completion etc) are always available.

• Multiple Errors: The following screenshot is for demo only. It shows how multiple errors can be generated by a single parser run. These errors are required for "Code Actions" (no plans to replace errors reported by ralphc). It's also not tested yet, this version of the parser is only tested to ensure that an AST is always available.
  
• Unknown Tokens: Unknown syntax tokens are stored as Unresolved in the Syntax Tree.

  Unresolved(code = 🚀, index = SourceIndex(113,2,None))

Order-insensitive

The order of the code is not enforced. For example, the following code is parseable:

fn function() -> Bool {
   Contract MyContract() { }

   fn nestedFunction() -> ABC
}

Releases & Incremental implementation

Being error-insensitive allows SoftParser to be implemented and integrated incrementally. For example, the following is valid Ralph code, but let bool = true is parsed as Unresolved by SoftParser because it does not recognise this syntax yet.

fn function() -> Bool {
  let bool = true
}

• This ensures that any future changes to Ralph syntax in Node will not delay the release of ralph-lsp with the latest Node version.
• This also allows SoftParser to used by jump-definitions, code-completion etc, even though it is only partially implemented.

Convert SoftAST to Code

SoftAST can always be converted back to the original string code using the toCode() method call:

val code: String = ast.toCode()

toCode() can be also be invoked on any sub-tree, resulting in code just for that tree.

See Demo.scala.

Formatting

• Towards Formatter & Linter #236
• Since the AST can be converted code, text-document formatting requests can be processed by updating the SoftAST.Space instances within an AST.
• SoftParser does not enforce strict ordering. For example, fn function() is parsable even when it is not embedded within a Contract or TxScript. This allows the implementation of on type formatting.

AST as String

Invoke the following to generate a String tree representation of the AST.

val tree: String = ast.toStringTree()

See Demo.scala.

Next steps

• Pending documentation.
• Instead of implementing formatting as a new feature, I think we should integrate this parser into the currently implemented APIs, so that existing features are complete before implementing on new features. WDYT?

[tdroxler]

Instead of implementing formatting as a new feature, I think we should integrate this parser into the currently implemented APIs, so that existing features are complete before implementing on new features. WDYT?

Sound's good to me

[tdroxler]

I translated lexeme to see what was the french world: lexème
.
.
.
Now I add to google the definititon of lexème 😅 So thx to you I learnt a new word.

[tdroxler]

Amazing work

[simerplaha]

Lexème sounds better in French 😆