Switch to New Parser #55

iwillspeak · 2022-06-12T15:31:21Z

Bringing the new parser up to speed. Replacing all uses of the old parser with the new one, and
binning the old parser.

TODO:

iwillspeak · 2022-06-12T15:40:16Z

src/Feersum.CompilerServices/Compile/Compiler.fs

@@ -1234,7 +1234,7 @@ module Compilation =

        let ast, diagnostics =
            let nodes, diagnostics =
-                List.map Parse.parseFile sources
+                List.map LegacyParse.parseFile sources
                |> List.fold (fun (nodes, diags) (n, d) -> (List.append nodes [ n ], List.append d diags)) ([], [])

            { Location = TextLocation.Missing


We probably want to come up with a better way of modelling this. The previous parser was run on each input, and then we pasted them together into a single big SEQ. Instead I guess we should have a wrapper Compilation type, which contains a set of BoundCompilationUnits. We can bind them in sequence, and propagate some state (lib definitions etc.). The key is we don't want to have public items from one unit just become 'magically' available in another. This will probably want to be different for script compilations however. In that case we'd want multiple passes at compilation to share definitions from the previous.

We may want to handle this by just having the last compilation unit in the compilation be treated specially and allow definitions from that to leak into the root binder scope. That way inherited compilations from scripting would have access to the previous definitions.

src/Feersum.CompilerServices/Syntax/Parse.fs

src/Feersum.CompilerServices/Syntax/Tree.fs

iwillspeak · 2022-06-19T17:02:08Z

src/Feersum/ParseRepl.fs

+    ReadLine.Read("[]> ")
+    |> Parse.readProgram "repl.scm"
+
+let private print (result: ParseResult<Program>) =


Some kind of ast explorer which allows you to cd and ls around the tree might be nice.

test/Feersum.Tests/LexTests.fs

test/Feersum.Tests/SpecTests.fs

iwillspeak · 2022-06-23T09:28:25Z

https://gist.github.com/iwillspeak/6d87958184ea0062b0409f31f36d7131

iwillspeak · 2022-07-03T11:34:31Z

src/Feersum.CompilerServices/Binding/Binder.fs

@@ -6,6 +6,14 @@ open Feersum.CompilerServices.Diagnostics
 open Feersum.CompilerServices.Syntax
 open Feersum.CompilerServices.Utils

+module private BinderDiagnostics =
+
+    // TODO: remove this and replace with better binder errors


TODO: Distinct binder diagnostics.

iwillspeak · 2022-07-03T16:24:36Z

src/Feersum.CompilerServices/Binding/Binder.fs

+                ctx.Diagnostics.Emit
+                    BinderDiagnostics.bindError
+                    (List.last formals).Location
+                    "Saw dot but no ID in formals"


Dots should be handled by the parse, not the bind

iwillspeak · 2022-07-03T16:25:02Z

src/Feersum.CompilerServices/Binding/Binder.fs

+        match node.Kind with
+        | AstNodeKind.Ident i -> BoundDatum.Ident i
+        | AstNodeKind.Constant c -> BoundDatum.SelfEval(BoundLiteral.FromConstant c)
+        | AstNodeKind.Dot -> BoundDatum.Ident "." // FIXME: This is definitely not right


Fix me. Datum dots.

iwillspeak · 2022-07-03T16:39:06Z

src/Feersum.CompilerServices/Syntax/Tree.fs

+    inherit Expression(red)
+
+    member public _.OpeningParen =
+        red.ChildrenWithTokens()


Should / could lazies work here?

iwillspeak · 2022-07-03T22:00:20Z

src/Feersum.CompilerServices/Diagnostics.fs

+            if s.Line = e.Line then
+                sprintf
+                    "%s(%d,%d-%d): %s: %s"
+                    (s.Source |> normaliseName)


This breaks click through and problem matching in VS Code. Maybe Code needs a PR?

microsoft/vscode#154054

Compilations are now run on `CompileInput`s containing either a `Program` or `ScriptProgram`. This is then lowered to a legacy tree for binding but should allow compile API surface changes to be fleshed out. Some `eval` tests working directly with legacy trees may benefit from a syntax builder API.

Fixes a bug where trailing junk would not be identified as an error in parsing.

Sketching out the structure of syntax factories. These now need implementing for the remaining node types. Heavily inspired by `Teasel`'s `SyntaxFactory` work.

Factory for bool nodes.

Update the REPL to allow statements to cross lines.

Introduce factories for string and character values. This should mean we now have all the main constant types covered.

Bring back some of the `EvalTests` by adding in factories for our root program types.

Add factory support for quoted values.

Introduce the form and symbol factories, and use them to implement the remaining disabled `eval` tests.

Rather than relying on dup and storing the value on the statck while we perform the `#f` check we can instead store the temporary value in a local. This simplifies the codepath for the `true` case.

These files are getting unwieldly and unreadable when embedded in json. Update the assertions to read them from disk instead.

iwillspeak · 2024-06-23T07:38:42Z

spec/builtin-macros.ast

+    "
+    ATMOSPHERE: (98..162) ";;        exporting or re-exporting macro definitions currently."
+    ATMOSPHERE: (162..167) "
+    "


Do we need to teach Firethorn some escapes?

Upgrade to the latest versions of most packages. This includes a breaking change to the debug output of `Firethorn` so test needed re-generating too.

iwillspeak commented Jun 12, 2022

View reviewed changes