Self-modifying AST

[ovalnine]

I'm trying to write a program that can self-modify its AST.

# eval.janet

(def input (parse-all (slurp "read.janet")))
(forever
  (eval input))

# read.janet

(def current 1)
(var answer @"")
(getline (string "Change current " current " to: ") answer)
(array/remove input 0)
(array/insert input 0 '(def current (string/trimr answer)))

The idea is to have an evaluator program that takes a janet file and parses it generating the AST.
Then it evaluates the program that reaches out from itself to get the AST and update the value.

In the example above I just get the input binding that stores the AST and update the value.

This example is obviously very simplified, but what I want to eventually do is to have a way to interactively modify the file from inside the executing program.

Does this seem like the right approach?
Is there a better way for self modifying the AST?

[iacore]

What's the purpose of modifying AST? You can modify data.

[ovalnine]

I'm trying to figure out a way to build something like this: https://glisp.app/commit:e7fbaae/

It's not so much about modifying the data dynamically, more about the actual modification of the AST.

Maybe there is a different approach to achieve this.

[sogaiu]

Not quite following but possibly of interest is the dynamic variable *redef* / :redef. See this bit of the changelog for a bit more on it.

This gets turned on via janet -d as well (though that also turns the debugging functionality on too).

[iacore]

@ovalnine the app you linked is very flaky
one semicolon breaks it

[ianthehenry]

Janet doesn't have native concrete syntax trees so it will be difficult to do the sort of bidirectional sync that you see in the demo you linked.

That said, you can do an approximation of it with the AST by reserializing it whenever you change previews and such. But any time you edit the "preview" values, it would lose some formatting in the text display. Janet does store position information in tuple forms, but not in any other forms.

But you're really parsing the AST -> modifying it as a regular data structure, with functional updates since the AST is mostly immutable -> serializing that back into a form.

I don't think Janet has built-in functional update helpers but you can implement the ones that you need:

repl:1:> (defn update-at [t i f] [;(tuple/slice t 0 i) (f (t i)) ;(tuple/slice t (+ i 1))])
<function update-at>
repl:2:> (update-at (parse "(+ 1 2)") 1 inc)
(+ 2 2)
repl:3:> (string/format "%q" _)
"(+ 2 2)"

[ovalnine]

Thanks, that sounds like what I was looking for, a native CST that I could take advantage from. So it looks like I'll have to do the hard work myself. There is probably a way to build some form of CST using the parser module.

I think building the update helpers won't be too hard, I'll probably build some macro so the updates are not explicit. Or I can use the parser/insert function to add those update calls to the AST at specific marks.

[ianthehenry]

Maybe worth mentioning: some AST nodes have some position information associated with them. All (round-parens) and [square brackets] have a [line col] position associated with them that you can get with tuple/sourcemap. You can then reverse out the byte position and form length from those, assuming that you only have unix line endings... but it might be easier to modify the parser itself to produce concrete syntax nodes.

[sogaiu]

For some scenarios I think it's practical to use pegs for parsing and that allows one to get position information for other constructs.

janet-usages-as-tests uses a peg for parsing Janet code [1] to get this kind of information (but not limited to tuples) and makes use of it for reporting location information for failed tests [2].

Performance might be an issue for some kinds of projects though.

---

[1] If interested, there is more of that type of thing here.

[2] As certain # (line comments) are basically being repurposed within (comment ...) forms as a sort of indicator of a test, the location of things that look roughly like # => needed to be determined. AFAIU, that's not available via parse and friends because comments are discarded.

[ovalnine]

This is all very useful, I think if one wants to start building a custom parser for building the CST, PEGs are definitely more practical. For more performance and extensibility building a custom module by forking the native parser is a great idea.

Another idea I had was to use my editor (neovim) and use the RPC/API to query from the tree-sitter plugin. I'm not sure how far fetched the idea is, but my original idea was already to use my own editor and its environment instead of building one. I'll do a little bit of research to see if this might be possible. I'm not familiar with emacs, but it would be interesting to know if something similar is possible in it as well.