DAG Extension

[jonpry]

I am using PKL to create acyclic graphs. This is working well on the language side. However, attempting to serialize the results is cumbersome. This is in part because the export formats don't support references. In cases where the graph leads to a reduction, such as:

  name: String
}

class Foo {
  ref: Node
}

node = new Node {
   name = "Hello"
}

foo1 = new Foo {
   ref = node
}

foo2 = new Foo {
  ref = node
}

I am unable to determine that foo1.ref == foo2.ref. This happens even in Java where there is no serialization per se, just a marshalling of the VM objects into java objects.

As a workaround, I added a function to pkl::base to create UUID's.

  public abstract static class UUID extends ExternalMethod0Node {
    @Specialization
    protected String eval(VmTyped self) {
      return java.util.UUID.randomUUID().toString();
    }
  }

This works in that I can parse the graph, remove duplicate nodes, and restore the DAG structure before serialization. This is cumbersome in that it is possible to amend a node, preserving its UUID ,and yet the newly amended node is not equivalent to the original. Not to mention that PKL itself must be patched to support this because there otherwise no stateful functions with which to construct rand().

I propose that some infrastructure be added to support this use case. I am not very familiar with the code base, but it seems to me that VmValue could have a member that caches the Java type after marshaling, so if this same value is marshaled again, the same reference can be returned.

[jonpry]

I was able to get somewhere with this with the following patch:

diff --git a/pkl-core/src/main/java/org/pkl/core/runtime/VmTyped.java b/pkl-core/src/main/java/org/pkl/core/runtime/VmTyped.java
index f03fbc4..23aef51 100644
--- a/pkl-core/src/main/java/org/pkl/core/runtime/VmTyped.java
+++ b/pkl-core/src/main/java/org/pkl/core/runtime/VmTyped.java
@@ -20,7 +20,6 @@ import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary;
 import com.oracle.truffle.api.frame.MaterializedFrame;
 import org.graalvm.collections.EconomicMap;
 import org.graalvm.collections.UnmodifiableEconomicMap;
-import org.pkl.core.Composite;
 import org.pkl.core.PModule;
 import org.pkl.core.PObject;
 import org.pkl.core.ast.expression.unary.ImportNode;
@@ -150,17 +149,21 @@ public final class VmTyped extends VmObject {
 
   @Override
   @TruffleBoundary
-  public Composite export() {
+  public Object export() {
+    if (exported != null) return exported;
     if (!isModuleObject()) {
-      return new PObject(clazz.getPClassInfo(), exportMembers());
+      exported = new PObject(clazz.getPClassInfo(), exportMembers());
+      return exported;
     }
 
     var moduleInfo = getModuleInfo();
-    return new PModule(
-        moduleInfo.getModuleKey().getUri(),
-        moduleInfo.getModuleName(),
-        clazz.getPClassInfo(),
-        exportMembers());
+    exported =
+        new PModule(
+            moduleInfo.getModuleKey().getUri(),
+            moduleInfo.getModuleName(),
+            clazz.getPClassInfo(),
+            exportMembers());
+    return exported;
   }
 
   @Override

Then by modifying pkl-cli to write an ObjectOutputStream of the evaluator.evaluate(moduleSource).rawValue, I obtained a java serialized output that preserves equality in the tree.

This patch fails a number of tests. I'm not sure if there are mutexes that need to be held or if some assumption ends up being broken by this. The 'as' operator needs a lot of work from what I can tell. There are issues with subclasses that make the output types of the tree incorrect, so I did not attempt to use it.

Instead, I load the java serial object into python and a short script decrustifies the PModule,PObject etc to get a clean object tree made out of dataclasses which is then pickled. I then load this pickle into my javascript application for visualization. An interesting hodgepodge of languages and serialization formats :). It works, and these binary formats are so much easier to deal with than text when references are involved.

[bioball]

Have you thought about rendering the node's name instead here? You can even create a converter that turns references into their names.

Something like this:

output {
  renderer = new JsonRenderer {
    converters {
      [Node] = (it) -> it.toMap().put("ref", it.ref.name)
    }
  }
}

It's important that all Pkl APIs are deterministic, and referentially transparent. Therefore, we're not likely to introduce a UUID API. But if we do, it would be essentially a linked list type of thing generated from a seed.

We are considering how we can make it possible to render references, but it's a little too early to even pseudocode what that might look like.