Templating File Systems in Pkl

[ghyatzo]

Hello,

I am exploring how to define a nested structure, which, at a certain point, has an element that is a copy of the object itself (except the said element), let me put that in code:
The desired structure would be in the end this

obj1 {
      element1
      element2
      obj1_mirror {
          element1
          element2
          // I don't want obj1_mirror itself here.
      }
}

but of course, just calling

obj1 {
      element1
      element2
      obj1_mirror = (this) {}
}

incurs (rightly so) into an infinite recursion.
How can I go on about this? Will this require some form of reflection?
Or is there a way to exclude obj1_mirror from the amending?

[HT154]

A few clarifying questions:

• Is obj1 a Dynamic? Or something Typed? The mix of elements and properties in your snippets are a little confusing—it would be odd to mix them even with Dynamic.
• Do you need obj1_mirror to be readable in your Pkl module? Or do you just need it in the output?
• Can you give a full example of your expected output (in your desired format) given that input?

[ghyatzo]

I generalized the problem in the wrong way. I apologize.

obj1 is dynamic. The elements in the example were just a very poor choice of words given we are talking about pkl. In reality they are (most often empty) just other dynamic objects:

obj1 {
      inner1 {}
      inner2 {}
      obj1_mirror {
          inner1 {}
          inner2 {}
          // I don't want obj1_mirror itself here.
      }
}

What I am trying to do is defining a folder structure template, which presents some forms of self-similarities in certain spots.

• each object is a folder
• files are a property like name = extension

So, Given the (idealized) input of

obj1 {
      inner1 {}
      inner2 {}
      obj1_mirror = (this[Not(obj1_mirror)]) {}
}

I would like to obtain a

obj1 {
      inner1 {}
      inner2 {}
      obj1_mirror {
          inner1 {}
          inner2 {}
          // I don't want obj1_mirror itself here.
      }
}

after fiddling a bit (I always do it after asking the question, silly me) I've ended up with something that kind of works:

local innergroup = {
    inner1 {}
    inner2 {}
}

obj1 = (innergroup) {
    obj1_mirror = (innergroup) {}
}

I am not completely satisfied with this approach since I would need to clearly recognize and define all the innergroup objects in the file system I am trying to template. Since it the real case there are many cases where two subfolder only differ by a few files and there are a lot of different variations it would require either many different kind of innergroups or trying to find the least common divisor elements and then amend those with the differences, but both approaches feel pretty brittle and hard to maintain.

But if there is a fancier way of excluding specific elements/properties/entries from the amending process, some kind of "managed self referentiality", I'd be very happy to know it!

[holzensp]

This all still goes to more detail about how you want to use it, as per @HT154's questions. For example, it makes a difference whether you want it for rendered output, or for Pkl code accessibility. If it has to be Dynamic - which is generally somewhat of a smell - you can do this:

obj1 {
  inner1 {}
  inner2 {}
  obj1_mirror = (this) { obj1_mirror = null }
}

but that does give you a obj1_mirror = null. If you use a renderer that has omitNullProperties = true, that goes away again, so it doesn't matter. If it's for use from Pkl code and it's important that there is no property obj1_mirror at all, you might be able to do

obj1 {
  inner1 {}
  inner2 {}
  obj1_mirror = this.toMap().remove("obj1_mirror").toDynamic()
}

but you should be mindful that this destroys some laziness and dynamic bindings. There's some work underway to introduce first-class deletion, but it's not fully decided that will happen.

All of this becomes a lot easier when you can define it as a class, because then you can define obj1_mirror as hidden, which automatically excludes it from rendering (and from things like toMap).

That said, I have a sneaking suspicion that needing obj1_mirror might be an instance of the XY Problem.

[ghyatzo]

Thank you for your explanation.
I also believe that this might be an instance of the XY Problem, but on one side I asked X very much from a "language discovery" perspective, so the problem Y was instrumental to learn more about pkl. On the other, I didn't want to directly ask Y because that would mean offloading my work on to you which I don't find kind nor appropriate, so the ill-posed question X was an attempt to avoid that.
I apologize for the confused question.

The need is to output a template for a folder structure. I am going at it with a bottom up approach, I started by defining a generic folder-like structure and then started thinking on how to generalize and generate variations of it based on configuration. That is why everything is still Dynamic (and ill defined). This template (JSON) will then be read by another program to actually create the folders and populate them with various files, but that is out of scope of pkl in general.

The obj1 is one example of the kind of weird structures of the file system I am trying to generate; where obj1_mirror is a subfolder that contains a copy of the parent folder. obj1_mirror does not need to be necessarily readable somewhere else in the pkl module.

The obj1_mirror = null trick is super interesting, thanks!

[HT154]

@holzensp's advice here is spot on!

Filesystem modeling is tough in Pkl in my experience because of the lack of recursive typealiases. Here's how you can go about it (today, and with currently-not-possible ideal solution in comments):

typealias Directory = Mapping<String, Any?>
// in a hypothetical future where recursive typealiases exist
// typealias Directory = Mapping<String, (*Directory|String)?>

rootDir: Directory = new {
  ["obj1"] = new Directory { // in that hypothetical future, this can be just `["obj1"] {`
    ["inner1"] = "inner1 content"
    ["inner2"] = "inner2 content"
    ["obj1_mirror"] = (this) {
      ["obj1_mirror"] = null
    }
  }
}

output {
  files {
    ...flattenDir(rootDir, new Listing {})
  }
}

function flattenDir(dir: Directory, path: Listing<String>): Mapping<String, String> = new {
  for (filename, dirEntry in dir) {
    when (dirEntry is String) {
      [(path) { filename }.join("/")] = dirEntry
    }
    when (dirEntry is Mapping<String, Any?>) {
//  or with recursive typealiases
//  when (dirEntry is Directory) {
      ...flattenDir(dirEntry, (path) { filename })
    }
  }
}

Evaluating this (with -m <base> set) would result in a directory with this structure:

<base>
\- obj1
   |- inner1 # content: "inner1 content"
   |- inner2 # content: "inner2 content"
   \- obj1_mirror
      |- inner1 # content: "inner1 content"
      |- inner2 # content: "inner2 content"

[ghyatzo]

Sorry for the late reply,
This is incredible! Thank you.

I couldn't get it to work at first, I had to change the return value of flattenDir to Mapping<String, FileOuput> and then set the text property to dirEntry

[(path) { filename }.join("/")] { text = dirEntry }

but then It works like a charm!
I can even think of just reading stuff from online resources and "pasting" them in now. I was opting for parsing the pkl output and do the generation from another program, but I don't think I need to do that anymore!!

Indeed recursive typealiases would be the cherry on top.

[holzensp]

I apologize for the confused question.

No need for apologies and if my mentioning of the XY Problem sounded in any way like a rebuke; it's me who should apologize. It's just that I find myself in XY Problems every once in a while and find it helpful when it's pointed out ;)

I understand that it's often ergonomic for fast prototyping to avoid too much type information, but I find Pkl's types so pleasantly lightweight that I actually use them in the prototyping cycle. @HT154's suggestions are again solid; a few notes there;

typealias Directory = Mapping<String, Any?>

It's true that without recursive typealiases, this can't be defined very precisely, but I would avoid immediately bailing to Any; Mapping<String, *Mapping|String> at least allows to do some local as Directory casting to regain autocomplete. Still less than ideal, admittedly.

Depending on how you intend to use/hold these FS models (especially how you instantiate them), classes might still be helpful here. The downside, is that you have extra levels of indirection, but the upsides include that a) you can define them recursively and b) you can define some logic like the path computations in an earlier stage (so that paths are available in Pkl code also). This does lead to some more verbose code (like the mirroring requiring explicit corrections of path properties.

[ghyatzo]

It didn't sound like a rebuke, it was actually helpful hearing it.
I've included your suggestion of not using Any? thanks for the pointer.

At the moment I don't plan any particular complex or dynamical manipulation with these file systems. After defining them correctly with all the conditional logic implemented they should be pretty static and generated through an eval -m. Later steps will require more complex operations within these file systems but I'll be outside of pkl already.
After all the discoveries in this discussion, though, I'll try and see how much I can reasonably do within pkl.

Thanks again, you have been beyond helpful. I'll actually change the title of the discussion so that it's less cryptic and maybe someone else can find it useful!