How to amend in a for generator

[djarnis73]

Hi

I'm brand new to pkl, but saw the talk at devoxx antwerp and was immediately in love. I'm trying to use the tool to autogenerate a lot of configuration files that we are currently writing manually with a lot of copy/pasting.

One need I have come across is to amend an object in a for generator.

If I try the following:

hidden template {
  name
  body = """
      This refers to the name [\(name)] which must be set by amending the template later on
    """
}

root {
  for (n in List("a", "b")) {
    x = (template) {
      name = n
    }
  }
}

I get the following error:

A for-generator cannot generate object properties (only entries and elements).

Is there a way for this to work? I tried to keep the above example as simple as possible, my real scenario is a bit more complex.

[HT154]

Can you show an example of what you'd like the output to look like here, given that input? I'm almost certain there's a solution for what you're trying to do, but I want to be sure to answer the question you're actually asking!

[djarnis73]

Yeah I also feel confident that it can be done, but I'm probably still missing the flair to express it the kpl way.

I have the following module (actual question it in the toValues function):

module Application

name: String
deployment: Deployment
// lots of fields and defaults has been removed for brevity

class Deployment {
  targets: Listing<String>
  monitors: Listing<DatadogMonitor>
}

class DatadogMonitor {
  application_name_with_deploy_target: String = "<default service name should always be amended>"
  name: String
  description: String
  prodPriority: DatadogMonitorPriority
  nonProdPriority: DatadogMonitorPriority = "P3"
  query: String? // expression that queries metrics to set the state of the monitor
  // more fields left out for brevity
}

typealias DatadogMonitorPriority = "P1"|"P2"|"P3"

function toValues(target: String): Dynamic = new {
  datadog {
    monitors {
      for (monitor in deployment.monitors) {
        let (app_name_with_target = "\(name)-\(target)")
        // how can I amend the monitor here to set its application_name_with_deploy_target
        // to the value of app_name_with_target before calling toMap

        // convert the monitor (that contains two priority values into one with a single value depending on if we are currently
        // in a prod or non-prod deployment
        monitor.toMap()
          .put("priority", if (target == "prod") monitor.prodPriority else monitor.nonProdPriority)
          .remove("prodPriority").remove("nonProdPriority")
          .toDynamic()
      }
    }
  }
}

output {
  files {
    for (target in deployment.targets) {
      // generate kubernetes valus file for each deployment
      ["deployment/\(target)_values.yaml"] {
        value = toValues(target)
        renderer = new YamlRenderer {}
      }
    }
  }
}

Which I use as:

amends "ApplicationTemplate.pkl"

name = "my-sample-app"
deployment {
  targets {
    // this application also has staging besides defeault test and prod targets
    "prod"
    "staging"
    "test"
  }
  monitors {
    new {
      name = "\(application_name_with_deploy_target) : My customapplication monitor"
      description = """
          Desription of how operations should react when this monitor goes into the alarm state
          Needs to be able to contain the application name including deploy target.
          ie my-sample-app-prod
        """
      prodPriority = "P1"
      nonProdPriority = "P2"
    }
  }
}

Then I generate files with pkl eval -m . MyApplication.pkl.

It can also be cloned from here: https://github.com/djarnis73/pkl-playground/tree/main/ApplicationExample

It is still a simplified version of a larger effort where I try to generate a lot of different files (dependabot.yml, .drone.yml, kubernetes values and template files, and spring boot config files and probably a few more once I get the hang of it).

[HT154]

Gotcha, that makes a lot of sense. What you're doing is an application of the Class-as-a-function pattern!

I took a few minutes to digest your example and revise it a bit in a way that solves your immediate problem and here's what I came up with:

module Application

name: String
deployment: Deployment
// lots of fields and defaults has been removed for brevity

// this is purely for IDE autocomplete, you can add any common/frequent target names here but still accept any string
typealias Target = "prod"|"staging"|"test"|String

class Deployment {
  targets: Listing<Target>
  monitors: Listing<DatadogMonitor>
}

class DatadogMonitor {
  application_name_with_deploy_target: String
  name: String
  description: String
  priority: DatadogMonitorPriority
  hidden priorities: Mapping<Target, DatadogMonitorPriority> = new {
    default = (_) -> "P3"
  }
  query: String? // expression that queries metrics to set the state of the monitor
  // more fields left out for brevity
  
  function render(deploymentName: String, target: String): DatadogMonitor = (this) {
    priority = priorities.getOrNull(target) ?? priorities.default.apply(target)
    application_name_with_deploy_target = "\(deploymentName)-\(target)" // I _think_ this was your intent
  } 
}

typealias DatadogMonitorPriority = "P1"|"P2"|"P3"

output {
  files {
    default {
      renderer = new YamlRenderer {}
    }
    for (target in deployment.targets) {
      ["deployment/\(target)_values.yaml"] {
        value = new Dynamic {
          datadog {
            monitors = new Listing { // <--- this is where your actual issue was I think, it's a Dynamic if the explicit type isn't given
              for (monitor in deployment.monitors) {
                monitor.render(name, target)
              }
            }
          }
        }
      }
    }
  }
}

Usage is very similar still:

amends "ApplicationTemplate.pkl"

name = "my-sample-app"
deployment {
  targets {
    // this application also has staging besides defeault test and prod targets
    "prod"
    "staging"
    "test"
  }
  monitors {
    new {
      name = "\(application_name_with_deploy_target) : My customapplication monitor"
      description = """
          Desription of how operations should react when this monitor goes into the alarm state
          Needs to be able to contain the application name including deploy target.
          ie my-sample-app-prod
        """
      priorities {
        default = (_) -> "P2"
        ["prod"] = "P1"
      }
    }
  }
}

A few notes on what I did here:

• The "union type of strings and String" approach can really turbo charge IDE experiences for open-ended properties that have a small set of known common values
• Using a hidden Mapping for priorities is a significantly more flexible here and removes the need to round-trip through Map during rendering
• I'd prefer a method on DatadogMonitor to encompass any "rendering" logic specific to that class instead of a module-level method that has to track how to render all the different nested types.
• In an ideal world you'd have a fully typed value for your output values so you wouldn't have to worry about mixing explicit typing into a Dynamic structure.

Hope this gives you a clearer direction of how to achieve what you're looking for!

[djarnis73]

Wow, that's elegant. Does the self amend construct/pattern you use in the render function (this) have a name? I did not find it mentioned in the documentation.

I agree on all points (and love the open ended property thing).

One goal I did not mention was for the using of my library to be as dead simple as possible (to get an easier sell to the team I'm working with). So I think I prefer to avoid having the need for constructs like:

priorities {
    default = (_) -> "P2"
    ["prod"] = "P1"
}

It can be a fallback in case the flexibility is needed (but I may change my mind once I introduce my proof of concept to the team depending on how they react).

Thanks a lot for the detailed answer, very much appreciated.

[holzensp]

That construct is called an amends expression. The fact that it amends this is not "special," compared to amending any other object. Since Pkl is lazy, you can define objects recursively (just don't try to render the recursive definition 😃).

As one minor nit on @HT154's excellent (as per usual) suggestion; default = (_) -> "P2" is moot. A Mapping's default is only used when amending a new object and Strings can't be amended, so there is no way to use this default.

When I see a function that is only used inside a template (here: Application.DatadogMonitor.render is only used in Application.DatadogMonitor.output.files), my spidey-senses tingle. I would use a straightforward amends expression, in this case:

module Application

name: String
deployment: Deployment
// lots of fields and defaults has been removed for brevity

// this is purely for IDE autocomplete, you can add any common/frequent target names here but still accept any string
typealias Target = "prod"|"staging"|"test"|String

class Deployment {
  targets: Listing<Target>
  monitors: Listing<DatadogMonitor>
}

class DatadogMonitor {
  application_name_with_deploy_target: String
  name: String
  description: String
  priority: DatadogMonitorPriority
  hidden priorities: Mapping<Target, DatadogMonitorPriority>
  hidden defaultPriority: DatadogMonitorPriority // <-- This still feels off; two `hidden` properties to express a degree of freedom that's probably not necessary
  query: String? // expression that queries metrics to set the state of the monitor
  // more fields left out for brevity
}

typealias DatadogMonitorPriority = "P1"|"P2"|"P3"

output {
  files {
    default {
      renderer = new YamlRenderer {}
    }
    for (target in deployment.targets) {
      ["deployment/\(target)_values.yaml"] {
        value = new Dynamic {
          datadog {
            monitors = new Listing {
              for (monitor in deployment.monitors) {
                (monitor) { // <-- No need for a function; every monitor just needs to get its priority and application_name_with_deploy_target amended
                  priority = priorities.getOrNull(target) ?? defaultPriority
                  application_name_with_deploy_target = "\(name)-\(target)"
                }
              }
            }
          }
        }
      }
    }
  }
}

That said, though... this can be defined "a priory" because Pkl is lazy. That is to say; we can "just" define these relationships in defaults in the Application template and change them if so desired. This suggestion assumes that Deployment can be changed (and doesn't involve other Listing properties that must be varied per Target). Furthermore, it assumes the users can be assumed to understand (some) Pkl. So, grain of salt, but I'd say:

Application.pkl:

module Application

name: String
deployment: Deployment = new {
  targets {
    default { target ->
      default {
        application_name_with_deploy_target = "\(module.name)-\(target)"
      }
    }
  }
}
// lots of fields and defaults has been removed for brevity

// this is purely for IDE autocomplete, you can add any common/frequent target names here but still accept any string
typealias Target = "prod"|"staging"|"test"|String

class Deployment {
  targets: Mapping<Target, Listing<DatadogMonitor>>
  // fixed targets: Listing<Target> = monitors.keys.toListing() // <- if you want `targets` to exist
}

class DatadogMonitor {
  application_name_with_deploy_target: String
  name: String
  description: String
  priority: DatadogMonitorPriority
  query: String? // expression that queries metrics to set the state of the monitor
  // more fields left out for brevity
}

typealias DatadogMonitorPriority = "P1"|"P2"|"P3"

output {
  files {
    default {
      renderer = new YamlRenderer {}
    }
    for (target, _monitors in deployment.targets) {
      ["deployment/\(target)_values.yaml"] {
        value = new Dynamic {
          datadog {
            monitors = _monitors
          }
        }
      }
    }
  }
}

and then using it as:

amends "Application.pkl"

name = "my-sample-app"
deployment {
  targets {
    default { target ->
      new {
        name = "\(application_name_with_deploy_target) : My customapplication monitor"
        priority = if (target == "prod") "P1" else "P2"
        description = """
          Desription of how operations should react when this monitor goes into the alarm state
          Needs to be able to contain the application name including deploy target.
          ie my-sample-app-prod
          """
      }
    }
    ["prod"] {}
    ["staging"] {}
    ["test"] {}
  }
}

The key point, here, is that you don't have to "wait until you know the targets." There are many ways in which you can assume you already know targets, even if you don't, and write code accordingly. That's the benefit of laziness.

[holzensp]

This does still involve default { target -> in user code, which might be less than ideal, but I'd say you don't even need the logic to determine the priority; you could just set it:

amends "Application.pkl"

name = "my-sample-app"
deployment {
  targets {
    default {
      new {
        name = "\(application_name_with_deploy_target) : My customapplication monitor"
        priority = "P2"
        description = """
          Desription of how operations should react when this monitor goes into the alarm state
          Needs to be able to contain the application name including deploy target.
          ie my-sample-app-prod
          """
      }
    }
    ["prod"] {
      priority = "P1"
    }
    ["staging"] {}
    ["test"] {}
  }
}

This is generally the more idiomatic approach; describe all variations you want for a thing and, per such thing, only describe what makes that thing different.

[HT154]

Does the self amend construct/pattern you use in the render function (this) have a name?

Just an amends expression! I guess you might consider the larger pattern (method returning an amended this) as similar to the builder pattern.

A Mapping's default is only used when amending a new object and Strings can't be amended, so there is no way to use this default.

I've actually become a fan of explicit default usage in cases where the key isn't present in the mapping. My example code does use this, but not via Pkl's amending mechanism:

priority = priorities.getOrNull(target) ?? priorities.default.apply(target)

This is certainly not always ideal, but it really comes in handy on occasion!

Agreed that if you can push this up into the definitions of the targets themselves, this less-explicit behavior disappears: a target is defined by its name and exactly how it differs from others / the default. For me, what @holzensp showed here is the Pkl happy path, but I've found that this doesn't always produce the most concise or usable interfaces when the value(s) being overridden are deeply nested and/or numerous.

One tool that I've found really helpful to address that is Mixins. If you have sets of values that are amended together often (eg. "set all monitor priories to X"), you can write a function that accepts the priority value and produces a Mixin that sets the priority of every monitor in a target to the given value. Mixins can be chained so they're a really nice way to build composable mods on top of some base configuration.