Code generating C headers/structs from PKL - fixed length Listings, values-as-types

[mexlez12345]

I'm trying to use pkl to generate some C headers (that define the types) and C sources (that define values of those types) from pkl types using the reflect module.

I can successfully generate definitions for basic structs and primitive types (and fixed-width integers), but Listings/arrays and arbitrary external identifiers are causing trouble because information that I need to define types in C (Listing/array length, the C type of an external identifier) are values in pkl (Listing length, literal C type is a String in a struct).

Ideally what I'd like to write is something like this:

// pkl
class MyConfig {
  fixed_len_array: Listing<UInt32, 3>
}

foo: MyConfig = new {
  fixed_len_array = new {
    1
    2
    3
  }
}

To get output like this:

// output .h
struct MyConfig {
  uint32_t fixed_len_array[3];
};

// output .c
#include "my_config.pkl.h"

struct MyConfig foo = {
  .fixed_len_array = {
    1,
    2,
    3,
  },
};

I can successfully generate variable-length arrays with a length field, but this approach isn't ideal because VLAs are a non-standard extension, they can only appear once per struct at the end, and sizeof() and friends behave unexpectedly.

struct MyConfig {
  size_t fixed_len_array_len;
  uint32_t fixed_len_array[];
};

I can constrain the length of a Listing, but that constraint seems to only be visible for pkl values, not types.
When I use reflect to traverse the module, I can't access the type constraint to figure out how long the declared array should be.

class MyConfig {
  fixed_len_array: Listing<UInt32> (length == 3)
}

External references (ex. a function pointer, externally declared C types) have a similar problem. I can define a class that contains information about the identifier (C type name, what header to include), but I can't access that information when reflecting over the declarations, because the information is part of a pkl value and not a type:

typealias HeaderFile = String(endsWith(".h") || endsWith(".hh"))

hidden abstract class ExternalIdentifier {
  header_include: HeaderFile
  name: String
  type: String
}

hidden class MyCallbackFunctionType extends ExternalIdentifier{
  type: String ()
}


class MyStruct {
  // Basic type check of C types
  function_pointer: ExternalIdentifier(type == "int (*\(name)) (int, int, int)")
}

an_instance: MyStruct = new {
  function_pointer = new ExternalIdentifier {
    header_include = "path/to/foo.h"
    name = "my_function_pointer"
    type = "int (*\(name)) (int, int, int)"
  }
}

Any ideas? Is this the kind of thing that I'd have to implement natively in pkl to support? I could potentially ex. export to json and then generate C from there, but that feels like it defeats most of the point of using pkl.

[holzensp]

This is an interesting question. It's true that reflect does not give you access to the constraints on a type. Really, reflect only works on values. We've been discussing a syntax module (or similar) that would allow Pkl to read .pkl files and produce syntax trees as Pkl structures. This might help you, as long as you can require some strict limitations to what constraints are supported for your code generation (e.g. length == <constant>, but would you also support <constant> == length? What about length.isBetween(0,10)?). The problem with constraints in Pkl is that they are general functions, which means that they're not (necessarily) statically analysable. This means that you can only generally evaluate them when you're considering concrete values, i.e. at "run-time" (scare-quotes, because Pkl doesn't have any other "time").

Is this the kind of thing that I'd have to implement natively in pkl to support?

It's not (yet) obvious to me how you could/would, besides creating a wholly new category of types, where you'd allow (literal? const?) values in type positions.

I see three options right off the bat.

1. Generate pointers

You could use the traditional C approach to variable length arrays; pointers. Let's take a look at an example Pkl definition that would be incompatible with VLAs (because multiple Listings)

// my_config.pkl
foo: Listing<String>
bar: Listing<Int>

would generate

struct my_config {
  size_t len_foo;
  char **foo;
  size_t len_bar;
  long long *bar;
}

This does require that you implement your malloc/free dynamic memory management and that my_config will not be entirely "hosted" on your stack.

2. Define FixedWidth structures in Pkl

Alternatively, you could define some bespoke types in Pkl. Whether this is a reasonable solution depends on the wider context in which you find yourself. You could define, for example:

class FixedWidthListing {
  width: UInt
  value: Listing(length == width)
}

This creates an extra level of indirection on the definition side and - because Pkl doesn't have user-level generics - it loses type information for the elements. Also, that extra level of indirection is carried over in any rendered output (if you're also using Pkl's rendered output besides mapping config to your generated C code). You can work around these things;

// my_config.pkl
hidden foo_definition: Listing<String>
fixed foo: FixedWidthListing = new {
  width = 3
  value = foo_definition
}
hidden bar_definition: Listing<Int>
fixed bar: FixedWidthListing = new {
  width = 42
  value = bar_definition
}

output {
  renderer {
    converters {
      [FixedWidthListing] = (it) -> it.value
    }
  }
}

3. Require naming convention

Slightly cleaner (but, again, depending on wider context) than defining bespoke structures in Pkl is to use a naming convention. If you require that for every foo: Listing<Bar> there exists a sibling property const len_foo: UInt, you can use that in Pkl to retain valuation and in your codegen to get the information you need;

// my_config.pkl
foo: Listing<UInt32>(length == len_foo)
const len_foo: UInt = 3

and you can then generate

struct MyConfig {
  uint32_t foo[3];
};

Naming conventions are hard to reliably "police," though.

[bioball]

Another approach is to use a known typealias. E.g.

typealias Listing10<T> = Listing<T>(length == 10)

myValues: Listing10<String>

When you reflect upon myValues's type, you can compare it to known typealiases. You can take a look at how we do it in our go bindings: https://github.com/apple/pkl-go/blob/main/codegen/src/internal/typegen.pkl

[mexlez12345]

Thanks for the thoughts folks! I looked into option (1) the pointer approach; the downsides for my use case are that:

1. My target platform is resource-constrained and not allowed to use the heap
2. I'd like to construct all of my structs in-place with all their values filled out; having a pointer type would require an extra step of declaring the pointed-to value and assigning the pointer (either in pkl or hand-written in C later)

Options 2 & 3 looked promising so I dug a bit deeper on that and I think by combining them I have a solution that meets my needs without needing naming conventions!

Not sure how pkl-ey my approach is yet...

I define a abstract class FixedLengthListing with a const fixed_length field.
The const fixed_length is key here because what it lets me do is construct an instance of the type while I'm generating C structs, and access the fixed_length field to declare my array's length.

// Mostly a "tag" so I can identify which class definitions/objects should be generated to C vs. which are "internal" pkl types
// `hidden` doesn't address this need because `hidden` seems to apply only to values, not types
abstract class Struct {
}

// Fixed length listing; the `const fixed` `fixed_length` field can be accessed at codegen time
abstract class FixedLengthListing {
  const fixed fixed_length: UInt
  data: Listing(length == fixed_length)
}

// Add a specific length/type to the FixedLengthListing
class MyArray extends FixedLengthListing {
  const fixed fixed_length = 2
  data: Listing<Int32>
}

// Class whose definition (and values) will get generated
class MyStructThatContainsArray extends Struct {
  array: MyArray
}

my_struct_instance = new MyStructThatContainsArray {
  array = new MyArray {
    data = {
      1
      2
     }
  }
}

Then in my codegen side, I can create an instance of the array type and access the fixed const fixed_length field (lots of omissions for brevity):

// function at the bottom of the codegen tree that generates C type names, still need to work out details to not double-declare names, etc.
const local function renderTypeName(name: String, type_: TypeAlias | Class | reflect.TypeAlias | reflect.Class): String =
  let (type =
    if (type_ is reflect.TypeAlias | reflect.Class)
      type_.reflectee
    else
      type_
    )
  if (type == Int8)
    "int8_t \(name)"
  else if (type == Int16)
    "int16_t \(name)"
  else if (type == Int32)
    "int32_t \(name)"
  // ...
  else if ((type) {} is types.FixedLengthListing) // TODO: why do we have to construct an instance to get type checks to match here, but not for other types?
    let (const_instance = (type) {}) // Construct an instance of the type so we can access const members
    // Plug in array length/field name, and recuse to get the type name of the data in the array
    " \(renderTypeName("", reflect.Class(const_instance.data.default)) \(name)[\(const_instance.fixed_length)]"

I can access the type contained in a Listing with something like this:

// Render properties of a `Struct` declaration
const local function renderPropertyDeclaration(prop: reflect.Property): String =
  let (referent = typ.referent)
  let (reflectee = referent.reflectee)
  if (reflectee == Listing)
    let (mirrorOfListingElement = typ.typeArguments[0].referent)  // Mirror for the type contained in the listing
    "\(renderTypeName(prop.name, mirrorOfListingElement.reflectee));"

So far from my playing around this seems to work as I expect; the MyArray is properly type checked, and the type constraint in the FixedLengthListing class properly checks the size of all created instances.