Complete rewrite of the generator

[MaikKlein]

The current design of the generator has become almost unmaintainable. There have been lots of changes to vk.xml and how we use it in ash in the past 5 years.

I started a very early WIP branch a couple of months ago, but haven't had much time to work on it so far. But I now have more time to spare and hopefully I can finish this up in the next couple of months.

I am going to list some of the changes/additions I have been thinking about:

• Fully switch to vk-parse, don't use the outdated vkxml anymore

• Don't use syn and quote anymore. They don't really have many benefits for pure codegen. Instead we will write directly to the files ourselves.

• The generator needs to be easily understandable and documented. Need to make it easier for people to contribute.

• Feature flags all the things: ash gets slower to compile every release.

• • Feature flag for every extension: By default everything will be active. But you can choose to opt-in into only the extensions are you want to use.
• • Feature flag for builders: While builders are nice, not everyone is using them and they add quite a bit of build time.
• • Feature flag provisional apis: Some extensions in vulkan are experimental, instead of generating those by default, you will have to opt-in into those
• • (Possibly?) Feature flags for the high level interface. For people that only want the low level bindings. Otherwise we still might split ash into ash and ash-sys.

• Generate everything: Currently only the low level part is generated. Every high level function is written by hand. Instead we will now generate 95% and only handwrite functions that are particularly difficult to generate correctly.

• • Extensions: We still can not automatically generate high level extensions by default because the output has to be verified by a human. Otherwise it would be too easy to run into breaking changes.
• • • To add a new extension: It should be as easy as to add the name of the extension to a list, re-run the generator, verify the output, and potentially write some custom overrides for some functions.
• • Feature flags would have to be generated as well. (in .rs and .toml)
• • Make it easy to manually override functions that are hard to generate.

• No build.rs: I have been thinking about this for a while. Initially I wanted to move the generation into a build.rs file, but has some limitations. We can't autogenerate the feature flags for extensions, and it will introduce dev-depencies. I'd rather have ash remain a 0 dependency library for now.

• No huge files anymore: We need to split everything into multiple files. Github already can't show the diffs anymore for some files, and rustc/rustdoc struggle as well with files that are 50k loc.

• ...

[MarijnS95]

@MaikKlein That'd be a great improvement, especially ditching vkxml that has now been unmaintained for quite some time and with a questionable conversion layer from vk-parse to vkxml format.

In particular pointers have bothered me. As explained in #341 (comment) I've deduplicated the type conversion system for safe functions (builders) and their unsafe struct/FFI counterpart in master...Traverse-Research:simplify. Pointers for dynamic arrays denote the same level as the array type, whereas the add up for static arrays.

I'm curious to the thought behind your choice to drop quote though. Having worked on GTK's gir very recently I've been in the process of converting parts to quote because it is much more readable than format strings and \n + \t littered all around codegen files. That latter problem is easily solved by removing them altogether, the generated files will be ran through rustfmt anyway. However, having the ability to write code-blocks effectively in-line, with syntax highlighting and easy expansion of optionals and arrays (not on members though) seems hugely beneficial to me.
In addition things written to a file can't be modified anymore, so you have to be really cautious when to write to a file and when to return the code as String instead (this is a problem in gir utility functions where both exist). With quote that thought process is gone.

I had also already performed the conversion to syn/quote/proc_macro v1: master...Traverse-Research:quote1.

Both of these branches have not been submitted yet to not take away review from KHR_ray_tracing and its preparation PR. Not sure if I should submit them now that those made it in, perhaps that might help when reworking existing parts of the codebase into the new, refactored generator?

[filnet]

If possible, please address #305 during the rewrite.

[MarijnS95]

@MaikKlein Perhaps something to take into account with the new generator: have you thought about what to do when multiple struct fields share the same length field? Right now the builder overwrites it which can either yield unexpected results or out of bounds reads. A good example:

I extended our use of SubmitInfo to take more wait_semaphores, but forgot to pass a longer array to wait_dst_stage_mask. "Fortunately" our ordering set wait_dst_stage_mask after wait_semaphores, overwriting the length of >1 with a length of 1, leaving me wondering why not all semaphores were waited for (in turn resulting in validation layer warnings at least). If it were the other way around we probably got a validation layer for the garbage that would have been read out of bounds of wait_dst_stage_mask.

There seem to be a couple ways to make this safer:

• Combine all arguments sharing the same length in a single builder call, to assert (or at least warn on) length mismatches. Downsides:
  • This makes it harder to distinguish individual members (ie. when looking at autocomplete);
  • Impossible or annoying to leave some members optionally empty.
• Keep track of all separate lengths in the builder struct;
• If the length field isn't 0 (some other array must have been set before), ensure the current array is of the same length;
  • This way fields cannot be re-set (to a different length).
• Or is this a level of safety we don't want to provide/enforce in Ash, being an intermediate "not really safe" but also "not really unsafe" layer?

[cheako]

get_device_proc_addr() feels like it should be callable from an ash::Device.

[aloucks]

I really like how errupt is laid out. It would be nice if ash considered structuring the code in a similar way.

I thought about switching to it at one point but ran into a few things that I didn't like (and I think ash should avoid or keep in mind)

1. I find gitlab much less ideal than github from a contribution and community perspective
2. The generator did not work on Window (at least at the time of when I was investigating it)
3. There were no unit tests
4. There was no CI at all

[Friz64]

Just for clarity's sake, generator Windows support and CI support was since addressed.

[Ralith]

We've discussed merging with erupt in the past, and trying to assemble the best of both worlds. A detailed writeup of where the libs currently differ, and proposals for which approach to retain, would be a great place for that effort to kick off.