diff --git a/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game1.png b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game1.png new file mode 100644 index 0000000..7453625 Binary files /dev/null and b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game1.png differ diff --git a/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game2.png b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game2.png new file mode 100644 index 0000000..576f5fd Binary files /dev/null and b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game2.png differ diff --git a/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game3.png b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game3.png new file mode 100644 index 0000000..66b517b Binary files /dev/null and b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game3.png differ diff --git a/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game4.png b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game4.png new file mode 100644 index 0000000..1867103 Binary files /dev/null and b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/game4.png differ diff --git a/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d.md b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d.md new file mode 100644 index 0000000..27cbbb0 --- /dev/null +++ b/_posts/prepub/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d/i-stopped-fighting-my-tools-and-built-a-game-engine-in-d.md @@ -0,0 +1,627 @@ +--- +title: I Stopped Fighting My Tools and Built a Game Engine in D +author: Alexandros Kapretsos + +categories: + - Community + - Guest Posts + - Project Highlights + - Code + - GameDev +--- + +Building games should be fun. +At some point, it stopped feeling that way for me. + +My primary workflow used to revolve around the [Godot Engine](https://godotengine.org/) and its scripting language. +It was a great fit for my needs (2D games with a retro feel), but there was always a little bit of friction. +Some of it was me wanting something different, and the rest was the engine shifting toward a more opinionated, editor-driven design. +I always preferred a code-driven approach for my projects. +One example of this can be found in my unfinished GDScript library, [Sashimi](https://github.com/Kapendev/sashimi). + +Eventually, that friction grew with new Godot releases, leading me to where I am now: developing my own game engine in D called [Parin](https://github.com/Kapendev/parin). + +Of course, Parin was not my first attempt at game development outside of Godot. +My initial goal was to see if I could create a workflow as nice as the one I was used to. +That led me on a long detour through languages like Nim, Go, Zig, C, and D. +After all that searching, I realized D was exactly what I needed. +It's a pragmatic and unopinionated language that gets out of my way. + +In this blog, I'll go over some features of D and how I use them to make games. +The TL;DR is: + +- A single language for game logic and scripting. +- Fast compile times under 1 second. +- The freedom to choose the best memory allocation strategy. +- Achieving C-like speed with a much cleaner developer experience. + +*Game Made with Parin: [Worms Within](https://kapendev.itch.io/worms-within)* +![Worms Within Screenshot](game1.png) + +## Memory Management + +D's unopinionated approach is most evident in the control it gives me over memory. +In Parin, I've structured the code so that it avoids the garbage collector by default. +It instead relies primarily on static data structures and an [arena allocator](https://en.wikipedia.org/wiki/Region-based_memory_management) that is cleared at the end of every frame. + +### Arena Allocators + +The engine implements two types of arenas: + +- `Arena`: A fixed-size buffer. It's perfect for temporary memory where the upper bound is known. +- `GrowingArena`: A linked list of `Arena` chunks. This provides a "pay-as-you-go" strategy. + +```d +struct Arena { + ubyte* data; + size_t capacity; + size_t offset; + // ... metadata for checkpoints. + Arena* next; +} + +struct GrowingArena { + Arena* head; + Arena* current; + size_t chunkCapacity; +} +``` + +From the two, `GrowingArena` is the type of the arena mentioned earlier. +To make these types more ergonomic, a RAII helper is used sometimes called `ScopedArena`. +It uses the destructor to automatically rollback the arena offset when a scope ends. +Combined with D's [`with statement`](https://dlang.org/spec/statement.html#with-statement), it creates an elegant way to work with arenas: + +```d +import parin; + +void main() { + ubyte[1024] buffer = void; + auto arena = Arena(buffer); + + with (ScopedArena(arena)) { + make!char('C'); // The `make` method of `ScopedArena` advances the offset. + with (ScopedArena(arena)) { + make!short(3); + make!char('D'); + assert(arena.offset == 5); + } + // The offset is back to where it was before the nested block. + assert(arena.offset == 1); + } + // The offset is back to the start. + assert(arena.offset == 0); +} +``` + +### Static Data Structures + +Similar to `Arena` and `GrowingArena`, many data structures take a compile-time argument to toggle between static or dynamic allocation. +The engine prefers the static versions because they avoid runtime allocations and allow for easy bundling of different data into a single block of memory. +Below is a simplified example of how this works: + +```d +// A list with a dynamic capacity. +struct List(T) { + T[] items; + size_t capacity; +} + +// A list with a fixed capacity. +struct FixedList(T, size_t N) { + T[N] data; + size_t length; + + T[] items() { + return data[0 .. length]; + } + + enum capacity = N; +} + +// A 2D grid. Type `D` defines its behavior. +struct Grid(T, D = List!T) { + D tiles; + int rowCount; + int colCount; + + void fill(T value) { + foreach (ref tile; tiles.items) { + tile = value; + } + } +} + +// A dynamic grid type. +alias Rooms = Grid!short; +// A static grid type. +alias Map = Grid!(short, FixedList!(short, 128 * 128)); +``` + +In the example above both `List` and `FixedList` share a common public interface (`items` and `capacity`). +While their underlying types differ, with `items` being a variable for the first and a [property](https://dlang.org/spec/function.html#optional-parenthesis) for the other, they remain functionally compatible. +Consequently, generic functions that work with `Grid` types will use either without issue. + +Below is a more complicated example of this from Parin: a [generational array](https://lucassardois.medium.com/generational-indices-guide-8e3c5f7fd594) (handle map) type: + +```d +struct GenList(T, D = SparseList!T, G = List!Gen) if (isGenContainerPartsValid!(T, D, G)) { + D data; + G generations; +} + +bool isGenContainerPartsValid(T, D, G)() { + static if (__traits(hasMember, D, "isBasicContainer")) { + static if (isSparseContainerPartsValid!(T, D.Data)) { + static if (__traits(hasMember, G, "isBasicContainer")) { + // NOTE: Can be written better, but I don't care. + return G.isBasicContainer && G.hasFixedCapacity == D.hasFixedCapacity; + } else { + return false; + } + } else { + return false; + } + } else { + return false; + } +} +``` + +### Dynamic Allocations + +For the parts that require dynamic allocation, the engine provides two paths. +It sometimes accepts user-allocated memory, meaning a user can decide exactly what kind of memory (GC, malloc, or stack) they want to use. +A good example of this is the experimental UI library for Parin called `ui2`: + +```d +import parin, parin.ui2; + +UiContext ui; +UiCommand[64] uiCommandsBuffer = void; +char[1048] uiCharDataBuffer = void; + +// Called once when the game starts. +void ready() { + lockResolution(320, 180); + // Manually manage memory for the UI using static buffers. + ui.readyUi(uiCommandsBuffer, uiCharDataBuffer); +} + +// Called every frame while the game is running. +bool update(float dt) { + ui.beginUiFrame(); + scope (exit) ui.endUiFrame(); + + // Define the UI layout and handle interactions. + auto screen = IRect(resolution.toIVec()); + screen.subAll(8); + auto menu = ui.rowItems(screen.subTop(20), 7, 5); + if (ui.button(menu.pop(), "1")) println("1!"); + if (ui.button(menu.pop(), "2")) println("2!"); + if (ui.button(menu.pop(), "3")) println("3!"); + + return false; +} + +// Creates a main function that calls the given functions. +mixin runGame!(ready, update, null); +``` + +For everything else, it uses a "nogc" utility library I wrote called [Joka](https://github.com/Kapendev/joka). +Memory allocated through Joka has to be freed manually. + +And that's it? +Well, not really. +A lot of programming languages would stop you right there, by making you pick a main allocation strategy and allowing limited support for other ones. +But D gives you more options than that. + +While Joka is designed for manual memory management, it includes a `JokaGcMemory` version flag. +When defined, the library's default memory allocations switch at compile time to using the garbage collector and any functions that free memory are basically a no-op. +This is similar to how some C libraries provide a way to replace internal functions. +Even in this setup, it is still possible to manage memory manually at runtime because Joka provides an allocator API for fine-grained control. +What this flag does in practice is simply replace the default allocator value passed to or used by every Joka (and Parin) function. +Since GC pointers in D have the same type as any other pointer, things continue to work out of the box when changing the defaults. + +Below is the allocator API for Joka: + +```d +struct MemoryContext { + void* allocatorState; + AllocatorReallocFunc reallocFunc; + + void* malloc(size_t alignment, size_t size, const(char)[] file, size_t line) { + return reallocFunc(allocatorState, alignment, null, 0, size, file, line); + } + + void* realloc(size_t alignment, void* oldPtr, size_t oldSize, size_t newSize, const(char)[] file, size_t line) { + return reallocFunc(allocatorState, alignment, oldPtr, oldSize, newSize, file, line); + } + + void free(size_t alignment, void* oldPtr, size_t oldSize, const(char)[] file, size_t line) { + reallocFunc(allocatorState, alignment, oldPtr, oldSize, 0, file, line); + } +} + +alias AllocatorReallocFunc = void* function(void* allocatorState, size_t alignment, void* oldPtr, size_t oldSize, size_t newSize, const(char)[] file, size_t line); +``` + +It's a simple API that works for my needs. +I know D already includes an experimental API in the standard library, but I made my own to learn how they work. +I also have a bit of a "Not Invented Here" problem sometimes. +We are just having fun here. + +An example of allocators in action: + +```d +import joka; + +void main() { + ubyte[1024] buffer = void; + auto arena = Arena(buffer); + auto i = 0; + // Use the arena to allocate memory for the numbers. + auto numbers = List!int(arena.toMemoryContext(), 1, 2, 3); + assert(numbers[i++] == 1); + assert(numbers[i++] == 2); + assert(numbers[i++] == 3); +} +``` + +To mitigate some memory bugs when `JokaGcMemory` is not enabled, Joka tracks all allocations in debug builds with a thread-local allocator. +Parin is set up to provide immediate feedback using that information if someone forgets to free memory or attempts an invalid free. +This works because the allocator API requires a file and line argument for everything it does. +The reports look like this: + +```d +Memory Leaks: 5 (total 934 bytes, 8 ignored) + 1 leak, 16 bytes, source/app.d:24 + 1 leak, 32 bytes, source/app.d:31 + 2 leak, 128 bytes, source/story.d:17 [group: "Actor"] + 1 leak, 32 bytes, source/story.d:40 [group: "Actor"] +``` + +The tracking system also includes features like ignoring leaks and grouping allocations under a name, so the output is less noisy. +It's not a 100% solution, but it covers many practical cases. +I prefer this simpler approach over smart pointer abstractions for the kind of code I write. + +There is one last thing both Joka and Parin can do with memory: changing the allocator used inside a scope implicitly. +So if I have a function allocating things dynamically, I can "intercept" it and force it to allocate things on the stack, for example. +It's a niche feature for exceptional cases and a thread-local variable called `__memoryContext` is what makes it work. +This mechanism is sometimes referred to as a context system. + +Here is an example of managing that thread-local variable via a RAII helper called `ScopedMemoryContext`: + +```d +import joka; + +void main() { + ubyte[1024] buffer = void; + auto arena = Arena(buffer); + auto i = 0; + // Use the arena to allocate memory for everything inside the `with` block. + // `ScopedMemoryContext` automatically restores the previous context when exiting this block. + with (ScopedMemoryContext(arena)) { + auto numbers = List!int(1, 2, 3); + assert(numbers[i++] == 1); + assert(numbers[i++] == 2); + assert(numbers[i++] == 3); + } +} +``` + +I am not the biggest fan of this approach because it can make things harder to reason about. +At least, that has been my experience with languages that provide a built-in way to do this with a special calling convention. +The reason it's more noticeable in those languages is that people tend to reach for built-in features heavily, so the bad parts become worse. +A context system is essentially a global variable that you have to account for. +It's like the [PICO-8](https://www.lexaloffle.com/pico-8.php) API with its pen color, but for memory management and with scope magic. +To prevent some implicit interactions, my library code avoids changing the context and it is strictly a user-side option. + +And that's it. All this combined gives me the choice to keep manual control, let D handle everything, or use a combination of both. +I can pick the best solution for a project without the compiler complaining about why I am doing things the "wrong" way. +That said, D does have features that enforce strictness, the [`@nogc`](https://dlang.org/spec/function.html#nogc-functions) attribute for example, but both Joka and Parin use those only when they don't introduce extra friction. +None of my libraries support `@nogc` fully and that is by design, even though in theory they could. +A combination of the `-vgc` flag and knowing what my code does has been working well instead. + +While mixing allocation strategies like this might sound weird to anyone used to a "one or the other" approach, I've found plenty of use cases for it, especially when collaborating. +When I'm working with people who aren't comfortable with manual memory management, I can simply tell them to use the garbage collector while I focus on the low-level parts. +This provides a setup similar to a C++ and Lua combination, but without the cross-language cost. + +One other use case for mixing GC and non-GC code is the tracking system mentioned earlier. +Yes, it's "secretly" using the garbage collector. +I just offload all of the work to it instead of worrying about allocations that don't matter to my program's performance. +It's debug-only code at the end of the day, so who cares if it uses the garbage collector or not? +That code is stripped out in release builds anyway. + +I think I covered almost everything I do with memory in D. +Might have missed one thing, but the point still stands. +Having this level of control without fighting the language is awesome! + +*Game Made with Parin: [A Short Metamorphosis](https://kapendev.itch.io/a-short-metamorphosis)* +![A Short Metamorphosis Screenshot](game2.png) + +## Metaprogramming + +Metaprogramming is something I'm not good at, but I do enjoy it sometimes. +D does a great job of providing a smooth experience for it because it feels like writing regular code instead of a different language. + +### Entity Systems + +One of my use cases is building entity systems. +While Parin doesn't force a specific one on you, it provides a tagged union that makes building one straightforward. +It looks like this: + +```d +alias UnionType = ubyte; + +struct Union(A...) if (A.length != 0) { + union UnionData { + // Creates the fields of the raw union. + static foreach (i, T; A) { + mixin("T _m", i.stringof, ";"); + } + } + + UnionData _data; + UnionType _type; +} + +// An example of a union that holds two types. +alias Entity = Union!(Marioni, Goombani); +struct Marioni { float x, y; int hp; } +struct Goombani { float x; } +``` + +The real type includes some extra information about its fields, which allows for safety checks at compile time. +I personally use a [`static assert`](https://dlang.org/spec/version.html#static-assert) in my games to ensure that every type in the tagged union shares the same first field, the "base" of the union as I call it. +This makes sure that I can safely access shared data (like position or size) without needing to manually check the active union type at runtime. + +For example: + +```d +// This guarantees that accessing `base` of `Entity` is always safe. +static assert(Entity.isBaseAliasingSafe); +// Access the base shared by all types and move everything to the right. +foreach (ref e; entities.items) e.base.x += 32; +``` + +To handle specific logic for different types, I use a templated function named `call`. +This generates a large `switch` statement that calls the correct method for the currently active type. + +An example of using the `call` function: + +```d +// Automatically calls `update` and `draw` for the underlying type. +foreach (ref e; entities.items) e.call!"update"(dt); +foreach (ref e; entities.items) e.call!"draw"(); +``` + +Since everything happens at compile time, the compiler will give clear error messages if a method is missing. +This can be combined with D's [`alias this`](https://p0nce.github.io/d-idioms/#Extend-a-struct-with-alias-this) feature to provide default implementations for types that lack the needed methods. +Below is an example of what a basic entity type looks like using this: + +```d +import parin; + +// The base type of every entity. +struct EntityBase { + Rect body; + + // The default implementations. + void update(float dt) {} + void draw() {} +} + +// Actor is a type of entity. +struct Actor { + EntityBase base; + alias base this; + + // Custom draw logic. + void draw() { + // `body` is part of `EntityBase`. + drawRect(body, orange); + drawText("Actor", body.position); + } +} +``` + +This keeps my code clean and lets me use a "mega struct" style approach, where every entity property is in one place, without the space inefficiency of an actual mega struct. +A [complete example](https://github.com/Kapendev/parin/blob/main/examples/basics/_018_entity.d) of the code above is available in the Parin repository. + +### Debug Tools + +Moving away from game logic, the same kind of compile-time introspection is quite handy for building debug tools. +Since the code can look at a struct and see every member inside it, I can for example write functions that automatically generate UI elements for those members. +In Parin, I have a helper called `headerAndMembers` that I use to build debug editors for any game object: + +```d +import parin, parin.addons.microui; + +Game game; + +struct Game { + int width = 50; + int height = 50; + IVec2 point = IVec2(70, 50); +} + +void ready() { + readyUi(engineFont, 2); +} + +bool update(float dt) { + beginUiFrame(); + scope (exit) endUiFrame(); + + drawRect(Rect(game.point.x, game.point.y, game.width, game.height)); + if (beginWindow("Edit", IRect(500, 80, 350, 370))) { + headerAndMembers(game, 125); + endWindow(); + } + return false; +} + +mixin runGame!(ready, update, null); +``` + +Instead of manually writing a line of UI code for every single member I want to tweak, I let the compiler handle it. +Any new variables added to the game state will simply appear in the editor the next time I run the game. + +To customize this further, I can also use [user-defined attributes](https://dlang.org/spec/attribute.html#uda) to control how things behave. +For example, applying `@UiMember("Health")` to a variable will override its display name. +You can even define constraints for sliders. +Applying `@UiMember("Volume", 0, 100, 1)` tells the editor to limit the value between 0 and 100 with a step of 1: + +```d +// The attribute used by the UI system. +struct UiMember { + const(char)[] name; // The name of the member. + UiReal low; // Used by sliders. + UiReal high; // Used by sliders. + UiReal step; // Used by sliders. +} + +alias UiReal = float; +``` + +### Joint Allocations + +Finally, one other interesting thing I do with metaprogramming is joint allocations. +This is the practice of allocating multiple arrays in a single contiguous block of memory to improve cache locality and reduce allocator overhead. +While you can do this manually, D's introspection allows for a much more elegant and safe solution. + +Here is a small example of this using the `jokaMakeJoint` function: + +```d +import joka, std.stdio; + +struct Ve2 { float x, y; } +struct Ve3 { float x, y, z; } + +struct Mesh { + Ve3[] positions; + int[] indices; + Ve2[] uvs; + + this(size_t positionsLength, size_t indicesLength, size_t uvsLength) { + // `jokaMakeJoint` calculates the total size and offsets for all arrays + // and performs a single allocation. + this = jokaMakeJoint!Mesh(positionsLength, indicesLength, uvsLength); + } + + void free() { + // The first slice has the pointer that needs to be freed. + jokaFree(this.tupleof[0].ptr); + } +} + +void main() { + auto mesh = Mesh(4, 6, 4); + writeln("Positions: ", mesh.positions); + writeln("Indices: ", mesh.indices); + writeln("UVs: ", mesh.uvs); + mesh.free(); +} +``` + +In the `free` method above, I use the [`tupleof`](https://dlang.org/spec/property.html#tupleof) property. +In D, this allows you to access the fields of a struct as a compile-time sequence. +Since `jokaMakeJoint` allocates one big block and points the first field to the start of it, freeing `this.tupleof[0].ptr` (the pointer of the first slice, `positions`) effectively frees the entire memory block. + +These are simple things, but combined they make my code simpler. + +*Game Made with Parin: [Twenty Seconds, Twenty Steps](https://kapendev.itch.io/twenty-seconds-twenty-steps)* +![Runani Screenshot](game3.png) + +## Compile Times + +D's compile times are remarkably fast. +This alone is a major reason why I use D. +On an older Ryzen 3 2200G running Ubuntu, my games currently compile in around 0.6 seconds without using a build system. +I usually use [DUB](https://dub.pm/) for building, but I'm avoiding it for this section to give a clearer picture of how fast things are without any extra build steps. +Additionally, I'm using the default linker that comes with Ubuntu. + +These times can drop to roughly 0.4 seconds when using the `-betterC` flag. +Below is a breakdown of compile times for "hello-world" programs using Parin and Joka: + +| Compiler | Parin | Parin & `-betterC` | Joka | Joka & `-betterC` | +| :------- | :----- | :----------------- | :----- | :---------------- | +| **DMD** | 0.585s | 0.370s | 0.296s | 0.134s | +| **LDC** | 1.918s | 1.634s | 0.565s | 0.565s | +| **GDC** | 3.535s | No flag | 0.906s | No flag | + +The files used in the benchmark can be found in the example folders of both libraries with the name `_001_hello.d`. +They intentionally import more modules than a minimal program to simulate a real-world setup. +From my tests, the module that takes the longest time to compile is the math module of Joka. +The basic Joka program below with `-betterC`, DMD and 4 imported modules (`joka.io` has 3 dependencies) takes 0.081s to compile: + +```d +import joka.io; + +extern(C) +void main() { + println("Hello world", 999, '!'); +} +``` + +Here is also an overview of the Parin and Joka codebase: + +| Project | D Files | D Blank | D Comment | D Code | +| :-------- | :------ | :------ | :-------- | :----- | +| **Parin** | 37 | 3270 | 2036 | 19634 | +| **Joka** | 11 | 1534 | 681 | 8169 | + +Overall, I get fast code and fast compile times. +Those numbers obviously will vary for every D project, depending mainly on the quantity and complexity of metaprogramming. + +## Workflow + +Because of the fast compile times and the helpful standard library (which I haven't mentioned until now), I also use D as a scripting language. +The script that creates web builds for my games is written entirely in D. +It handles packaging, asset copying, and the configuration needed for the web target. +Instead of maintaining separate scripts for different platforms, I use one language everywhere and it works fine. + +An example of using the web script with DUB: + +```sh +dub run parin:web +``` + +The same idea is used for a small setup script for DUB projects. +It generates the folders and files I usually want when starting a new game. +One of them is an `app.d` file containing a basic hello-world program. +The script can also include a minimal entity system by passing a flag to it called `entity`. + +An example of using the setup script with DUB: + +```sh +dub init -t parin -- entity +``` + +The bottom line is that the workflow is simple. +When I need automation or tooling, I just write more D. +This also allows me to share code between my game and my scripts if needed. +I still use shell and batch scripts when it makes sense, but most projects don't really need them. + +## Moving On + +I think I said a lot of nice things about D already, so I will stop here. +The main point of everything is not to say that I use D to save the world or to participate in language wars. +I just wanted to stop fighting my tools and get back to making games. +I'm definitely still figuring things out as I go, but for now, the friction is gone, I'm having fun, and I'm actually finishing projects again. +That's enough of a win for me. + +## Get Involved + +And this is the end. +I'm Alexandros F. G. Kapretsos, a game developer and Economics student at AUEB. +If you enjoyed this, feel free to check out my work: + +- Check [Parin](https://github.com/Kapendev/parin) and [Joka](https://github.com/Kapendev/joka) on GitHub. +- Take a look at [microui-d](https://github.com/Kapendev/microui-d), my rewrite of [rxi's microui](https://github.com/rxi/microui) with bug fixes, texture support and other D-specific improvements. [Parin comes with it out of the box](https://github.com/Kapendev/parin/blob/main/examples/integrations/microui.d)! +- See the engine in action by playing my games on [kapendev.itch.io](https://kapendev.itch.io/). +- Read my personal rants about game development on [dev.to/kapendev](https://dev.to/kapendev). + +*Game Made with Parin: [Runani](https://kapendev.itch.io/runani)* +![Runani Screenshot](game4.png) diff --git a/_posts/prepub/symmetry-autumn-of-code-report-importc-enhancements.md b/_posts/prepub/symmetry-autumn-of-code-report-importc-enhancements.md new file mode 100644 index 0000000..352f530 --- /dev/null +++ b/_posts/prepub/symmetry-autumn-of-code-report-importc-enhancements.md @@ -0,0 +1,157 @@ +--- +title: "Symmetry Autumn of Code Report: ImportC Enhancements" +author: Emmanuel Nyarko + +categories: + - Symmetry Autumn of Code + - Compilers & Tools + - The Language + - Code + - Community +--- + +Having a central compiler that can compile code from other interoperable languages has long been one of D's major goals. And of course, that language is C. Over the years, D has adopted a powerful initiative to compile C code directly through ImportC, and several improvements were made during the 2025 term of Symmetry Autumn of Code. + +You can [read more on ImportC here](https://dlang.org/spec/importc.html). + +The D programming community has engineered powerful compilers with very fast compile times. C has many libraries that are widely used in numerous systems. Considering D's edge in performance and safety, the ability to compile legacy C code enhances the ease of adoption in several domains. For example, a ticketing system written in C that has functioned for years cannot be easily rewritten in a modern programming language just to improve it. Not every development team is ready for that, considering the cost of re-engineering. + +## Breaking Through Technical Barriers + +### Complex numbers and s7 library support + +Complex numbers are heavily used in control systems, signal processing, graphics, and scientific computing. C has been the language of choice for writing libraries used in high-performance DSP, embedded systems, and real-time signal pipelines. + +Historically, we had a hard time compiling C complex numbers from the frontend. It was even harder, sometimes impossible, to compile libraries with complex signatures, like the scientific **s7** library. There have been massive improvements in compiling such code, and we can now compile almost any C complex code. Below is a simple code snippet that now compiles successfully with D: + +```c +#include +#include +#include + +struct com +{ + int r; + int c; +}; + +void foo() +{ + struct com obj = { 1, 3 }; + _Complex double x = { obj.r, obj.c }; // real + im*i + + assert(creal(x) == 1.000000); + assert(cimag(x) == 3.000000); + return; +} + +int main() +{ + foo(); + return 0; +} +``` + +## Deepening C99 support + +### Designated Initializers + +ImportC can now compile C's designated initializers. If you have tried compiling C code involving designated initializers in the past, you will have realized that nested struct initializers were not supported and caused compiler errors. This has been significantly improved and fixed. + +```c +struct top +{ + int a; + int b; +}; + +union t_union +{ + int f; + int g; +}; + +struct Foo +{ + int x; + int y; + struct top f; +}; + +struct Bar +{ + struct Foo b; + int arr[3]; + union t_union u; +}; + +struct Bar test = { + .b.x = 5, + .b.y = 7, + .b.f = {8, 9}, + .arr[0] = 10, + .arr[1] = 11, + .u.f = 13 +}; +``` +This is a simple snippet we can look at to the implementation of C struct designated initializers. As with C, you can go as deep as you want, and we will compile that for you while ensuring your struct members contain the desired data. + +### Compound Literals + +Taking the address of compound literals with ImportC did not compile before. This has been fixed, and compound literals can now be used as lvalues with ImportC. + +```c +int *c = &(int){90}; +``` +This is an integer pointer referencing a temporary int initialized to 90. Rest assured that you will read 90 at the memory address pointed to by `c`. + +### Function and Variable Redeclarations + +Function redeclarations are permissible in a local scope in C. We now do a great job compiling such redeclarations without compiler errors. Also, extern variable redeclarations at global scope have been hardened. ImportC has greatly improved type checking for both global and local redeclarations. + +```c +/* for variables */ +extern int x; +extern char x + +/* for function declarations */ +int foo(); +double foo(); +``` +Like in C, this is not permissible, and D has greatly improved to ensure these cases are checked. + +### C Macros + +Macros defined in C programs can now be imported from D. These can be easily passed as flags or function arguments, depending on your use case. While this hasn't always functioned well in the past, especially when imported and used in D code, significant improvements have been made. Beyond that, several builtin macros have also been implemented in the compiler. + +## Builtins support, Backend, and Linking Wins + +### GNU GCC CRC builtins + +Most of the GNU GCC Cyclic Redundancy Check (CRC) builtins have been implemented. If any C function referencing them is used in D, we provide the necessary builtin implementation. + +### DMD Backend Symbol duplication + +Redeclaration of global variables was initially problematic, as it often led to symbol duplication in the symbol table, particularly in DMD. This has been fixed, allowing D to link against large-scale C libraries that rely on redundant global declarations across multiple headers. + +### Static linking + +We previously had a difficult time creating static libraries from C modules, especially those involving forward declarations. Work has been done toward this, and we can now successfully create static libraries from C modules. + +```c +static void static_fun(); + +void lib_fun() +{ + static_fun(); +} + +static void static_fun() +{ +} +``` +You will need to pass the `-lib` command-line option when creating a static library. `dmd -lib file.c` now supports this workflow. + +## Community Acknowledgement + +This work was done through the 2025 Symmetry Autumn of Code. A big thank you to the D mentors, the D community, and Symmetry Investments for sponsoring this. diff --git a/_posts/prepub/teaching-an-ai-to-know-itself-building-a-local-llm-agent-in-d.md b/_posts/prepub/teaching-an-ai-to-know-itself-building-a-local-llm-agent-in-d.md new file mode 100644 index 0000000..9febb7f --- /dev/null +++ b/_posts/prepub/teaching-an-ai-to-know-itself-building-a-local-llm-agent-in-d.md @@ -0,0 +1,205 @@ +--- +title: "Teaching an AI to Know Itself: Building a Local LLM Agent in D" +author: Danny Arends + +categories: + - Community + - Guest Posts + - Project Highlights + - Tutorials + - Code + - Machine Learning +--- + +I've been writing D for a long time. [DaNode](https://github.com/DannyArends/DaNode), my self-contained web server, has been running in production for over 12 years. [DImGui](https://github.com/DannyArends/DImGui) is a full SDL + Vulkan renderer that supports skeletal animations via the Open Asset Import Library, HDR lighting, and compute shaders, written entirely in D calling into external libraries via `importC`. So when I decided to build a local agentic large language model (LLM) ([DLLM](https://github.com/DannyArends/DLLM)) from scratch, I'd sooner write it in Brainfuck than reach for Python. To be fair, the Python LLM ecosystem is enormous. However, by the time you have a working agent, you're sitting on top of a framework, which wraps a library, which calls into C++ via ctypes, which dispatches to CUDA kernels. Python all the way down to the metal, with several layers of abstraction you didn't write and can't easily debug. I wanted to understand what was actually happening. + +*[DLLM](https://github.com/DannyArends/DLLM) is my latest D project: a minimal, clean coding agent built directly on llama.cpp. No Python, no bindings, no overhead.* + +Here's a walkthrough of the two parts I'm most happy with: the `@Tool` UDA registration system, and grammar-constrained sampling. + +#### Starting Point: importC + +Before anything else, the foundation. D's `importC` lets you include C headers and use the API directly, as native D code. DLLM has one file, `includes.c`, that pulls in the llama.cpp and mtmd headers. From there, `llama_decode`, `llama_model_load_from_file`, `llama_sampler_sample`, the whole llama.cpp API, is available in D with full type safety and zero FFI overhead. + +This is the same trick I used in DaNode to wrap OpenSSL, and an integral part of DImGui to call into Vulkan, SDL, the Open Asset Import Library, and shaderC. `importC` is one of my favorite D features. I used to rely heavily on the Derelict & BindBC wrappers, and they were fantastic community contributions, but `importC` has made them almost obsolete. No wrapper libraries, no binding maintenance, no surprises when the upstream C API updates." + +#### The Tool System: Start With a Single UDA + +An LLM agent is only useful if it can *act*. DLLM's tools cover web search, file I/O, Docker-sandboxed code execution, image download, date and time, text encoding, and audio playback. To *act*, it needs tools that it can control, and in DLLM you can create a new tool that the agent can use like this: + +```d +@Tool("Count how many times substring appears in text.") +string nOccurrences(string text, string substring) { + try { + return to!string(text.count(substring)); + } catch (Exception e) { return(format("Error: %s", e.msg)); } +} +``` + +The `@Tool(...)` attribute is the entire registration step. No schema file to maintain, no separate dispatch table. The `Tool` struct itself is trivial: + +```d +struct Tool { + string description; +} +``` + +One string. That's the whole UDA definition. Everything else is derived from it and the function signature automatically. The description string is also used by the LLM agent to figure out what the tool is able to do. + +#### Building Up: RegisterTools + +At the top of each tool module, there's one line: + +```d +mixin RegisterTools; +``` + +This is a `mixin template` that injects a `static this()` module constructor. When the program starts, that constructor runs and populates a global tool definition array (`ToolDef[]`) called `ALL_TOOLS`. Here's how it works, step by step. + +First, it gets a reference to the current module using the `__MODULE__` string mixin trick: + +```d +mixin("alias ThisModule = " ~ __MODULE__ ~ ";"); +``` + +Then it loops over every symbol in that module using `__traits(allMembers, ...)` and `static foreach`: + +```d +static foreach(name; __traits(allMembers, ThisModule)) {{ + mixin("alias member = " ~ name ~ ";"); + static if (is(typeof(member) == function)) { + static if (hasUDA!(member, Tool)) { +``` + +For each function that has a `@Tool` attribute, it extracts the description and the parameter names: + +```d +enum description = getUDAs!(member, Tool)[0].description; +alias ParamNames = ParameterIdentifierTuple!member; +``` + +`ParameterIdentifierTuple` is a standard D trait that gives you the parameter names as a compile-time tuple: For `nOccurrences(string text, string substring)` that's `["text", "substring"]`. Then it builds an executor closure that unpacks the JSON arguments and calls the function: + +```d +auto executor = (JSONValue args) { + string[] argValues; + static foreach(paramName; ParamNames) { + argValues ~= args[paramName].type == JSONType.string ? + args[paramName].str : + args[paramName].toString(); + } + // mixin generates: return member(argValues[0], argValues[1]); + mixin(callStr); +}; +ALL_TOOLS ~= ToolDef(name, description, parameters, executor); +``` + +So after startup, `ALL_TOOLS`, the global tool definition array contains everything needed to both describe each tool to the LLM agent and allow it to call it by name at runtime. The function signature is the *single* source of truth. + +#### What Gets Generated: System Prompt and Grammar + +From `ALL_TOOLS`, two things are auto-magically generated. First, `toolsToJSON()` generates the JSON that goes into the system prompt, so the model knows what tools exist, and what they can do: + +```json +[{ + "name": "nOccurrences", + "description": "Count how many times substring appears in text.", + "parameters": { + "type": "object", + "properties": { + "text": {"type": "string"}, + "substring": {"type": "string"} + } + } +}] +``` + +Second, `buildJsonGrammar()` generates a [GBNF grammar](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md) for constrained sampling. A GBNF grammar is a set of rules that define exactly what sequence of tokens (text) is valid. A simple example for a yes/no answer would look like: +```d +root ::= "yes" | "no" +``` +That's it, the sampler can now only produce the word "yes" or "no", nothing else. For DLLM's tool calls, the grammar is more complex but the principle is identical. The toolname rule is generated dynamically from ALL_TOOLS, so only real tool names are valid. Everything else follows standard JSON structure rules. + +Unlike many Python based agent frameworks which handle tool calls with prompt engineering and output parsing. Grammar-constrained sampling on the other hand gives an iron clad guarantee. The grammar is fed to llama.cpp's sampler, and it restricts the token vocabulary at every step to only tokens that keep the output valid. The full GBNF grammar definition of valid JSON toolcalls is: + +```d +string buildJsonGrammar() { + auto names = ALL_TOOLS.map!(t => "\"\\\"" ~ t.name ~ "\\\"\"").join(" | "); + return(` +root ::= "{" ws "\"name\"" ws ":" ws toolname ws "," ws "\"arguments\"" ws ":" ws object ws "}" +toolname ::= ` ~ names ~ ` +object ::= "{" ws (string ws ":" ws value (ws "," ws string ws ":" ws value)*)? ws "}" +array ::= "[" ws (value (ws "," ws value)*)? ws "]" +value ::= string | number | object | array | "true" | "false" | "null" +string ::= "\"" ([^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]))* "\"" +number ::= "-"? ([0-9] | [1-9] [0-9]+) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? +ws ::= [ \t\n\r]* +`); +} +``` + +The key part is the `toolname` rule, which is generated dynamically from the global `ALL_TOOLS` tool definition array. If you've registered `webSearch`, `nOccurrences`, and `countWords`, the rule becomes: + +``` +toolname ::= "\"webSearch\"" | "\"nOccurrences\"" | "\"countWords\"" +``` + +The model can only produce a `name` field that contains a tool that actually exists. The grammar enforces it at the logit level. This solves the model hallucinating non-existing tools or producing malformed JSON; it's structurally impossible. + +#### The Sampler Switch + +Two samplers (the component responsible for selecting the next token) are set up at startup. The conversational sampler runs at temperature 0.7 during normal thinking and output generation. The JSON sampler runs at a lower temperature (0.3), and crucially, has the grammar constraint attached: + +```d +llama_sampler_chain_add(model.json, llama_sampler_init_temp(0.3f)); +llama_sampler_chain_add(model.json, llama_sampler_init_grammar(model.vocab, buildJsonGrammar().toStringz(), "root")); +llama_sampler_chain_add(model.json, llama_sampler_init_dist(LLAMA_DEFAULT_SEED)); +``` + +During generation, the code watches for `` and `` tags in the output stream. Switching samplers is a single line: + +```d +auto sampler = (agent.json && inToolCall) ? agent.json : agent.sampler; +auto token = llama_sampler_sample(sampler, agent.ctx, -1); +``` + +The moment a `` tag appears in the buffer, the grammar sampler takes over. The model *cannot* produce a malformed tool call while it's active. After `` closes, the grammar sampler is reset and the conversational sampler takes back over. + +No parsing heuristics, no fallback regex. Malformed tool calls are structurally impossible. + +#### The Self-Knowledge Trick + +The current version can read and reason about its own source code using just the Qwen 8B model. This isn't magic, it's Retrieval-Augmented Generation ([RAG](https://en.wikipedia.org/wiki/Retrieval-augmented_generation)). You can ask DLLM to index its own source code living in the *./src/* folder using the embedding model. Source code is chunked, chunks are tokenized, and embedded using a dedicated CPU-resident Nomic embed model, and stored with cosine similarity scoring: + +```d +float cosineSimilarity(float[] a, float[] b) { + float denom = sqrt(a.map!(x=>x*x).sum) * sqrt(b.map!(x=>x*x).sum); + return denom == 0.0f ? 0.0f : dotProduct(a, b) / denom; +} +``` + +The index is binary-persisted between sessions using `rawWrite` and `rawRead`, with a magic number to catch stale files. When you ask a question, the top-k most relevant chunks are retrieved and injected into context. + +What makes it interesting is what's being indexed. Because every tool is a plain D function with a `@Tool` attribute, the source files are already their own documentation. The model doesn't have to reverse-engineer intent from implementation. The description is right there in the attribute, and the implementation is a few lines below it. + +*The practical result: you can ask "how does web search work?" and the agent retrieves the `webSearch` function, reads the `@Tool` description, and explains it accurately. With a small model. Locally.* + +#### What's Included +DLLM is more than just the tool system and grammar sampler. Here's everything that's included out of the box: +* RAG with binary-persisted embeddings and cosine similarity ranking +* Vision support via mtmd (load an image, ask about it) +* Docker-sandboxed code execution (Python, JavaScript, Bash, R, D) +* Web search via SearxNG, and web fetch +* File I/O, date/time, encoding, audio playback tools +* KV cache condensation via a dedicated summary model +* Thinking budget enforcement via token limits +* Memento system, where the agent writes notes to its future self between sessions +* Full interactive and oneshot modes + +#### In closing + +D gave me `importC` for zero-overhead access to llama.cpp, UDAs and `__traits` for a tool system with one source of truth, and UFCS for code that reads the way I think. The entire tool registration and grammar generation system is about 150 lines. + +If you've been looking for a project to try D on, local AI tooling is a good fit. The space is young, the performance characteristics reward D's zero-overhead philosophy, and the metaprogramming needs of LLM agents map almost perfectly onto what D does best. + +DLLM is open source under GPLv3. The code is small enough to read in an afternoon, find it at [github.com/DannyArends/DLLM](https://github.com/DannyArends/DLLM).