Speed up durable wire ingest path

.github/workflows/benchmark.yml

@@ -39,10 +39,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Install Zig 0.15.2
+      - name: Install Zig 0.16.0
         uses: mlugg/setup-zig@v2
         with:
-          version: 0.15.2
+          version: 0.16.0
 
       - name: Install Python deps
         run: pip install pymongo psycopg2-binary

.github/workflows/ci.yml

@@ -17,10 +17,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Install Zig 0.15
+      - name: Install Zig 0.16
         uses: mlugg/setup-zig@v2
         with:
-          version: 0.15.2
+          version: 0.16.0
 
       - name: Build (Debug)
         run: zig build
@@ -53,10 +53,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Install Zig 0.15
+      - name: Install Zig 0.16
         uses: mlugg/setup-zig@v2
         with:
-          version: 0.15.2
+          version: 0.16.0
 
       - name: Build shared library
         run: zig build
@@ -78,10 +78,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Install Zig 0.15
+      - name: Install Zig 0.16
         uses: mlugg/setup-zig@v2
         with:
-          version: 0.15.2
+          version: 0.16.0
 
       - name: Build (ReleaseFast)
         run: zig build -Doptimize=ReleaseFast

.gitignore

@@ -1,14 +1,27 @@
 zig-out/
+zig-out-ffi/
+zig-out-*/
 .zig-cache/
 python/dist/
 *.egg-info/
 codedb.snapshot
 __pycache__/
 *.pyc
+.DS_Store
+.claude/
+.zigrep_archive
+run_status.txt
+turbodb_data/
+charts.png
 
 # ClickHouse local data
 access/
 preprocessed_configs/
 
-# Optimization docs (tracked)
-# optimizations/
+# Local agent/MCP workspaces
+optimizations/
+MCP/
+architecutre/
+LOCAL_README.md
+local-agent-readme.md
+benchmark-results/

AGENTS.md

@@ -0,0 +1,34 @@
+# Agent Runtime Rules
+
+## Container-Only Runtime
+
+Do not run TurboDB, `turbodb-gateway`, database services, long-lived servers,
+stress tests, or deployment simulations directly on the host macOS machine.
+
+Runtime work must use Apple `container`:
+
+```sh
+container --help
+```
+
+Reference: https://github.com/apple/container
+
+Allowed on the host:
+
+- Editing files.
+- Inspecting files.
+- Building binaries when no server is started.
+- One-shot commands that do not bind ports, create launch agents, or start
+  persistent services.
+
+Not allowed on the host:
+
+- `launchctl` service installation for TurboDB or its gateway.
+- Starting `turbodb` or `turbodb-gateway` directly on macOS.
+- Binding TurboDB, gateway, or test services to host ports.
+- Moving live database data into host-managed service directories.
+
+Before any database runtime, create or use an Apple `container` environment and
+mount the intended data directory explicitly. Database updates must preserve
+existing data: take a backup/snapshot first, reuse the existing data directory,
+and never delete or reinitialize data unless the user explicitly asks for that.

README.md

@@ -7,7 +7,7 @@
   <p align="center">
     <a href="https://pypi.org/project/turbodatabase/"><img src="https://img.shields.io/pypi/v/turbodatabase?color=3776AB&logo=python&label=PyPI" alt="PyPI"></a>
     <a href="https://www.npmjs.com/package/turbodatabase"><img src="https://img.shields.io/npm/v/turbodatabase?color=339933&logo=npm&label=npm" alt="npm"></a>
-    <img src="https://img.shields.io/badge/zig-0.15-F7A41D?logo=zig" alt="Zig 0.15">
+    <img src="https://img.shields.io/badge/zig-0.16-F7A41D?logo=zig" alt="Zig 0.16">
     <img src="https://img.shields.io/badge/license-MIT-blue" alt="MIT">
     <img src="https://img.shields.io/badge/status-alpha-orange" alt="Alpha">
   </p>
@@ -118,7 +118,7 @@ npm install turbodatabase     # Node.js
 ### Build from source
 
 ```bash
-# Requires Zig 0.15+
+# Requires Zig 0.16+
 git clone https://github.com/justrach/turbodb
 cd turbodb
 zig build
@@ -353,7 +353,7 @@ bash bench/setup_shard_bench.sh         # Full shard comparison (needs Docker +
 
 ## Contributing
 
-Contributions welcome! TurboDB is written in Zig 0.15 with no external dependencies.
+Contributions welcome! TurboDB is written in Zig 0.16 with no external dependencies.
 
 ## License
 

bench/README_apple_container_matrix.md

@@ -0,0 +1,109 @@
+# Apple Container Benchmark Matrix
+
+This harness compares TurboDB against PostgreSQL 18, MySQL, and optionally
+TigerBeetle using the same generated application shape. TurboDB is measured two
+ways: over HTTP and through the embedded C ABI/FFI bridge.
+
+It must be run through Apple `container`. The runner creates a private container
+network, starts each database in a container, runs the Python benchmark client in
+another container, and does not publish database ports to macOS.
+
+## Workload Shape
+
+- `users`: keyed user records with an email and JSON profile.
+- `orders`: keyed order records with `user_id`, amount, status, and JSON payload.
+- TurboDB stores `bench_users`, `bench_orders`, and a materialized
+  `bench_user_orders` edge collection. The benchmark client uses persistent
+  HTTP plus durable `POST /db/:col/bulk` ingestion by default. Use
+  `--turbodb-bulk-mode binary` to benchmark length-prefixed
+  `POST /db/:col/bulk_binary` ingestion. It uses
+  `POST /db/:col/batch_get?mode=count` for relationship fetches, so
+  lookup-heavy workloads do not pay to serialize document bodies they do not
+  consume. For the join-like, update, and delete workloads it assumes the
+  faster HTTP shapes exist and uses `POST /db/:edge_col/join?mode=count`,
+  `POST /db/:col/batch_update`, and `POST /db/:col/batch_delete`.
+- `turbodb_ffi` loads `/work/zig-out-ffi/lib/libturbodb.so` with Python `ctypes`.
+  This is an embedded/raw-engine path, not a network database path, and exists
+  to show the cost of HTTP/client transport separately from storage operations.
+  Bulk inserts, updates, deletes, and batched reads use C ABI batch exports so
+  the benchmark does not hide storage behavior behind one Python FFI call per
+  row.
+- PostgreSQL 18 and MySQL use normalized `users` and `orders` tables with a
+  secondary index on `orders.user_id`.
+- TigerBeetle uses accounts and transfers, so relationship lookups are modeled
+  as account transfer queries rather than SQL/document joins. Updates and
+  deletes are reported as not applicable because transfers are immutable.
+
+## Workloads
+
+- `ingest`: create all users and orders. TurboDB HTTP uses a collection-level
+  bulk insert path that writes committed WAL entries as a batch and flushes the
+  WAL before acknowledging the bulk request.
+- `point_get`: read users by primary key.
+- `relationship_lookup`: fetch all orders/transfers for a user. TurboDB HTTP
+  keeps this as a materialized edge read plus compact `batch_get`.
+- `join_or_join_like`: SQL join for PostgreSQL/MySQL, materialized edge fetches
+  for TurboDB FFI, compact `POST /db/:edge_col/join` for TurboDB HTTP, account
+  transfer query for TigerBeetle.
+- `update_orders`: update order status where the engine supports mutation.
+- `delete_orders`: delete orders where the engine supports deletion.
+
+TurboDB HTTP update/delete latency is reported as amortized row latency when
+the benchmark uses the batch endpoints; throughput remains row operations per
+second.
+
+TurboDB also exposes a gRPC-compatible bridge for binary bulk mutations on the
+same server:
+
+- `POST /grpc/:col/BulkInsert`
+- `POST /grpc/:col/BulkUpdate` / `BulkUpsert`
+- `POST /grpc/:col/BulkDelete`
+
+The bridge accepts one uncompressed gRPC message frame for `application/grpc`
+requests, or the raw binary payload directly. It is not a full HTTP/2/protobuf
+server yet; it is a fast route shape for clients/gateways that already speak in
+gRPC-style frames.
+
+## Run
+
+Small smoke run:
+
+```sh
+python3 bench/run_apple_container_bench.py \
+  --users 50 \
+  --orders-per-user 3 \
+  --samples 25 \
+  --skip-tigerbeetle
+```
+
+Larger run with all engines:
+
+```sh
+python3 bench/run_apple_container_bench.py \
+  --users 10000 \
+  --orders-per-user 5 \
+  --samples 2000
+```
+
+Useful flags:
+
+- `--skip-turbodb`, `--skip-postgres`, `--skip-mysql`, `--skip-tigerbeetle`
+- `--skip-turbodb-ffi`
+- `--turbodb-bulk-mode ndjson|binary`
+- `--batch-size 16384` to align with TurboDB's larger default durable bulk
+  chunk and reduce per-request/WAL-flush overhead on large ingests
+- `--turbodb-ffi-target aarch64-linux-gnu`
+- `--turbodb-ffi-prefix zig-out-ffi`
+- `--turbodb-ffi-lib /work/zig-out-ffi/lib/libturbodb.so`
+- `--mysql-image mysql:8.4`
+- `--postgres-image postgres:18`
+- `--tigerbeetle-image ghcr.io/tigerbeetle/tigerbeetle:latest`
+- `--tigerbeetle-memory 2G` because the replica journal needs more than the
+  Apple container 1 GiB default in practice.
+- `--keep` to leave containers, network, and volumes behind for inspection.
+
+Results are written under ignored `benchmark-results/`.
+
+PostgreSQL 18 is mounted at `/var/lib/postgresql`, not
+`/var/lib/postgresql/data`, because the official image stores data in
+versioned subdirectories for upgrade compatibility.