Skip to content

Commit 38bf642

Browse files
michaelnealemichaelneale
authored
ci: prove the relay-driven mesh lifecycle — discover, join, infer, deny — with real nodes (#3862)
## Summary CI now proves the full Buzz shared-compute join story end to end: a member can discover another member's served model **through the Buzz relay alone** and run inference over the mesh, while a non-member gets nothing — the relay rejects its auth, and the mesh refuses to route for it even holding a leaked endpoint address. This is deliberately different from mesh-llm's own CI smokes (which bootstrap two nodes with a hand-carried invite token / mdns): here the **relay is the control plane**, exactly like the desktop app: 1. **Membership** — identities A and B are added via `buzz-admin` (kind:13534 NIP-43 roster); C is not. 2. **Advertise** — each member publishes a client-signed kind:30003 discovery note carrying its MeshLLM owner binding and (for the serve node) `serveTargets[].endpointAddr`, covered by an endpoint-binding signature — the exact payload shape the desktop coordinator publishes. 3. **Trust** — the serve node derives its admission allowlist from the relay (statuses ∩ roster) and requires the **exact expected {A, B} owner-id set** before starting with `TrustPolicy::Allowlist`. 4. **Join** — the client verifies owner + endpoint bindings and membership, then dials the relay-discovered endpoint (the desktop join-watcher's `dial_endpoint_addr` step). No out-of-band token. 5. **Infer** — a chat completion against the client's local OpenAI endpoint routes over QUIC to the serve node's model (CPU, SmolLM2-135M, ~105MB). 6. **Deny (differential)** — the stranger's NIP-42 auth must fail with the relay's own membership rejection (`restricted: not a relay member` — successful auth or any unrelated connect error fails the run), and dialing the leaked endpoint must not produce a routed inference — **while the trusted client re-proves inference immediately afterwards**, so a dead serve node can't masquerade as an admission denial. ## What's in the PR - `crates/buzz-relay/examples/mesh_relay_lifecycle_smoke.rs` — the harness. One process per node (mesh-llm keeps process-global state under `~/.mesh-llm`), orchestrator + serve/client/stranger roles, byte-identical binding payloads to `desktop/src-tauri/src/mesh_llm/identity.rs` (called out with keep-in-sync comments). Child stdout is pumped through a reader thread so every wait has a hard deadline; timed-out children are killed; exit statuses are checked. - `scripts/ci-mesh-lifecycle-smoke.sh` — provisions a membership-gated relay (throwaway owner + signing identities via `buzz-admin generate-key`), runs the harness, cleans up. Fails fast if :3000 is already occupied (a stale open relay would mask gating). - `scripts/start-relay-for-tests.sh` — gains opt-in NIP-43 membership env passthrough (`BUZZ_REQUIRE_RELAY_MEMBERSHIP` + `RELAY_OWNER_PUBKEY` + `BUZZ_RELAY_PRIVATE_KEY`). Default behavior unchanged. - `.github/workflows/mesh-lifecycle.yml` — separate, path-filtered, non-required workflow (mesh paths, the harness's dependency crates, `Cargo.lock`, dispatch), pinned to `ubuntu-24.04`. Caches the mesh native runtime + HF model keyed on the lockfile hash, so a mesh pin bump rolls the runtime cache. Uploads relay + harness logs on failure. ## Scope This is an **independent protocol harness**: it speaks the same wire protocol and payload shapes as the desktop but re-implements the binding/verification logic (the desktop crate is outside the workspace). Regressions inside the desktop's own discovery filtering are the desktop unit tests' job; what this smoke proves is that the relay + mesh-llm SDK + admission stack support the lifecycle end to end. ## Relationship to mesh-llm's CI Follows the shape mesh-llm's own CI proved stable (tiny CPU model, one runner, multiple real mesh-llm processes over real QUIC — cf. their `ci-two-node-client-serving-smoke.sh`), but swaps the token bootstrap for the relay-driven lifecycle, which is the part only Buzz can test. ## Validation Green on GitHub Actions (ubuntu-24.04) across three runs, including after rebases onto the mesh v0.74 upgrade (#3467) and latest main: ``` PASS 1/6: relay-derived allowlist is exactly {A, B} PASS 2/6: serve member ready + advertised model: jc-builds/SmolLM2-135M-Instruct-Q4_K_M-GGUF:Q4_K_M PASS 3/6: client member discovered + joined via relay PASS 4/6: inference routed over the mesh: "PONG" PASS 5/6: relay rejected the stranger's NIP-42 auth (membership gate) PASS 6/6: stranger denied (gossip visible, inference rejected: 503 all tunnels failed) while trusted inference still routes PASS: full relay-driven mesh lifecycle verified ``` Also validated locally on macOS. `cargo fmt --all --check` and `cargo clippy -p buzz-relay --all-targets -- -D warnings` pass. ## Notes - The harness follows the repo's mesh `[dev-dependencies]` pin automatically, so it doubles as a canary for future mesh upgrades (it already caught the v0.73.1 → v0.74.0 bump during development). - The stranger "deny" accepts either shape mesh-llm exhibits: no model visibility at all, or gossip visibility with inference refused — mesh-llm applies the receiving node's owner policy after the gossip handshake, so admission gates *routing*, not gossip. The differential trusted-inference re-check (PASS 6/6) is what makes that a real denial rather than a dead server. - Model-visibility windows are tunable via `MESH_CLIENT_WINDOW_SECS` / `MESH_STRANGER_WINDOW_SECS` if shared runners prove slow — pin a longer window in the workflow env rather than re-running the job. --------- Signed-off-by: Michael Neale <michael.neale@gmail.com>
1 parent e2796d4 commit 38bf642

6 files changed

Lines changed: 1311 additions & 0 deletions

File tree

‎.github/workflows/mesh-lifecycle.yml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
name: Mesh Lifecycle
2+
# Relay-driven mesh lifecycle smoke: membership → signed discovery notes →
3+
# relay-derived allowlist → join → CPU inference over QUIC → stranger denied
4+
# (relay membership rejection + no routed inference, with a differential
5+
# trusted-inference health proof so a dead serve node can't fake a denial).
6+
# Runs the full Buzz "shared compute" join story with three real mesh-llm
7+
# node processes on one runner, using the Buzz relay as the control plane
8+
# (no hand-carried invite tokens). Mirrors the shape mesh-llm's own CI uses
9+
# for its two-node smokes (tiny CPU model, one runner, real QUIC mesh).
10+
11+
on:
12+
push:
13+
branches: [main]
14+
paths:
15+
- 'crates/buzz-relay/examples/mesh_*.rs'
16+
- 'crates/buzz-relay/Cargo.toml'
17+
- 'crates/buzz-admin/**'
18+
- 'crates/buzz-test-client/**'
19+
- 'crates/buzz-ws-client/**'
20+
- 'Cargo.lock'
21+
- 'desktop/src-tauri/src/mesh_llm/**'
22+
- 'scripts/ci-mesh-lifecycle-smoke.sh'
23+
- 'scripts/start-relay-for-tests.sh'
24+
- '.github/workflows/mesh-lifecycle.yml'
25+
pull_request:
26+
paths:
27+
- 'crates/buzz-relay/examples/mesh_*.rs'
28+
- 'crates/buzz-relay/Cargo.toml'
29+
- 'crates/buzz-admin/**'
30+
- 'crates/buzz-test-client/**'
31+
- 'crates/buzz-ws-client/**'
32+
- 'Cargo.lock'
33+
- 'desktop/src-tauri/src/mesh_llm/**'
34+
- 'scripts/ci-mesh-lifecycle-smoke.sh'
35+
- 'scripts/start-relay-for-tests.sh'
36+
- '.github/workflows/mesh-lifecycle.yml'
37+
workflow_dispatch:
38+
39+
concurrency:
40+
group: mesh-lifecycle-${{ github.event_name == 'pull_request' && github.ref || github.sha }}
41+
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
42+
43+
env:
44+
CARGO_TERM_COLOR: always
45+
46+
jobs:
47+
lifecycle-smoke:
48+
name: Relay-Driven Mesh Lifecycle Smoke
49+
runs-on: ubuntu-24.04
50+
timeout-minutes: 45
51+
permissions:
52+
contents: read
53+
steps:
54+
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
55+
56+
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
57+
58+
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
59+
with:
60+
save-if: ${{ github.event_name != 'pull_request' }}
61+
62+
# The mesh-llm SDK downloads a signed native runtime (llama.cpp CPU
63+
# build) on first init, and the serve node downloads the smoke model
64+
# from HuggingFace on first run. Key on the lockfile so a mesh pin bump
65+
# rolls the runtime cache; the model ref is stable.
66+
- name: Restore mesh runtime + model caches
67+
id: mesh-caches
68+
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
69+
with:
70+
path: |
71+
~/.cache/mesh-llm/native-runtimes
72+
~/.cache/huggingface/hub
73+
key: mesh-lifecycle-${{ runner.os }}-smollm2-135m-${{ hashFiles('Cargo.lock') }}
74+
restore-keys: |
75+
mesh-lifecycle-${{ runner.os }}-smollm2-135m-
76+
77+
- name: Start integration services
78+
run: |
79+
for attempt in 1 2 3; do
80+
if docker compose up -d postgres redis minio minio-init; then
81+
break
82+
fi
83+
if [ "$attempt" -eq 3 ]; then
84+
echo "docker compose up failed after 3 attempts" >&2
85+
exit 1
86+
fi
87+
echo "docker compose up failed (attempt $attempt), retrying in $((attempt * 5))s..." >&2
88+
sleep $((attempt * 5))
89+
done
90+
91+
- name: Run relay-driven mesh lifecycle smoke
92+
run: ./scripts/ci-mesh-lifecycle-smoke.sh 2>&1 | tee /tmp/mesh-lifecycle-harness.log
93+
94+
- name: Save mesh runtime + model caches
95+
if: github.ref == 'refs/heads/main' && steps.mesh-caches.outputs.cache-hit != 'true'
96+
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
97+
with:
98+
path: |
99+
~/.cache/mesh-llm/native-runtimes
100+
~/.cache/huggingface/hub
101+
key: mesh-lifecycle-${{ runner.os }}-smollm2-135m-${{ hashFiles('Cargo.lock') }}
102+
103+
- name: Upload relay + harness logs
104+
if: failure()
105+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
106+
with:
107+
name: mesh-lifecycle-logs
108+
path: |
109+
/tmp/buzz-relay.log
110+
/tmp/mesh-lifecycle-harness.log
111+
if-no-files-found: ignore

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎crates/buzz-relay/Cargo.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,11 @@ dev = ["buzz-auth/dev"]
8686
[dev-dependencies]
8787
mesh-llm-sdk = { git = "https://github.com/Mesh-LLM/mesh-llm.git", tag = "v0.74.0", package = "mesh-llm-sdk", default-features = false, features = ["client", "serving"] }
8888
mesh-llm-host-runtime = { git = "https://github.com/Mesh-LLM/mesh-llm.git", tag = "v0.74.0", package = "mesh-llm-host-runtime", default-features = false, features = ["dynamic-native-runtime"] }
89+
# Relay-driven mesh lifecycle smoke (examples/mesh_relay_lifecycle_smoke.rs):
90+
# the relay client for discovery notes and the exact ed25519 the mesh owner
91+
# keys use for binding verification.
92+
buzz-test-client = { path = "../buzz-test-client" }
93+
ed25519-dalek = "=3.0.0-rc.0"
8994
buzz-core = { workspace = true, features = ["test-utils"] }
9095
buzz-auth = { workspace = true, features = ["dev"] }
9196
reqwest = { workspace = true }

0 commit comments

Comments
 (0)