Skip to content

chore: mesh upgrade, clean up legacy special case code, simplify model selection for mesh - #5289

Open
micspiral wants to merge 10 commits into
mainfrom
micspiral/mesh-llm-0.75
Open

chore: mesh upgrade, clean up legacy special case code, simplify model selection for mesh#5289
micspiral wants to merge 10 commits into
mainfrom
micspiral/mesh-llm-0.75

Conversation

@micspiral

@micspiral micspiral commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Shared compute now has exactly two model choices: MeshLLM's virtual mesh
model, or a model you name. Buzz picks between them in one place, and
buzz-agent no longer knows meshes exist.

What changed

  • MeshLLM v0.74.0 → v0.75.1. v0.75.0 added degrade_to_single_model, so a
    model=mesh request is answered by one served model when there is no
    committee to form, instead of failing. v0.75.1 adds Mesh-LLM#1196, which
    skips stale pre-0.75 runtime cache entries rather than aborting startup on
    them — without it, anyone who had run mesh on 0.73/0.74 could not start.
  • Deleted the client-side mesh catalog probe. buzz-agent used to poll
    /v1/models (5s TTL, 30s cooldown, two-observation debounce) to decide
    whether mesh was safe to send. MeshLLM now decides per request, so the
    polling, its hysteresis, and its 503 fallback are gone.
  • One mapping point. relay_mesh_wire_model() turns the stored value into
    a wire name: auto becomes mesh, a named model passes through. The spawn
    env, the ACP harness, and the readiness probe all use it, so they cannot
    disagree — previously BUZZ_ACP_MODEL and the probe both said auto, a name
    the mesh does not advertise.
  • Removed the nostr-relay-pool advisory exception. ci(security): allow retired relay pool advisory #5404 allowed
    RUSTSEC-2026-0243 "after mesh-llm migrates to nostr-sdk >= 0.45". v0.75.1
    does, so the retired crate is gone from both lockfiles and the exception
    would only mask a future advisory for it.
  • Deleted scripts/ensure-mesh-native-runtime.sh and its six justfile call
    sites. It built llama.cpp from source into the runtime cache; the app already
    downloads the signed release runtime itself, and CI never called it.

Why it is better

−639 lines of Rust. Availability is decided by the node that knows the
answer, per request, instead of by a client cache that could be stale for up to
30 seconds. A second worker joining now takes effect on the next request rather
than after two confirming probes.

Behaviour change

A 503 on an explicit mesh request takes the ordinary transport retry under
the same model instead of failing over to a second one — there is no second
model to fail over to now. MoA repairs partial committee results internally
before it reaches that point.

Validation

crates/buzz-relay/examples/mesh_agent_e2e.rs now sends mesh where it
previously sent auto or the physical model id, so no leg was covering what
Buzz actually puts on the wire. 4/4 on gemma-4-E4B, gemma-4-26B-A4B, and
Qwen3-8B — including a real ACP tool call through mesh into buzz-dev-mcp,
asserted by reading the written file back off disk.

Hand-tested in the desktop app on both gemma-4 sizes: picked Auto, agent logged
model_id=mesh, replied in channel.

Not covered

A committee that forms and then loses a worker returns 502, and that needs two
workers to reproduce — not testable on one machine.

Bumps the eight mesh-llm pins (six in the desktop Tauri crate, two in
buzz-relay) from v0.74.0 to v0.75.0 and refreshes both lockfiles.

Also deletes scripts/ensure-mesh-native-runtime.sh and its six call sites.
The script built llama.cpp from source locally to pre-seed a native-runtime
cache, but nothing needs that:

- The app installs the signed runtime itself. `initialize_host_runtime()`
  reaches `install_native_runtime()`, whose `allow_download` defaults to
  true, and v0.75.0 publishes `native-runtimes.json` (13 artifacts,
  including darwin-aarch64-metal) alongside the release. Verified by
  fetching the manifest and artifact and checking both sha256 sidecars.
- CI never called it. The mesh lifecycle workflow caches the app's own
  `~/.cache/mesh-llm/native-runtimes` and says so in a comment: "the
  mesh-llm SDK downloads a signed native runtime on first init". The
  desktop build job compiles llama.cpp through mesh-llm's own
  prepare-llama.sh / build-llama.sh.
- The three `mesh-e2e-*` recipes that also used it are hand-run only,
  referenced nowhere else in the repo.

`mesh=1` behaviour is unchanged; only the pre-seed line is gone from
dev / staging / production.

KNOWN ISSUE (do not merge before resolving): on a machine that already has
a pre-0.75 runtime cache, startup fails with "native runtime artifact ...
does not declare file checksums". v0.75.0 added
`NativeRuntimeManifest::verify_contents`, which requires per-file
checksums; caches written by older loaders have none, and the startup path
enumerates the whole cache strictly, so one stale entry aborts the scan
even when a valid 0.75.0 entry is present. Reproduced and bisected here:
failing with the stale entries present, passing with only 0.75.0 cached,
and passing with the stale entries present once a bundle dir is supplied.
That is why the installed CLI is unaffected -- it ships a runtime directory
beside its binary, which is discovered leniently.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
The runbook told readers to run scripts/ensure-mesh-native-runtime.sh,
which this branch removes. The desktop installs the signed runtime itself
on first init, so the only manual step left is clearing stale pre-0.75
cache entries.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
@micspiral micspiral self-assigned this Aug 8, 2026
…ttee

Shared-compute `auto` now maps straight onto MeshLLM's virtual `mesh` model
and sends it unconditionally. MeshLLM >= 0.75.0 resolves that per request:
a Mixture-of-Agents committee when two or more workers are reachable, and a
single served model when they are not
(`moa_gateway::degrade_to_single_model`). Before 0.75.0 a `model=mesh`
request could 503 on a lone node, so the agent kept its own hysteretic view
of whether a committee was currently possible and only then dared send
`mesh`.

That client-side machinery is now redundant, and removing it deletes the
polling and the debounce it needed to avoid flapping:

- `resolve_openai_model`, `observe_mesh_virtual_model`,
  `cool_down_collective`, `mesh_catalog_supports_collective`
- `MeshAutoState`, `MeshCatalogObservation`, and the `Llm` field holding it
- the `/models` probe: 5s catalog TTL, 2s probe timeout, 30s cooldown, and
  the two-observation confirmation count
- `PostError::MeshFallback` with `is_mesh_moa_unavailable_body` /
  `is_mesh_moa_failure_body` and the `detect_mesh_fallback` parameter
- `looks_like_unstructured_tool_call`, whose only call site was gated on the
  same adaptive-mesh flag

`looks_like_unstructured_tool_call` arrived with this feature (#2825) rather
than as a general guard, so it leaves with it. It retried once through `auto`
when a committee answered tool-call markup as prose; if that turns out to
matter in its own right it should come back provider-agnostic, not gated on
shared compute.

Behaviour that is deliberately unchanged: an explicit model is still sent
verbatim and never rewritten, and plain OpenAI `auto` (a real provider model
name) is still left alone. What does change is that a 503 from an explicit
`mesh` request now takes the ordinary transport retry under the same model
instead of failing fast to a second model -- there is no second model to
fall back to once MeshLLM resolves `mesh` itself.

Three test helpers (`moa_failure`, `model_catalog`,
`complete_model_with_tool`) went unused once the fourteen catalog/debounce
tests were removed. New tests pin the surviving contract: shared-compute
`auto` sends `mesh` and never requests `/v1/models`, plain `auto` is
untouched, and an explicit model keeps its name across retries.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
v0.75.1 carries the upstream fix (Mesh-LLM/mesh-llm#1196): startup now scans
the native-runtime cache leniently (`installed_lenient`), skipping unusable
entries instead of aborting the whole scan on the first one.

That removes the blocker this branch has carried since the v0.75.0 bump. On a
machine with a pre-0.75 cache, v0.75.0 failed with:

  native runtime artifact meshllm-native-runtime-darwin-aarch64-metal
  does not declare file checksums

because caches written by older loaders have no per-file checksums, and the
strict enumeration covered every version under the cache root.

Verified on a machine that still has the offending entries in place
(0.73.1, 0.74.0, 0.74.0-rc8 all with `runtime.files = 0`), with no bundle
directory and no MESH_LLM_BUILD_VERSION override:

- no checksum abort
- the v0.75.1 runtime installed itself alongside the stale entries
- crates/buzz-relay/examples/mesh_agent_e2e passed 4/4 (explicit-model chat,
  auto-model chat, oversized-budget rejection, agentic tool use)

Pins move in both manifests (six mesh-llm crates in the desktop Tauri crate,
two in buzz-relay) with both lockfiles refreshed. The v0.75.1 release
publishes native-runtimes.json with 13 artifacts including
darwin-aarch64-metal at Skippy ABI 0.1.35, which matches what the crates
compile against.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
Shared compute either uses MeshLLM's virtual `mesh` model or a model the user
named. Nothing else. That decision now happens once, in the desktop
translation layer that already turns the native provider into an
OpenAI-compatible transport:

    stored "auto" (or blank legacy)  ->  BUZZ_AGENT_MODEL=mesh
    stored "<a model>"               ->  BUZZ_AGENT_MODEL=<a model>

MeshLLM resolves `mesh` per request -- a Mixture-of-Agents committee when two
or more workers are reachable, a single served model when they are not
(`moa_gateway::degrade_to_single_model`) -- so the caller never has to know
which of those will happen.

buzz-agent therefore needs no mesh concepts at all. Removed:

- the `auto` -> `mesh` mapping in `openai_request`
- `MESH_VIRTUAL_MODEL_ID` / `MESH_AUTO_MODEL_ID`
- `Config::prefer_mesh_for_auto` and the `BUZZ_AGENT_PREFER_MESH_FOR_AUTO`
  env var that fed it

`openai_request` now sends the configured model verbatim, which is what it
does for every other provider. `auto` stays a real provider model name for
plain OpenAI hosts, unchanged, because nothing rewrites it any more.

The previous commit moved this decision from a `/models` probe into a
compile-time mapping; this moves it out of the agent entirely. The stored
value and the picker label both stay "auto" -- that is the user's word for
"let the mesh decide", and it is still what gets persisted.

Tests: `relay_mesh.rs` gains three that pin the translation (stored `auto`
and blank both become `mesh`; a named model is sent verbatim). In
`llm.rs`, the two mapping tests are replaced by one asserting the configured
model reaches the wire unchanged for `auto`, `mesh`, and a named model, with
no catalog request on any of them.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
Two env vars told the agent process which model to use, and only one of them
was translated. `BUZZ_AGENT_MODEL` said `mesh`; `BUZZ_ACP_MODEL` was written
straight from the stored value and still said `auto`
(`runtime.rs`, before the mesh override runs). `auto` is not a model the mesh
advertises, so buzz-acp rejected it as unavailable and fell back to its default
with a warning on every new session — recoverable, but the two consumers were
being told different things.

`relay_mesh_wire_model` is now the one place a stored value becomes a wire
name: `auto` (and a blank legacy value) becomes MeshLLM's virtual `mesh` model,
anything else is a model the user named and passes through unchanged. Both the
LLM transport env and the ACP harness go through it, so they cannot disagree.

Also narrows a comment I had overstated. Sending `mesh` is not "always safe":
`degrade_to_single_model` is a pre-flight capacity decision, so a committee that
forms and then loses a worker still surfaces as a failed turn. MoA repairs
partial results internally (`repair_tool_result_answer`,
`fallback_arguments`) before it reaches that point, which is why no client-side
reroll is being added back.

The headless harness now exercises the shipped path. P2, P3, and P4 send `mesh`
where they previously sent `auto` (MeshLLM's own router) or the physical model
id (bypassing mesh routing entirely) — so no leg was covering what Buzz
actually puts on the wire. With one served model there is no committee, which
makes these a direct test of the degradation path.

Verified on this machine at this commit, three models, 4/4 each:
gemma-4-E4B-it-Q4_K_M, gemma-4-26B-A4B-it-UD-Q4_K_M, Qwen3-8B-Q4_K_M. P4 is
the load-bearing one: a real ACP tool call through `mesh` into buzz-dev-mcp,
asserted by reading `BUZZ_OK` back off disk rather than trusting the agent's
own account of itself. P3 confirms the wire name in its error text
(`llm: (mesh) exhausted retries: 503`).

The harness comment claiming it supplies "exactly" what `apply_relay_mesh_env`
does was wrong — it omits `BUZZ_AGENT_REQUIRE_REPLY`, which needs Buzz's
publish tools to mean anything. It now says transport subset.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
origin/main added `post()` call sites carrying `detect_mesh_fallback`, which
this branch removed along with `PostError::MeshFallback`. Neither side touched
the same lines, so the merge applied cleanly and did not compile — the failure
only appears in `--all-targets`, which is why per-crate `cargo check` missed
it and CI (testing the merge result) did not.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
`wait_for_mesh_inference` posted the caller's stored model id. For
shared-compute `auto` that is not a name the mesh advertises, so readiness
validated a route no agent uses — and could fail startup while the real route
worked. It now translates through `relay_mesh_wire_model`, the same mapping the
spawn env and the ACP harness use. Named models pass through unchanged, so the
serve-side callers that already pass a real model are unaffected.

Translating inside the probe rather than at its seven call sites keeps the
mapping in one place, which was the point of the previous commit.

Also corrects the e2e runbook: it still told readers to clear stale pre-0.75
runtime cache entries by hand, which is exactly what pinning v0.75.1 removed the
need for.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
@micspiral micspiral changed the title chore(mesh): move to MeshLLM v0.75.0 and stop pre-seeding the native runtime Shared compute: one model decision, made by the mesh Aug 10, 2026
@micspiral
micspiral marked this pull request as ready for review August 10, 2026 08:39
@micspiral
micspiral requested a review from a team as a code owner August 10, 2026 08:39
@michaelneale michaelneale changed the title Shared compute: one model decision, made by the mesh chore: mesh upgrade, clean up legacy special case code, simplify model selection for mesh Aug 10, 2026
…n retired

#5404 allowed RUSTSEC-2026-0243 with the removal condition written into the
comment: "Remove after mesh-llm migrates to nostr-sdk >= 0.45, which absorbed
the standalone relay pool." MeshLLM v0.75.1 does exactly that, so
nostr-relay-pool is no longer in either lockfile and the advisory has nothing
to fire on.

That PR measured 13 MeshLLM API compilation errors from bumping nostr-sdk to
0.45 against v0.74; upstream did the migration in v0.75, which is why this
needs no source change here.

Verified with the same commands #5404 used:
bin/cargo-deny --locked check --config deny.toml advisories, and
bin/cargo-deny --locked check -- advisories ok, bans ok, licenses ok, sources ok.

Leaving a satisfied exception in place would suppress a real future advisory
for this crate, so it goes with the bump that retired it rather than waiting for
someone to notice.

Signed-off-by: Michael Neale <michael.neale@gmail.com>

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on Wes's behalf at 8e3b15019d09a17de0cd21144ac62b366f7ac5d7.

No blocking findings.

I traced the effective provider/model resolution through EffectiveAgentConfig::relay_mesh_model_id, spawn_agent_child, the mesh process env, BUZZ_ACP_MODEL, and wait_for_mesh_inference. Stored auto/legacy blank consistently becomes the wire model mesh; explicit model names stay explicit. Linked definitions and global inheritance continue to use the existing effective-config precedence rather than stale instance bytes.

I also checked the removed buzz-agent catalog/hysteresis/fallback path, all six deleted Justfile pre-seed call sites, both dependency manifests/lockfiles, the retired advisory exception, and the focused e2e changes. The removed fallback's behavioral change is accurately disclosed: terminal committee failure retries the same mesh route rather than changing models. MeshLLM 0.75's single-worker degradation makes the old eligibility fallback obsolete, while an already-formed committee losing a worker remains a known two-worker failure case.

Validation evidence: git diff --check passed locally on the exact head; all GitHub checks are green, including Unit Tests, Rust Lint, Desktop Core, Security, Relay E2E, and Relay-Driven Mesh Lifecycle Smoke. I did not duplicate the broad CI suite locally.

One non-blocking caveat: BUZZ_ACP_MODEL remains intentionally user-overridable through layered env, so “cannot disagree” applies to Buzz's derived defaults, not an explicit power-user override.

@wpfleger96

Copy link
Copy Markdown
Member

🤖 Reviewed at head 8e3b150 — no blocking concerns, but I think the automesh translation at the top of wait_for_mesh_inference (desktop/src-tauri/src/commands/mesh_llm.rs:603) left the catalog-visibility check just below it slightly stale.

The branch at line ~632 still compares model_id == crate::mesh_llm::AUTO_MODEL_ID, but after the translation model_id can only ever be mesh or a named model, so that comparison looks unreachable now. The practical effect: mesh-llm 0.75.1 only advertises the virtual mesh model in /v1/models when there are ≥2 non-mesh models (should_advertise_virtual_mesh in moa_gateway/context_selection.rs), so on a single-worker mesh with stored auto, model_ever_visible stays false for the whole wait. If readiness then times out (e.g. the lone worker is still loading weights), the failure gets classified as CatalogNeverSynced — the "network path problem" message — when RoutingNeverCompleted ("found the host, try again shortly") is the truth. Before this PR the any-model-counts rule for auto classified that case correctly.

The happy path is fine — the chat probe sends model: mesh and degrade_to_single_model handles it server-side — so this is only about which error message a user sees on a readiness timeout. I think the fix is a one-liner: compare against managed_agents::RELAY_MESH_VIRTUAL_MODEL_ID instead, keeping the same "delegated choice — any advertised model counts" semantics keyed on the translated name. That would also stop the failure text from quoting "mesh" at a user who picked "Auto".

Happy to be corrected if the single-worker catalog behaves differently in practice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants