Skip to content

conformance: measure the runtime through an ssh tunnel, not the Lium port proxy - #1739

Merged
anderdc merged 4 commits into
testfrom
conformance-lium-ls-flake
Sep 1, 2026
Merged

conformance: measure the runtime through an ssh tunnel, not the Lium port proxy#1739
anderdc merged 4 commits into
testfrom
conformance-lium-ls-flake

Conversation

@anderdc

@anderdc anderdc commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Three of the four failures since the last conformance run were the harness, not the runtime — and the one pass it produced (#1740) carried a value that would have failed every honest miner.

R6 failed twice on 91.224.44.226 with an image that passed on 91.224.44.81 on 8/31. Rented that host with ssh and measured: a 2 s HTTP handler answers 16 parallel requests in 2.4 s … 38.5 s through the Lium-mapped port and 2.2 s … 2.2 s on-box — the port proxy caps concurrent connections at ~5 per pod (5 clean, 6+ serialise). That serialises the 16-wide overload and speed bursts: a false R6 failure, and an aggregate_decode_tps of 25.9/46.9 against 263 — the number the validator has priced tokens from since #1737 ($0.69/M would have read $7.5/M). Through one ssh connection the same 32 parallel finish in 2.5 … 3.6 s, so the checker now reaches the runtime over ssh -L 18080:127.0.0.1:8080. Lium's SDK registers only ~/.ssh/id_ed25519 with a pod (it ignores LIUM_SSH_KEY_PATH and the config file), so the workflow generates the key there. Run 33532638861 on the same host: CONFORMANT, R6 16×200 in 32.4 … 32.9 s.

attest.vram_model_reserved_bytes was being measured on a card without the model. The attest pod is a bare gt-attest:v1 card, so the checker reported the attest server's own 573 MB; in the release that makes attest.py expect ~32 GB free on a 5090, and a card holding the 23.7 GB model can fill only ~8 GB < 0.6 × 32 GB → every honest miner under-filled. The checker now reports the field only from a card with ≥ 8 GB resident and the pin-bump keeps the loadout's existing value otherwise (jq fallback verified).

Pin-bump now carries aggregate_decode_tps into the loadout — #1737 made the checker emit it and derives the per-token rate from it, but the jq step only copied single_stream_decode_tps and decode_per_request.

A non-JSON lium ls line killed run 33529543919 before renting (jq: parse error … column 6 — the shape of a leading Hint:/Error: line). free_node now drops anything before the first [, checks the listing parses as an array, and otherwise logs and retries.

ruff / format / pyright clean on the checker; bash -n, shellcheck -S warning, YAML parse clean.

https://claude.ai/code/session_011rz9LRxHEUXn37Jjqbxxh9

anderdc added 2 commits September 1, 2026 11:06
lium ls occasionally answers with a plain-text error on stdout; piped
straight into jq under bash -e that ended the job before a card was
rented (run 33529543919). free_node now checks the listing parses as an
array and treats anything else as no card this minute.

Claude-Session: https://claude.ai/code/session_011rz9LRxHEUXn37Jjqbxxh9
…port proxy

Lium's port proxy caps concurrent connections per pod (~5 on host
91.224.44.226, measured: 16 parallel 2 s requests finish 2.4 s..38.5 s
through the mapped port and 2.2 s..2.2 s on-box). The 16-wide overload
and speed bursts serialised there, failing R6 twice on an image that
passed on another host and producing an aggregate_decode_tps of 25.9
against 263. One ssh connection carries every stream (32 parallel:
2.5 s..3.6 s), so the checker now reaches the runtime through
ssh -L; the workflow generates the key lium registers. The pin-bump
also carries aggregate_decode_tps, which the validator prices tokens
from since #1737, and the executor listing tolerates a leading hint line.

Claude-Session: https://claude.ai/code/session_011rz9LRxHEUXn37Jjqbxxh9
@anderdc anderdc changed the title conformance: a non-JSON lium listing retries instead of killing the run conformance: measure the runtime through an ssh tunnel, not the Lium port proxy Sep 1, 2026
…erved_bytes

The attest pod is a bare gt-attest image; its 573 MB reservation written
into the release would fail every honest miner as under-filled. The
checker reports the field only from a card with >= 8 GB resident and
the pin-bump keeps the loadout's value otherwise.

Claude-Session: https://claude.ai/code/session_011rz9LRxHEUXn37Jjqbxxh9
@anderdc
anderdc merged commit 1d267e8 into test Sep 1, 2026
7 of 9 checks passed
@anderdc
anderdc deleted the conformance-lium-ls-flake branch September 1, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant