conformance: measure the runtime through an ssh tunnel, not the Lium port proxy - #1739
Merged
Conversation
added 2 commits
September 1, 2026 11:06
lium ls occasionally answers with a plain-text error on stdout; piped straight into jq under bash -e that ended the job before a card was rented (run 33529543919). free_node now checks the listing parses as an array and treats anything else as no card this minute. Claude-Session: https://claude.ai/code/session_011rz9LRxHEUXn37Jjqbxxh9
…port proxy Lium's port proxy caps concurrent connections per pod (~5 on host 91.224.44.226, measured: 16 parallel 2 s requests finish 2.4 s..38.5 s through the mapped port and 2.2 s..2.2 s on-box). The 16-wide overload and speed bursts serialised there, failing R6 twice on an image that passed on another host and producing an aggregate_decode_tps of 25.9 against 263. One ssh connection carries every stream (32 parallel: 2.5 s..3.6 s), so the checker now reaches the runtime through ssh -L; the workflow generates the key lium registers. The pin-bump also carries aggregate_decode_tps, which the validator prices tokens from since #1737, and the executor listing tolerates a leading hint line. Claude-Session: https://claude.ai/code/session_011rz9LRxHEUXn37Jjqbxxh9
… tunnel key there Claude-Session: https://claude.ai/code/session_011rz9LRxHEUXn37Jjqbxxh9
…erved_bytes The attest pod is a bare gt-attest image; its 573 MB reservation written into the release would fail every honest miner as under-filled. The checker reports the field only from a card with >= 8 GB resident and the pin-bump keeps the loadout's value otherwise. Claude-Session: https://claude.ai/code/session_011rz9LRxHEUXn37Jjqbxxh9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three of the four failures since the last conformance run were the harness, not the runtime — and the one pass it produced (#1740) carried a value that would have failed every honest miner.
R6 failed twice on
91.224.44.226with an image that passed on91.224.44.81on 8/31. Rented that host with ssh and measured: a 2 s HTTP handler answers 16 parallel requests in 2.4 s … 38.5 s through the Lium-mapped port and 2.2 s … 2.2 s on-box — the port proxy caps concurrent connections at ~5 per pod (5 clean, 6+ serialise). That serialises the 16-wide overload and speed bursts: a false R6 failure, and anaggregate_decode_tpsof 25.9/46.9 against 263 — the number the validator has priced tokens from since #1737 ($0.69/M would have read $7.5/M). Through one ssh connection the same 32 parallel finish in 2.5 … 3.6 s, so the checker now reaches the runtime overssh -L 18080:127.0.0.1:8080. Lium's SDK registers only~/.ssh/id_ed25519with a pod (it ignoresLIUM_SSH_KEY_PATHand the config file), so the workflow generates the key there. Run 33532638861 on the same host: CONFORMANT, R6 16×200 in 32.4 … 32.9 s.attest.vram_model_reserved_byteswas being measured on a card without the model. The attest pod is a baregt-attest:v1card, so the checker reported the attest server's own 573 MB; in the release that makesattest.pyexpect ~32 GB free on a 5090, and a card holding the 23.7 GB model can fill only ~8 GB < 0.6 × 32 GB → every honest minerunder-filled. The checker now reports the field only from a card with ≥ 8 GB resident and the pin-bump keeps the loadout's existing value otherwise (jq fallback verified).Pin-bump now carries
aggregate_decode_tpsinto the loadout — #1737 made the checker emit it and derives the per-token rate from it, but the jq step only copiedsingle_stream_decode_tpsanddecode_per_request.A non-JSON
lium lsline killed run 33529543919 before renting (jq: parse error … column 6— the shape of a leadingHint:/Error:line).free_nodenow drops anything before the first[, checks the listing parses as an array, and otherwise logs and retries.ruff / format / pyright clean on the checker;
bash -n,shellcheck -S warning, YAML parse clean.https://claude.ai/code/session_011rz9LRxHEUXn37Jjqbxxh9