Skip to content

Commit 1f7702f

Browse files
cursoragentechobt
cursoragent
and
echobt
committed
docs: eval image contract, and refresh the stale pins
docs/EVAL-IMAGE.md documents the run, the request and every refusal, the document shape, the operator environment, what the image will not do, and how to build and pin it. The README and miner docs still named Qwen/Qwen3.8-Flash-Next, zai-org/GLM-5.3 as the teacher, and Inferact/GLM-5.3-NVFP4. The live pins are Qwen/Qwen3.8-27B, wire id glm-5.3, and incoai/GLM-5.3-NVFP4 served from RELEARN_TEACHER_LOCAL_DIR. Co-authored-by: Mathis <echobt@users.noreply.github.com>
1 parent b061452 commit 1f7702f

3 files changed

Lines changed: 276 additions & 23 deletions

File tree

‎README.md‎

Lines changed: 50 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,41 @@
22

33
# Relearn
44

5-
Public challenge repo for the Cortex **Relearn** subnet.
5+
Public challenge repo for the Cortex **Relearn** subnet (challenge id
6+
`relearn`).
67

78
**This repo is miner + eval only.** It does not contain the Cortex control
89
plane, gateway, or validator. Those live in [`CortexLM/cortex`](https://github.com/CortexLM/cortex).
910

1011
| Pin | Value |
1112
|-----|--------|
12-
| Base model | `Qwen/Qwen3.8-Flash-Next` |
13-
| Teacher / judge | `zai-org/GLM-5.3` |
14-
| Teacher NVFP4 | `Inferact/GLM-5.3-NVFP4` |
15-
| Score | Displacement vs the previous champion |
16-
| Trust | No TDX / no Phala CVM. Miner pays Lium. Holdout unseals after digest freeze. |
13+
| Base model | `Qwen/Qwen3.8-27B` (Apache-2.0, native VLM) |
14+
| Teacher / judge | HTTP API, wire id `glm-5.3` (operator sets `RELEARN_TEACHER_*`) |
15+
| Teacher weights | `incoai/GLM-5.3-NVFP4`, served from `RELEARN_TEACHER_LOCAL_DIR` |
16+
| Eval image | `ghcr.io/cortexlm/relearn-eval`, pinned by digest |
17+
| Score | Displacement vs the previous champion on a private holdout |
18+
| Trust | No TDX / no Phala CVM. Miner pays Lium. The holdout reaches the pod only for the length of a run |
1719

18-
Cortex pins this repo's git SHA and the eval image digest in
19-
`config/relearn-pin.toml`. Deploy = bump the pin after this repo's CI is green.
20+
The pin of record is [`config/relearn-pin.toml`](https://github.com/CortexLM/cortex/blob/main/config/relearn-pin.toml)
21+
in the control plane: it carries this repo's git SHA and the eval image digest.
22+
Deploy = bump the pin after this repo's CI publishes a green image.
23+
24+
While `eval_image_digest` is empty, live hosts answer **503** on
25+
`POST /v1/submissions` rather than scoring — there is no simulated fallback.
26+
27+
## Eval image
28+
29+
The image is the whole live scorer. The control plane boots it on a digest pin,
30+
stages the run request into `/tmp/relearn_eval`, and accepts a score only when
31+
the pod prints `RELEARN_METRICS=<document>` and `RELEARN_EVAL_OK`.
32+
33+
```bash
34+
docker build -f eval/Dockerfile -t relearn-eval:dev .
35+
relearn-eval score --request request.json --out metrics.json
36+
```
37+
38+
Full contract, environment, and operator notes: [`docs/EVAL-IMAGE.md`](./docs/EVAL-IMAGE.md).
39+
Normative source: `docs/RELEARN.md` § Eval image contract in the control plane.
2040

2141
## Miner path
2242

@@ -27,24 +47,35 @@ See [docs/](./docs/) and the control-plane mirror
2747
curl -sS -X POST https://<gateway>/challenge/relearn/v1/submissions \
2848
-H 'content-type: application/json' \
2949
-H "X-Lium-Api-Key: $LIUM_API_KEY" \
30-
-d '{"miner_hotkey":"<64-hex>","artifact_digest":"<sha256>"}'
50+
-d '{
51+
"miner_hotkey": "<64-hex>",
52+
"artifact_digest": "<sha256>",
53+
"manifest": {"train_item_ids": [1], "train_dataset_ids": ["my-sft-mix-v3"]}
54+
}'
3155
```
3256

33-
Never commit `LIUM_API_KEY` or any secret.
57+
`manifest` is required evidence: an undeclared one fails the contamination gate
58+
rather than skipping it. Never commit `LIUM_API_KEY` or any secret.
3459

3560
## Layout
3661

3762
| Path | Role |
3863
|------|------|
39-
| `eval/Dockerfile` | Digest-pinned eval image |
40-
| `eval/harness/` | Pod entry: load base + miner artifact, score holdout |
41-
| `eval/generators/` | Disjoint train / eval synthetic factory |
42-
| `eval/teacher/` | Frozen GLM-5.3 judge (HTTP API; never serves miner weights) |
43-
| `eval/decontam/` | Official-bench n-gram blocklist |
44-
| `docs/` | Miner-facing human docs |
64+
| `eval/Dockerfile`, `eval/entrypoint.sh` | The digest-pinned eval image |
65+
| `eval/src/relearn_eval/contract.py` | Markers, schema, and the metrics document |
66+
| `eval/src/relearn_eval/request.py` | The harvest request, and what it refuses |
67+
| `eval/src/relearn_eval/scoring.py` | Every series in the document, measured |
68+
| `eval/src/relearn_eval/verify.py`, `harvest.py` | The control plane's acceptance checks, mirrored |
69+
| `eval/src/relearn_eval/teacher.py` | Frozen judge over HTTP; never serves miner weights |
70+
| `eval/src/relearn_eval/catalog/` | Public, canary, general-canary, agent-trace slices |
71+
| `eval/src/relearn_eval/generators.py`, `decontam.py` | Miner-side disjoint factory and bench blocklist |
72+
| `tests/` | Image contract tests |
73+
| `docs/` | Miner-facing and operator-facing docs |
4574

4675
## Teacher serving
4776

48-
Prefer NVFP4 on Lium when an 8× Blackwell-class host is available.
49-
v0 fallback: teacher-only HTTP API. The scored artifact is always the
50-
miner weights loaded **inside** the eval image — never via the teacher API.
77+
The teacher is judge-only, and the scored artifact is always the miner weights
78+
loaded **inside** the eval image — never anything served through the teacher
79+
API. Download the NVFP4 weights, then point vLLM at
80+
`RELEARN_TEACHER_LOCAL_DIR`; never pass the Hugging Face repo id to vLLM. GPU
81+
shape is the operator's, described in the control-plane pin.

‎docs/EVAL-IMAGE.md‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# The Relearn eval image
2+
3+
The live scorer for challenge `relearn`. Everything that turns an artifact into
4+
numbers is here; the control plane cannot compute a score and has no simulated
5+
fallback, so a pod that does not return a well-formed, correctly bound document
6+
is a **503** rather than a verdict.
7+
8+
Normative contract: `docs/RELEARN.md` § Eval image contract in
9+
[`CortexLM/cortex`](https://github.com/CortexLM/cortex). The client is
10+
`crates/relearn-lium-harvest`. Where this document and those disagree, they win.
11+
12+
## One run
13+
14+
The control plane, per submission:
15+
16+
1. boots `eval_image@<digest>` on a Lium pod the **miner** pays for, with the
17+
master SSH public key, under the price / GPU / lifetime guardrails;
18+
2. writes `request.json` into `/tmp/relearn_eval` **over stdin** — no run input
19+
is interpolated into the remote command;
20+
3. runs `relearn-eval score --request request.json --out metrics.json`;
21+
4. reads back `RELEARN_METRICS=<document>` and `RELEARN_EVAL_OK`;
22+
5. scrubs the workdir, terminates the pod, and requires *verified* termination
23+
before accepting anything the run returned.
24+
25+
The image's side of that is one command and two markers:
26+
27+
```bash
28+
relearn-eval score --request request.json --out metrics.json
29+
# RELEARN_METRICS={"schema_version":1,…}
30+
# RELEARN_EVAL_OK
31+
```
32+
33+
`metrics.json` is exactly one line with no trailing newline, because the
34+
harvest reconstructs the marker line with `printf 'RELEARN_METRICS='; cat
35+
metrics.json`. `RELEARN_EVAL_OK` is the last thing the process prints, and only
36+
after the document has passed the control plane's own acceptance checks
37+
(mirrored in `verify.py` and `harvest.py`). Any failure exits non-zero with no
38+
marker and no sidecar.
39+
40+
## The request
41+
42+
`HarvestRequest`, mirroring the harvest client:
43+
44+
```json
45+
{
46+
"schema_version": 1,
47+
"submission_digest": "<frozen run>",
48+
"artifact_digest": "<sha256, or base-relearn-champion>",
49+
"base_model": "Qwen/Qwen3.8-27B",
50+
"teacher_model": "glm-5.3",
51+
"eval_image_digest": "sha256:…",
52+
"holdout_commitment": "<64 hex>",
53+
"holdout": [{"id": 801, "prompt": "…", "dataset_id": "…", "task": "text", "image_hash": ""}]
54+
}
55+
```
56+
57+
Unknown fields are tolerated, so the control plane can grow the request without
58+
a new image. Everything the image acts on is checked, and each of these is a
59+
failed run rather than a scored one:
60+
61+
| Refusal | Why |
62+
|---------|-----|
63+
| `schema_version` is not 1 | the document would not deserialize |
64+
| no `submission_digest` / `artifact_digest` / `base_model` / `teacher_model` | nothing to bind the run to |
65+
| `eval_image_digest` is not a `sha256:` pin | the control plane only rents digest-pinned images |
66+
| empty holdout, duplicate ids, empty prompt, unknown task | not a scorable split |
67+
| a vision item with no 64-hex `image_hash` | it would be scored without its image |
68+
| the items do not hash to `holdout_commitment` | the request was edited in flight |
69+
| `RELEARN_EVAL_IMAGE_DIGEST` is set and disagrees | the request is for another image build |
70+
71+
`artifact_digest: "base-relearn-champion"` is the boot baseline: the base model
72+
with no artifact, which is how a live host records a champion to compare
73+
against.
74+
75+
**The request carries the private holdout.** The image never writes a holdout
76+
prompt to stdout, to a log, or to a persisted path: the document's series keys
77+
are `h<id>`, `x<id>`, `p<id>`, `c<id>`, `g<id>` — ids, never text — and the
78+
verifier refuses a document whose keys are any other shape.
79+
80+
## The document
81+
82+
`RelearnEvalMetrics`: a `BaselineMeasurement` envelope (`#[serde(flatten)]`)
83+
plus the run identity.
84+
85+
```json
86+
{
87+
"schema_version": 1,
88+
"submission_digest": "<echo of the request>",
89+
"artifact_digest": "<echo of the request>",
90+
"eval_image_digest": "sha256:…",
91+
"holdout_commitment": "…",
92+
"holdout": {}, "public": {}, "perturbed": {},
93+
"canaries": {}, "general_canary": {},
94+
"agent_trace": 0.0,
95+
"vision_shuffle": {}
96+
}
97+
```
98+
99+
| Series | Items | Graded by |
100+
|--------|-------|-----------|
101+
| `holdout` | the request's private split | frozen teacher |
102+
| `perturbed` | the same items, pinned rewrite | frozen teacher |
103+
| `public` | the published split | frozen teacher |
104+
| `canaries` | shipped known-answer items | reference match |
105+
| `general_canary` | shipped MMLU / MMMU-style choices | choice letter |
106+
| `agent_trace` | shipped ordered-plan tasks | rubric coverage and order |
107+
| `vision_shuffle` | one entry per vision family in the holdout | teacher, real vs shuffled pixels |
108+
109+
The holdout and the public split are judged by the same judge on the same
110+
scale, because the gap between them is itself a gate. Scores are rounded to six
111+
decimals so two runs of the same model on the same items agree exactly.
112+
113+
The same document is what an operator installs as `RELEARN_BASE_CHAMPION_FILE`:
114+
run the pinned image on the base model once and use the output as-is.
115+
116+
## Environment
117+
118+
Nothing below is baked into the image, and none of it is a secret this repo
119+
knows. All of it is pod environment the operator sets.
120+
121+
| Variable | Role |
122+
|----------|------|
123+
| `RELEARN_TEACHER_API_URL` | OpenAI-compatible judge endpoint. **Required**: with no judge the run fails |
124+
| `RELEARN_TEACHER_API_KEY` | Bearer for that endpoint. Never logged |
125+
| `RELEARN_TEACHER_MODEL` | Wire id override (default `glm-5.3`) |
126+
| `RELEARN_BASE_MODEL_DIR` | Local base weights. Preferred over pulling the pinned id per run |
127+
| `RELEARN_ARTIFACT_DIR` | Content-addressed artifact store, checked first |
128+
| `RELEARN_ARTIFACT_URL_TEMPLATE` | Fallback fetch, e.g. `.../{digest}.tar` |
129+
| `RELEARN_IMAGE_STORE` | Content-addressed image store. Required when the holdout has vision items |
130+
| `RELEARN_PUBLIC_FILE` | The live public split. Strongly recommended — see below |
131+
| `RELEARN_CANARY_FILE`, `RELEARN_GENERAL_CANARY_FILE`, `RELEARN_AGENT_TRACE_FILE` | Replace a shipped slice |
132+
| `RELEARN_EVAL_BACKEND` | `auto` (default), `vllm`, or `transformers` |
133+
| `RELEARN_TENSOR_PARALLEL`, `RELEARN_MAX_NEW_TOKENS` | Runtime shape and decode width |
134+
| `RELEARN_EVAL_IMAGE_DIGEST` | Pin the image's own digest so a request for another build is refused |
135+
| `RELEARN_MAX_ARTIFACT_BYTES` | Ceiling on a fetched artifact |
136+
| `RELEARN_LOG_LEVEL` | stderr verbosity |
137+
138+
The artifact is always verified against `artifact_digest` before it is loaded,
139+
whichever source produced it. A store that serves different bytes fails the run.
140+
141+
### The shipped slices are the CI default, not the live seal
142+
143+
The request carries only the holdout, but the document must carry `public` and
144+
`general_canary` or the control plane refuses the champion at boot. So the image
145+
owns those slices, and ships a synthetic set — the same arrangement as the
146+
committed `holdout_commitment` in the control plane, which is the CI seal rather
147+
than the production one.
148+
149+
A live host should point `RELEARN_PUBLIC_FILE` at the real published records
150+
(the ids the pin publishes as trainable). With the shipped default, the
151+
public–holdout gap gate still runs, but against items no miner trained on, so it
152+
is weaker than it looks. The shipped items are deliberately not drawn from any
153+
official benchmark: a general-bench canary lifted from a bench cannot detect
154+
regression on that bench.
155+
156+
## What the image will not do
157+
158+
* **No simulated numbers.** There is no offline harness and no fallback judge.
159+
If the model cannot be loaded, the judge cannot be reached, or a vision item's
160+
pixels are missing, the run ends without a document. A hole in a series would
161+
be read as a low score nobody measured.
162+
* **No second protocol.** `verify.py` and `harvest.py` are mirrors of the
163+
control plane's checks, used so a transcript this image produces is proven
164+
acceptable before it is printed.
165+
* **No teacher weights, hostnames, or credentials in the repo**, and no Modal.
166+
The teacher API is judge-only: a payload that looks like weights (safetensors,
167+
GGUF, NVFP4, ckpt) or a model id that looks like an artifact digest is refused
168+
before any request is sent.
169+
* **Never DFlash2** (CC BY-NC-ND) and never the Flash teacher variants.
170+
171+
## Building and pinning
172+
173+
```bash
174+
# what CI publishes: contract plus the torch / transformers runtime
175+
docker build -f eval/Dockerfile -t relearn-eval:dev .
176+
177+
# contract-only, for a fast local loop: cannot score, and says so
178+
docker build -f eval/Dockerfile --build-arg WITH_RUNTIME=0 -t relearn-eval:contract .
179+
180+
# a vLLM or CUDA base of the operator's choosing
181+
docker build -f eval/Dockerfile \
182+
--build-arg BASE_IMAGE=<base@sha256:…> \
183+
--build-arg TORCH_INDEX_URL=<accelerator wheel index> .
184+
```
185+
186+
CI publishes `ghcr.io/cortexlm/relearn-eval` and prints the pushed
187+
`sha256:` digest. Put that digest — never a tag — in `eval_image_digest` in the
188+
control plane's `config/relearn-pin.toml`, together with this repo's git SHA in
189+
`relearn_git_sha`, and re-sign the trust root.
190+
191+
The pod's default command (`serve`) keeps the container reachable over SSH so
192+
the harvest can stage the request and run the scorer; any other argument list is
193+
passed straight to `relearn-eval`, which is how CI and a local operator drive
194+
the image.
195+
196+
## Checking a run
197+
198+
```bash
199+
# does this document belong to this run?
200+
relearn-eval verify --request request.json --metrics metrics.json
201+
202+
# would the control plane accept this pod transcript?
203+
relearn-eval verify --request request.json --transcript run.log
204+
```
205+
206+
Both apply the control plane's acceptance rules: schema, run identity, image
207+
digest, holdout commitment, one score per requested item, the series the gates
208+
need, and a pixel-shuffle control for every vision family in the holdout.

‎docs/README.md‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,25 @@
11
# Relearn miner docs
22

3-
Improve `Qwen/Qwen3.8-Flash-Next`. Score is paired displacement vs the
4-
current champion. Regressions are never crowned.
3+
Post-train `Qwen/Qwen3.8-27B` (Apache-2.0, native VLM). Score is paired
4+
displacement against the current champion on a private holdout. Regressions are
5+
never crowned.
56

67
- Submit over HTTP to the Cortex gateway (`/challenge/relearn/v1/submissions`).
78
- Pay Lium yourself (`LIUM_API_KEY` / `X-Lium-Api-Key`).
8-
- Holdout is unsealed only after your submission digest freezes.
9-
- Official public benchmarks are out of bounds (see `eval/decontam`).
9+
- Declare what you trained on in `manifest`. An undeclared manifest fails the
10+
contamination gate — it does not skip it.
11+
- The holdout reaches the eval pod only after your submission digest freezes,
12+
and only for the length of that run.
13+
- Official public benchmarks are out of bounds; the blocklist is
14+
`eval/src/relearn_eval/decontam.py`.
15+
- Only the public ids on `GET /challenge/relearn/v1/status` are trainable.
16+
17+
Check `GET /challenge/relearn/v1/status` before submitting: while `can_score`
18+
is `false`, submissions answer 503. `eval_backend` tells you whether a verdict
19+
came from the pinned eval image (`lium`) or an operator's offline harness
20+
(`sim`, CI and local only).
21+
22+
Eval image contract and how your artifact is loaded and scored:
23+
[`EVAL-IMAGE.md`](./EVAL-IMAGE.md).
1024

1125
Control plane: <https://github.com/CortexLM/cortex>

0 commit comments

Comments
 (0)