Skip to content

Commit 2e4ce6a

Browse files
committed
docs(deepagent): m30b HF main stable pin N=9 in public docs
Document HF main as the stable current product pin (N=9) and test as the automation/dev mirror (also N=9). Align root + package READMEs and .env.example.
1 parent 5469597 commit 2e4ce6a

3 files changed

Lines changed: 43 additions & 27 deletions

File tree

README.md

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
**Hard, Docker-verifiable software-engineering benchmarks from real merged PRs**
66

77
[![License](https://img.shields.io/github/license/BaseIntelligence/deepagent)](https://github.com/BaseIntelligence/deepagent/blob/main/LICENSE)
8-
[![HF](https://img.shields.io/badge/HF-BaseIntelligence%2Fdeepagent-yellow.svg)](https://huggingface.co/datasets/BaseIntelligence/deepagent/tree/test)
8+
[![HF](https://img.shields.io/badge/HF-BaseIntelligence%2Fdeepagent-yellow.svg)](https://huggingface.co/datasets/BaseIntelligence/deepagent/tree/main)
99
[![Python](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/)
1010
[![CLI](https://img.shields.io/badge/CLI-deepagent-black.svg)](deepagent/README.md)
1111

@@ -25,7 +25,8 @@ DeepAgent ships **real_pr Harbor hardness packs**: live-mined multi-file pull re
2525
| **N** | **9** certified packs |
2626
| **unique_repos** | **7** |
2727
| **max packs / repo** | **2** (M28 diversity) |
28-
| **HF dataset** | [`BaseIntelligence/deepagent`](https://huggingface.co/datasets/BaseIntelligence/deepagent) revision **`test`** |
28+
| **HF stable pin** | [`BaseIntelligence/deepagent`](https://huggingface.co/datasets/BaseIntelligence/deepagent) revision **`main`** (N=9) |
29+
| **HF automation mirror** | same repo revision **`test`** (also N=9; CI/dev default write target) |
2930
| **Primary CLI** | `deepagent` (`generate` / `upload` / `pull` / `eval` / `oracle`) |
3031
| **Scoreboard** | [`deepagent/datasets/panel_prod_hard_deepswe_med_m28`](deepagent/datasets/panel_prod_hard_deepswe_med_m28/) |
3132

@@ -72,7 +73,8 @@ flowchart LR
7273
Mine[Live mine real_pr] --> Dual[Dual-run F2P/P2P]
7374
Dual --> Oracle[HarborDocker sol=1 null=0]
7475
Oracle --> Product[prod_hard_deepswe_med N=9]
75-
Product --> HF[HF BaseIntelligence/deepagent @test]
76+
Product --> HF[HF BaseIntelligence/deepagent @main]
77+
Product --> HFtest[HF @test automation mirror]
7678
Product --> Eval[deepagent eval Pier + Harbor]
7779
Eval --> Board[panel_prod_hard_deepswe_med_m28]
7880
```
@@ -99,17 +101,21 @@ deepagent generate \
99101
--out datasets/prod_hard_deepswe_med \
100102
--live-mine --oracle docker --panel offline --pier scripted
101103

102-
# Push pack trees to HF revision test
104+
# Push pack trees to HF stable pin (main) and/or automation mirror (test)
105+
deepagent upload \
106+
--src datasets/prod_hard_deepswe_med \
107+
--repo-id BaseIntelligence/deepagent \
108+
--revision main
103109
deepagent upload \
104110
--src datasets/prod_hard_deepswe_med \
105111
--repo-id BaseIntelligence/deepagent \
106112
--revision test
107113

108-
# Pull from HF
114+
# Pull stable product pin from HF main
109115
deepagent pull \
110116
--repo-id BaseIntelligence/deepagent \
111-
--revision test \
112-
--out datasets/hf_pull_test
117+
--revision main \
118+
--out datasets/hf_pull_main
113119

114120
# Dual-model Pier + Harbor eval (n_concurrent 1..5; hard-stop $600)
115121
deepagent eval \

deepagent/.env.example

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@ FACTORY_PANEL_MODELS=x-ai/grok-4.5,moonshotai/kimi-k2.6
1414
# Hard spend cap (exact + unresolved reserved must stay under this)
1515
FACTORY_BUDGET_USD=600
1616

17-
# Hugging Face (upload/pull BaseIntelligence/deepagent @test)
17+
# Hugging Face (upload/pull BaseIntelligence/deepagent)
18+
# Stable product pin: revision main (N=9). Automation/dev mirror: revision test (also N=9).
1819
# HF_TOKEN=
1920

2021
# GitHub REST + Search for live Real-PR mine.

deepagent/README.md

Lines changed: 28 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212

1313
[![Python](https://img.shields.io/badge/python-%3E%3D3.12-blue.svg)](pyproject.toml)
1414
[![License](https://img.shields.io/badge/license-MIT-green.svg)](pyproject.toml)
15-
[![HF](https://img.shields.io/badge/HF-BaseIntelligence%2Fdeepagent-yellow.svg)](https://huggingface.co/datasets/BaseIntelligence/deepagent/tree/test)
15+
[![HF](https://img.shields.io/badge/HF-BaseIntelligence%2Fdeepagent-yellow.svg)](https://huggingface.co/datasets/BaseIntelligence/deepagent/tree/main)
1616

1717
</div>
1818

@@ -29,7 +29,8 @@ isolation.
2929
| Surface | Path / ref | Role |
3030
|---|---|---|
3131
| **Current product** | `datasets/prod_hard_deepswe_med` | DeepSWE-median hard Real-PR, **N=9** (M27 floors + M28 max 2 packs/repo, unique_repos=7) |
32-
| **HF revision** | `BaseIntelligence/deepagent` **`test`** | Live Hub mirror of the current product |
32+
| **HF stable pin** | `BaseIntelligence/deepagent` **`main`** | Current product on Hub (N=9) |
33+
| **HF automation mirror** | `BaseIntelligence/deepagent` **`test`** | CI/dev write target; also N=9 after M29/M30 |
3334
| **Scoreboard** | `datasets/panel_prod_hard_deepswe_med_m28` | Grok 4.5 + Kimi 2.7-code dual-model matrix |
3435
| Product detail | `datasets/prod_hard_deepswe_med/PRODUCT_README.md` | Floors, p50 stats, diversity notes |
3536
| Historical M16 wave | `datasets/test_n10` | M16 N=10 only — **not** current product |
@@ -52,7 +53,8 @@ New product work should prefer `deepagent`.
5253
| **N** | **9** certified packs |
5354
| **unique_repos** | **7** |
5455
| **max packs / repo** | **2** (M28 diversity) |
55-
| **HF dataset** | [`BaseIntelligence/deepagent`](https://huggingface.co/datasets/BaseIntelligence/deepagent) revision **`test`** |
56+
| **HF stable pin** | [`BaseIntelligence/deepagent`](https://huggingface.co/datasets/BaseIntelligence/deepagent) revision **`main`** (N=9) |
57+
| **HF automation mirror** | same repo revision **`test`** (also N=9; CI/dev default write target) |
5658
| **Primary CLI** | `deepagent` (`generate` / `upload` / `pull` / `eval` / `oracle`) |
5759
| **Scoreboard** | [`datasets/panel_prod_hard_deepswe_med_m28`](datasets/panel_prod_hard_deepswe_med_m28/) |
5860
| **Default eval models** | `x-ai/grok-4.5` + `moonshotai/kimi-k2.7-code` |
@@ -129,17 +131,21 @@ deepagent generate \
129131
--out datasets/prod_hard_deepswe_med \
130132
--live-mine --oracle docker --panel offline --pier scripted
131133

132-
# Push pack trees + manifest to HF revision test
134+
# Push pack trees + manifest to HF stable pin (main) and/or automation mirror (test)
135+
deepagent upload \
136+
--src datasets/prod_hard_deepswe_med \
137+
--repo-id BaseIntelligence/deepagent \
138+
--revision main
133139
deepagent upload \
134140
--src datasets/prod_hard_deepswe_med \
135141
--repo-id BaseIntelligence/deepagent \
136142
--revision test
137143

138-
# Pull packs from HF revision test
144+
# Pull packs from HF stable pin (main)
139145
deepagent pull \
140146
--repo-id BaseIntelligence/deepagent \
141-
--revision test \
142-
--out datasets/hf_pull_test
147+
--revision main \
148+
--out datasets/hf_pull_main
143149

144150
# Pier mini-swe + Harbor dual-model eval (current median models)
145151
deepagent eval \
@@ -160,8 +166,8 @@ deepagent --help
160166
| Command | What it does |
161167
|---|---|
162168
| `generate` | Live mine hard `real_pr` via ship-deepagent path; product out `datasets/prod_hard_deepswe_med`; Docker oracle only; refuses fixture pad |
163-
| `upload` | Validate local pack root, push trees + `pack_manifest` to HF (`BaseIntelligence/deepagent`, default revision **`test`**) |
164-
| `pull` | Download pack trees from HF revision (`main` or `test`) into local out dir |
169+
| `upload` | Validate local pack root, push trees + `pack_manifest` to HF (`BaseIntelligence/deepagent`). Prefer **`main`** for the stable product pin; **`test`** remains the automation/dev default write target |
170+
| `pull` | Download pack trees from HF revision (`main` stable pin or `test` automation mirror) into local out dir |
165171
| `eval` | Pier + mini-swe-agent + HarborDocker model eval (`n_concurrent` 1..5, hard-stop **$600**, fidelity **`pier_miniswe_harbor`**). Current median panel: **grok-4.5** + **kimi-k2.7-code** |
166172
| `oracle` | HarborDocker dual-truth cert on one pack dir (solution reward 1, null 0; refuse fake) |
167173
| `version` | Package version identity |
@@ -190,19 +196,20 @@ auto-drop hardness packs (intrinsic policy).
190196

191197
| Ref | Content |
192198
|---|---|
193-
| **`test`** (current) | Live N=9 packs from `datasets/prod_hard_deepswe_med` |
194-
| `main` | Reserved for larger / product cuts when promoted |
199+
| **`main`** (stable product pin) | Current N=9 packs from `datasets/prod_hard_deepswe_med` |
200+
| **`test`** (automation / dev mirror) | Same N=9 product after M29/M30; default write target for CI and live automation |
195201

196-
Operator loop: **generate****upload** (revision `test`) → **pull**
197-
**eval** / **oracle**. Do not embed tokens in CLI flags; use `HF_TOKEN` env /
198-
`.env` only.
202+
Operator loop: **generate****upload** (prefer revision `main` for the stable
203+
public pin; keep `test` in sync for automation) → **pull****eval** /
204+
**oracle**. Do not embed tokens in CLI flags; use `HF_TOKEN` env / `.env` only.
199205

200206
## Wave dataset vs archives
201207

202208
| Path | N | Status |
203209
|---|---:|---|
204210
| `datasets/prod_hard_deepswe_med` | **9** | **Current product** — DeepSWE-median + M28 diversity |
205-
| HF `…/deepagent` `@test` | **9** | **Live on Hub** (mirror of current product) |
211+
| HF `…/deepagent` **`@main`** | **9** | **Stable product pin** on Hub |
212+
| HF `…/deepagent` `@test` | **9** | Automation / dev mirror (also N=9) |
206213
| `datasets/panel_prod_hard_deepswe_med_m28` | 9 scored | **Authoritative** Grok/Kimi2.7 scoreboard |
207214
| `datasets/test_n10` | 10 | **Historical** M16 wave only |
208215
| `datasets/prod_hard_keep` | ~10 | Soft M25/M26 band — audit only |
@@ -252,7 +259,8 @@ flowchart LR
252259
Label --> Oracle[Docker oracle sol=1 null=0]
253260
Oracle --> Export[Harbor pack export]
254261
Export --> Product[datasets/prod_hard_deepswe_med N=9]
255-
Product --> HF[HF BaseIntelligence/deepagent @test]
262+
Product --> HF[HF BaseIntelligence/deepagent @main]
263+
Product --> HFtest[HF @test automation mirror]
256264
HF --> Eval[deepagent eval Pier mini-swe Harbor]
257265
Eval --> Board[panel_prod_hard_deepswe_med_m28]
258266
```
@@ -362,7 +370,7 @@ Prefer `pytest -n 0` (or ≤ `-n 2`); never `pytest -n auto` on shared hosts.
362370
3. **Label** fail-to-pass and pass-to-pass node ids via dual-run suites.
363371
4. **Oracle-cert** with Docker: solution reward 1, null reward 0; refuse fake.
364372
5. **Export** Harbor v1.1 tree with held-out tests and isolation-clean agent view.
365-
6. **Upload / pull** HF mirror (`BaseIntelligence/deepagent` `@test`).
373+
6. **Upload / pull** HF (`BaseIntelligence/deepagent` **`@main`** stable pin; `@test` automation mirror).
366374
7. **Eval** Pier mini-swe + Harbor under hard-stop budget (fidelity
367375
`pier_miniswe_harbor`) with Grok 4.5 + Kimi 2.7-code for the current median panel.
368376

@@ -411,8 +419,9 @@ docs/
411419

412420
## Limits and non-goals
413421

414-
- Current product N is `prod_hard_deepswe_med` / HF `test` only. Historical
415-
`test_n10`, soft `prod_hard_keep`, hybrid/seed/fixture surfaces never substitute.
422+
- Current product N is `prod_hard_deepswe_med` / HF **`main`** (stable pin; `test`
423+
is the automation mirror at the same N=9). Historical `test_n10`, soft
424+
`prod_hard_keep`, hybrid/seed/fixture surfaces never substitute.
416425
- Not a browser UI; CLI + Docker + Pier only.
417426
- Copyleft / unknown licenses are fail-closed and never appear in PROVENANCE.
418427
- On constrained hosts, prefer serial Docker cert and lower `n_concurrent` eval.

0 commit comments

Comments
 (0)