test: add containerized real-codex E2E (docker) (#686)

[ranxianglei]

What

Containerized real end-to-end for codex, per the ask in #686 (“需要真的端到端…docker 起 codex 的那种”). It wraps the existing tests/e2e/e2e-codex.test.ts suite -- real codex CLI → bili proxy → real Responses upstream → full warmup / load-growth / ACP-compress / purity(/forge) lifecycle -- inside a reproducible Docker image, following the containerized-e2e pattern from Tyan66666/billion-context-dsh#128.

This is "real" in the sense #686 wants: actual model round-trips and compression through the proxy, not just the headless mcp list config check from PR #688. The two are complementary (#688 = Windows MCP-registration layer; this = full lifecycle in an isolated container).

Files

• scripts/e2e-docker/Dockerfile -- node:22-slim + npm ci + npm run build + @openai/codex; runtime baked into the image, host needs no node/codex.
• scripts/e2e-docker/run.sh -- host orchestrator: build image → run the suite in-container, forwarding E2E_UPSTREAM_URL/KEY/MODEL/FORGE/TMO/CODEX_BIN, mounting tmp/docker-e2e for persistent logs.
• .dockerignore -- keeps host node_modules/dist out of the build context.
• .github/workflows/ci-e2e-docker.yml -- workflow_dispatch job mirroring ci-e2e.yml (secrets-backed upstream); runs preflight then full.
• package.json -- npm run e2e:docker / e2e:docker:preflight.

Local usage

# zero-token smoke (boots container, prints codex version + dist + upstream probe)
E2E_UPSTREAM_URL=http://host.docker.internal:8199/v1 E2E_UPSTREAM_KEY=... npm run e2e:docker:preflight
# full lifecycle
E2E_UPSTREAM_URL=... E2E_UPSTREAM_KEY=... npm run e2e:docker
# if the upstream is on the host loopback, share the host netns instead:
E2E_DOCKER_NETWORK=host E2E_UPSTREAM_URL=http://127.0.0.1:8199/v1 npm run e2e:docker

Validation done here (Linux sandbox)

• bash -n scripts/e2e-docker/run.sh clean; workflow YAML parses; package.json valid JSON, version untouched.
• Confirmed the suite's work dir is process.cwd()/tmp (tests/e2e/e2e-codex.test.ts:18), so the -v <log>:/app/tmp mount captures logs exactly where CI uploads them.
• ⚠️ Docker is not available in this agent's sandbox, so I could NOT build the image or run the container here. The image build + first in-container run must happen on a docker host or via the workflow_dispatch job. That is why the CI job is manual-dispatch (zero risk to other PRs until triggered).

Review note

Reuses the proven host e2e unchanged -- the only new surface is "does @openai/codex install & run inside node:22-slim" plus container→upstream networking (documented in run.sh). Suggest one workflow_dispatch run to confirm before treating it as a standing gate.

Refs #686

---

中文摘要：按你"要真·端到端、用 docker 起 codex"的要求，把现有真实 codex e2e 套件封装成可复现的 Docker 镜像（node:22-slim + bili dist + codex），照 billion-context-dsh#128 的容器化模式做了 run.sh 编排 + 手动触发的 CI job + npm run e2e:docker；本机已验证脚本/YAML/JSON 语法与日志挂载路径，但本沙箱没有 docker，无法实际构建/运行容器，需在有 docker 的主机或 dispatch job 上跑首次确认。

[github-actions]

📦 Built Package Artifact

Branch: 2026-09-10_docker-codex-e2e (106737c)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-697

Each push to this PR publishes a new version under the pr-697 npm tag.

Option B — Download artifact

1. Download the artifact from the Actions run
2. Extract the tarball and install:

tar xzf billion-context-pr697.tgz
npm install -g package

---

This comment is automatically updated on each push.

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Reviewed PR #697 (containerized real-codex E2E). Verdict at the bottom.

Duplicate screening — no exact duplicate. Related items:

• E2E verify on real Windows: codex accepts CODEX_HOME overlay and registers the bili MCP server (#681 / PR #683) #686 (issue, parent ask) + test: add Windows codex overlay MCP-registration check (#686) #688 (sibling PR) target the Windows MCP-registration layer (headless mcp list); this PR is the full containerized lifecycle (real model round-trips + compression). Complementary, not overlapping.
• Model: glm-5.3 — test: e2e suite — real codex through bili (warmup/load/compress/purity/forge) #329 (closed) created the host-based tests/e2e/e2e-codex.test.ts suite this one wraps — it builds on that, not a dup.
• Pattern source Tyan66666/billion-context-dsh#128 (external fork).

What I verified (evidence-backed):

• Diff is clean: 5 files, +109/−1, all on-topic; no lockfile/whitespace churn; package.json version untouched (still 0.1.92) — only 2 scripts added.
• .dockerignore excludes .git → safe. The suite spawns codex with --skip-git-repo-check (tests/e2e/e2e-codex.test.ts:129), so no git workspace is required in-container.
• Dockerfile's npm ci --ignore-scripts → verified safe for this build. Only 3 packages carry install scripts (esbuild top-level, tsx/node_modules/esbuild, and darwin-only fsevents); every native binary ships via @esbuild/linux-x64 etc. optional deps present in package-lock.json (lines ~1134, ~2126), which npm ci installs regardless of --ignore-scripts. fzstd (bundled via tsup.config.ts:18, imported at src/content-encoding.ts:3) has no install script (lockfile 1191–1197, self-contained). So both npm run build and the runtime import work.
• Log-mount path is consistent end-to-end: WORK_ROOT = process.cwd()/tmp (tests/e2e/e2e-codex.test.ts:18) → /app/tmp in-container (WORKDIR /app); run.sh mounts -v $LOG_DIR:/app/tmp; the workflow uploads tmp/docker-e2e/. codex runs with cwd=WORK (under /app/tmp) so its artifacts are captured too.
• Networking model is sound: the bili proxy is spawned in-container at 127.0.0.1:<port>; codex→proxy stays in-container; only proxy→UPSTREAM_URL crosses the boundary. Default bridge reaches public/NAT upstreams; a loopback upstream needs E2E_DOCKER_NETWORK=host (documented in run.sh). Matches the existing ci-e2e.yml assumption that hosted runners can't hit 127.0.0.1.
• Workflow faithfully mirrors ci-e2e.yml (workflow_dispatch + forge input + secrets + upload-artifact) and correctly drops setup-node since node comes from the image.

Minor findings (none blocking):

1. Double image build in CI. run.sh unconditionally runs docker build, and the job calls it twice (preflight then full) → the image (apt-get + npm ci + npm run build + codex install) is built twice per dispatch. Wastes ~1–3 min; not broken. Suggested fix: guard with docker image inspect "$IMAGE" >/dev/null 2>&1 || docker build … (safe — GH runners are ephemeral, so no stale-image carryover). Happy to apply this as a follow-up commit if you want it.
2. Reproducibility wording. The PR calls it a "reproducible image," but uses floating tags node:22-slim + @openai/codex@latest. This matches the existing ci-e2e.yml (also @openai/codex@latest, line 21) and the image is rebuilt every run anyway, so it's consistent with repo practice — not a regression. If bit-for-bit pins are ever wanted, pin codex + node digest. Informational.
3. CI network precondition. Because the job defaults to bridge, the repo secret E2E_UPSTREAM_URL must be publicly/reachably routable from the GH runner (not loopback/LAN). Confirm that before treating this as a standing gate.

Verdict: No blocking issues; diff is clean and the containerization logic checks out against the actual test mechanics. As you noted, docker isn't available in this sandbox, so the image build + first in-container run still need to happen on a docker host or via the workflow_dispatch job — do that one confirmation run before treating it as a standing gate. I have not merged (merges are human-only).

---

中文摘要：审查了容器化真实 codex e2e（#697）——diff 干净、版本未动；逐条核对了 .git 排除（codex 用 --skip-git-repo-check）、npm ci --ignore-scripts（三个带脚本依赖的二进制都走 optionalDeps，安全）、日志挂载路径与网络模型，均与现有套件机制吻合。仅两处非阻塞小问题：CI 里镜像被构建两次、"可复现"措辞与浮动 tag 略有出入。建议先跑一次 workflow_dispatch 确认后再作为常设门禁，可以合并。

[ranxianglei]

评估测试有效性 而不是形式

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 重新按「测试有效性」评估（不看形式）。读完全部断言逻辑（tests/e2e/e2e-codex.test.ts:219–322）后的结论：

它真正证明了什么（有效信号）

• 真实全链路：真 codex 二进制 → 真 bili 进程 → 真模型往返 + SSE 重写。仓库里没有别的测试驱动真实客户端二进制，这是它独有的集成价值。
• 模型无关的硬事实（读 bili 自己的日志/产物，不依赖模型配合）——这部分最可信：
  • usage grew（:228，inputs[last] > inputs[0]）
  • compression fired（:241，日志出现 preflight compressed|compress requested|[Compressed m）
  • usage dropped post-fold（:243）
  • forge：codex compact intercepted（:276）+ rollout 含 fc_bili_ 项（:305）
• 唯一的保真度检查在 purity 段（:245–254）：压缩后仍能召回 1500 和哨兵值 30217，且不得回显植入的污染值 1400。这是唯一验证「压缩没丢信息/没泄漏脏数据」的地方。

有效性偏弱 / 被高估的地方（核心问题）

1. Docker 封装几乎没有增加对 bili 的行为信号。 run.sh 调的是与 ci-e2e.yml:30 逐字相同的测试命令（ACP_TEST_E2E=1 … e2e-codex.test.ts），env 也一样。差异 100% 是环境层：node 来自镜像、codex 装在镜像内、容器 netns、bind-mount 收日志。所以它没有让 e2e 比原来更「真」——真实端到端在 Model: glm-5.3 — test: e2e suite — real codex through bili (warmup/load/compress/purity/forge) #329 / 现有 host ci-e2e.yml 里早就有了。它对 E2E verify on real Windows: codex accepts CODEX_HOME overlay and registers the bili MCP server (#681 / PR #683) #686「要真·端到端」的回答，实际增量很窄：「@openai/codex 能否在 node:22-slim 里装起来并跑通 + 容器→上游网络是否可达」。价值在于运行环境可复现/降低 runner 漂移导致的假失败，而不是更广或更深的覆盖。标题措辞容易让人误以为它新增了端到端能力。
2. 保真度断言是「模型相关 + 软」的。 purity 通过的条件是模型能从摘要里召回数字，且走 turnExpect 三次重试（:164）。这把 bili 的压缩保真度和所选模型的召回能力混在一起了——模型弱/幻觉时，bili 完全正确也会挂；而且从没 pin 过模型契约（E2E_MODEL/上游都是操作者给的），一次绿灯不可跨时间/跨模型比较。最关键的可执行改进：加一条模型无关断言——直接检查持久化的摘要/blockContents 里确实包含哨兵值、且不再含 1400 那条消息体。把「模型记没记住」换成「摘要到底存没存住」，才能把 bili 正确性与模型质量解耦。目前它信任模型如实读自己的摘要，却没有核对摘要内容本身。
3. 路径覆盖窄。 只覆盖 OpenAI/Responses 线 + proxy 模式 + codex 客户端 + provider 名 "OpenAI"。没有 Anthropic 线、没有 plugin 模式、没有 SGLang 单 system（[Bug] ACP: ≥2 compressed blocks render as multiple system messages → SGLang (vLLM-compatible) upstreams return 400 #377）、没有其他客户端。而 AGENTS.md §3 建议合并请求管线改动前跑它——推荐门禁范围 > 实际覆盖，这条片外改动绿灯会给虚假信心。
4. 最高价值阶段默认关闭。 forge（拦截 + forged-summary 交接，最能体现 bili 差异性的行为）需 E2E_FORGE=1；本地默认和 CI job 默认都关着。所以默认门禁根本不跑拦截。
5. load 段 ack 很松（:222–224：收到#i 或任意 /compress/i 提及即过）。真正有牙的是 usage-growth 日志断言，不是 ack。「确定性」的说法只对输入成立（filler 逐字上线），判定仍是模型相关的。

让它更有效（建议，按性价比排序）

1. 加模型无关的摘要内容断言（哨兵值在持久化块中、1400 消息体已不在）——把保真度从「软召回」升级成「可核对的事实」。这是单点收益最大的一条。（改的是共享测试文件，会同时影响 host ci-e2e，我可以起草。）
2. CI 的 dispatch 默认把 forge 打开（或加一个明确标注的第二选项），否则常设门禁漏掉拦截这一核心行为。
3. Pin 一个最小模型契约（固定 E2E_MODEL + 期望行为），并在文档写明「verdict 与所选模型绑定」，让绿灯可比较。
4. 在 PR 描述里如实写清覆盖切片（proxy 模式 × codex × Responses 线），别让人当通用管线门禁；若要扩面，补一条 Anthropic 线或第二客户端。

底线

作为 proxy 模式下 codex/OpenAI-Responses 全生命周期的冒烟+集成门禁，它是有效的（尤其那几条读 bili 日志的硬断言）。但作为通用回归门禁不够：覆盖窄、保真度靠模型、最高价值阶段默认关。而本 PR（docker 化）本身的增量主要是运行环境可复现性，不是新的测试信号——建议把它定位成「把已有真 e2e 搬进可复现容器 + 验证 codex-in-node:22-slim」，并按上面第 1、2 条把它从「好看」变成「真的能拦住回归」。我未合并（合并仅人工）。

---

中文一句话：按有效性而非形式重评——底层套件对 proxy 模式 codex 全链路确实有效（多条读 bili 日志的模型无关硬断言 + 唯一一处保真度检查），但保真度断言依赖模型召回、覆盖仅限 OpenAI/Responses×proxy×codex、forge 默认关闭；而本 PR 的 docker 化几乎不新增行为信号，主要是运行环境可复现性，建议补「模型无关的摘要内容断言」并把 CI 默认开 forge 才算真门禁。