(feat) 容器化 headless E2E 集成测试套件:并行分道 + smoke/full 分集 - #128
Open
Tyan66666 wants to merge 2 commits into
Open
Conversation
问题:每个 PR 依赖手工在 web profile 里跑真实会话验证——慢、不可重复, issue #54 那类会话砖化肉眼几乎不可能第一时间发现。 修复:新增 scripts/e2e/ 容器化 headless E2E 套件(设计文档 docs/e2e-testing-design.md v4)。不改 src/;dist 无漂移。 - Dockerfile:node:22-slim + 钉版本 @deepseek-ai/dsh@0.1.2-rc.1,运行时可复现 - run.mjs:并行分道编排(weight 均衡,默认 3 道)、smoke/full 分集、凭据 经 --env-file 注入不落盘 - scenarios.mjs:9 场景(S1-S4 基线 + S54/S47/S60/S9/S35 bug 回归), 断言只读会话日志现有事件(compaction/*、tool/call、tool/result 原文) - container-main.mjs:容器内执行器(组装 profile→逐场景 headless run→ 解包断言→JSON 报告) 验证:agicto 路由首轮 7/9 场景实测,4 通过(S2 压缩后检索命中暗号、 S3 连续 6 次压缩合法、S54 CJK 影子价非负不砖化、S60 Checkpoint seqs 行在); 4 失败均为测试侧问题并已修复(引擎零回归)。S47 现场额外验证了模型自发 tier-2 蒸馏与 acp_status 报告的正确性。
# Conflicts: # docs/README.md
Contributor
|
我来协助看一下这个问题,请分析并回复处理结果。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
每个 PR 目前靠手工验证:把 worktree 以
file:装进 web profile、开真实会话跑任务、人肉观察 nudge/压缩/检索,再翻日志确认投影没被染负。慢、不可重复,且 issue #54 那类"会话砖化"(投影计数变负 → 后续每个 turn 被 zod 拒绝)在真实长会话里肉眼几乎不可能第一时间发现。修复
新增容器化 headless E2E 集成测试套件,设计记录见
docs/e2e-testing-design.md(v4)。不改src/——环境组装、运行、断言全部使用 DSH 原生机制(headless runner + profile + 会话日志),dist/无漂移(src/未动,CI 校验通过)。组成
scripts/e2e/Dockerfilenode:22-slim+ 钉版本@deepseek-ai/dsh@0.1.2-rc.1+ zstd/pnpm——运行时钉进镜像可复现,宿主机不需要装 dsh CLIscripts/e2e/run.mjsscripts/e2e/scenarios.mjsscripts/e2e/container-main.mjs关键设计(详见设计文档决策记录 1-9)
compaction/*事件配对与shadowedTokenCount ≥ 0、tool/call/tool/result逐字原文(acp_status 给模型看的每一行都可断言);硬/软分界——引擎契约硬、模型自由意志软、被明确指示的调用形态硬DSH_PERMISSION_MODE=danger-full-access完全开放(headless 下ask无应答者必然 fail closed;容器本身是隔离边界)e2e.env经--env-file注入,不落盘不入库npm run e2e(默认 smoke 集 S4/S54/S60,约 1 分钟)、npm run e2e:full(全量 9 场景,3 道并行约 5 分钟)、ACP_E2E_SCENARIOS=...任意指定验证
agicto 路由首轮 7/9 场景实测:
search_context命中被遮蔽原文里的暗号)、S3(连续 6 次压缩全部合法)、S54(CJK 重负载影子价全程非负、无投影 zod 拒绝)、S60(ACTIVE 块带 Checkpoint seqs 行)request/context披露的是 provider 窗口而非配置覆盖值)、S9 降软(模型意愿归层 B mock)、S47 时间对齐+蒸馏吸收扣除——S47 现场还额外验证了模型自发 tier-2 蒸馏(b6 吸收 b1-b5)时 acp_status 报告"3 active"的正确性待办
mimo-v2.5)的验证跑已配置、待执行