Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ on:
- "pyproject.toml"
- "uv.lock"
- "scripts/aliyun/**"
- "scripts/a2a/**"
- "skills/iac-code/**"
- "skills/alicloud-ros-agent/**"
- "skill-runtime/**"
- ".github/workflows/test.yml"
pull_request:
Expand All @@ -24,7 +26,9 @@ on:
- "pyproject.toml"
- "uv.lock"
- "scripts/aliyun/**"
- "scripts/a2a/**"
- "skills/iac-code/**"
- "skills/alicloud-ros-agent/**"
- "skill-runtime/**"
- ".github/workflows/test.yml"

Expand All @@ -51,6 +55,8 @@ jobs:
run: |
python -m py_compile skills/iac-code/scripts/iac_code.py
python skills/iac-code/scripts/iac_code.py --help
python -m py_compile skills/alicloud-ros-agent/scripts/ros_agent.py
python skills/alicloud-ros-agent/scripts/ros_agent.py --help

lint:
name: Lint
Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,9 @@ default = true

[tool.pytest.ini_options]
timeout = 30
markers = [
"integration: process-level integration tests that use local services only",
]

[tool.coverage.run]
source_pkgs = ["iac_code"]
Expand Down
108 changes: 104 additions & 4 deletions scripts/a2a/e2e/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,104 @@
# A2A E2E Session Recovery and Redaction

## Real StartChat permission-wait matrix

`run_start_chat_permission_wait.py` is the credential-gated, repeatable chain
for this feature. It runs Qoder's real LLM through the installed
`alicloud-ros-agent` Skill, its Python bridge, the native `aliyun` CLI, the
StartChat-only HTTPS relay, local iac-code A2A servers, and real iac-code
LLM/cloud calls. Before taking a private isolated config copy, it refreshes
OAuth-backed STS in the caller-selected `--source-config-dir` in place. This is
important because OAuth refresh tokens may rotate: refreshing only a disposable
copy can invalidate the source for the next scenario. It then fixes the server
policy at `300 / 300 / 30`, enables the shared-backup commit protocol, uses
unique Stack/VSwitch names, and performs an exact-name cleanup fallback.

The Qoder flag that bypasses host Bash/file confirmation applies only to the
test driver. It does not approve ROS Agent permissions: the isolated iac-code
settings use the default permission mode, explicitly allow incidental tools,
and ask for cloud-mutating tools. The A2A servers also keep
`auto_approve_permissions: false`, and every non-read-only cloud operation is
answered through the correlated StartChat permission envelope.

Run one scenario per fresh directory:

The real headless Qoder turn timeout defaults to 900 seconds so a two-candidate
Pipeline can finish without weakening the scenario. Override it with
`--qoder-turn-timeout` when diagnosing a slower provider.

```bash
uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-normal-before \
--mode normal

uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-pipeline-before \
--mode pipeline

# Answer during the 30-second grace after the 300-second resident deadline.
uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-normal-grace \
--mode normal \
--answer-delay-seconds 305

# Answer after non-failure suspension.
uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-pipeline-suspended \
--mode pipeline \
--answer-delay-seconds 335

# Kill only the selected local A2A process at the first permission boundary,
# restart it with the same persistence/config directories, then answer.
uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-normal-restart \
--mode normal \
--restart-at-first-permission
```

Repeat the grace, suspended, and restart variants for both `normal` and
`pipeline`. The repository prompt is
`permission_wait/permission_wait_start_chat_prompt.md`. Each run writes bounded Qoder turn
summaries, permission observations, relay metrics, a safe result manifest, and
local server logs. Before exit, it derives bounded read-only evidence and removes
the copied credential files and complete session transcripts. A read-only
permission prompt or an out-of-scope cloud target fails the run without being
approved. The final checks require a real non-read-only permission, local and
shared serial checkpoint evidence, no Sub Pipeline checkpoint, native
StartChat usage, exact resource cleanup, and retention of the pre-existing VPC
inventory.

The controlled Sub-Pipeline fixture uses a real `PipelineRunner` with two real
`AgentLoop` candidates. One candidate parks at an actual permission Future while
the other completes naturally; after the configured hard timeout, the parent
aggregates both conclusions, reaches candidate selection, and completes. The
production A2A backup hook also proves that the Sub permission itself did not
trigger a critical permission backup. The acceptance run uses the production
300-second value:

```bash
uv run python scripts/a2a/e2e/permission_wait/run_sub_pipeline_permission_timeout.py \
--run-dir /tmp/iac-pwait-sub-pipeline-300 \
--timeout-seconds 300
```

Its accelerated regression and the fast deterministic process-restart matrix
are:

```bash
uv run pytest -q tests/a2a_e2e/test_sub_pipeline_permission_timeout.py
uv run pytest -q tests/a2a_e2e/test_permission_wait_restart.py
```

The restart matrix covers Normal/Pipeline × allow/deny without real
credentials. The Sub-Pipeline fixture asserts one denial ToolResult, continued
Agent-loop execution, parent candidate selection/completion, and the absence of
grace, durable permission checkpoints, and permission-critical backup.

This directory contains headless end-to-end checks for A2A pipeline session
recovery and redaction regressions. The runner drives the public A2A JSON-RPC
streaming endpoint and records SSE events and pipeline snapshots. Recovery
Expand Down Expand Up @@ -105,9 +204,10 @@ uv run python scripts/a2a/e2e/run_recovery_scenarios.py \
--scenario selection-during-backup
```

An E2E-only fixture delays the Step 4 `input_required` backup by at least 10
seconds. The client submits its selection as soon as the event arrives. The
scenario proves that dispatch happened inside the backup started/finished
The runner arms an E2E-only fixture before the initial request so that the Step
4 `input_required` backup is delayed by at least 10 seconds. It submits the
selection as soon as the backup started marker appears and continues to verify
the candidate-selection event. The scenario proves that dispatch happened inside the backup started/finished
window, the message was consumed as candidate input, and no
`interrupt_received` / `interrupt_classified` event was emitted.

Expand Down Expand Up @@ -170,7 +270,7 @@ the rest of the tests.
| `redaction-step4` | Force A2A safe mode and stop when the real mini-app backend/database task reaches step 4 candidate selection | None; no candidate selection is submitted | Canonical password parameters are not placeholders; public A2A passwords equal canonical values; token counters stay numeric when present; known server paths become `[PATH]` only in the public copy; deployment never starts. |
| `scenario1` | After pipeline completion and one normal-chat follow-up | Ask what the previous normal-chat question was | Normal-chat history survives restart; VSwitch evidence exists. |
| `scenario1-performance-backup` | Full `scenario1` with `IAC_CODE_A2A_EXTREME_PERFORMANCE=true` and `IAC_CODE_CONFIG_BACKUP_DIR=<run-dir>/session-backup` | After the Step4 backup is durable, stop the server, remove the matching primary session under `projects`, restart, and select without `taskId`; later normal-chat recovery also omits `taskId` | Only the backup session exists before restart; restart alone does not recreate the primary session; selection without `taskId` restores the primary session from backup and hydrates the recovered task; full scenario1 passes. |
| `selection-during-backup` | Step 4 `input_required` is published, then an E2E fixture blocks its backup for at least 10 seconds | Immediately send `你随便选一个方案。` with the active pipeline `taskId` while backup is running | Dispatch falls inside the backup started/finished window; the request is queued and consumed as candidate input; no interrupt events are emitted; the pipeline completes. |
| `selection-during-backup` | Arm the E2E fixture before the initial request; when the Step 4 `input_required` backup starts, it writes a started marker and blocks for at least 10 seconds | Immediately send `你随便选一个方案。` with the active pipeline `taskId` while backup is running, then verify the Step 4 event | Dispatch falls inside the backup started/finished window; the request is queued and consumed as candidate input; no interrupt events are emitted; the pipeline completes. |
| `selection-waiting` | Step 4 waits for candidate selection | `你随便选一个方案。` without `taskId` | Waiting step4 task is recovered and selected; VSwitch evidence exists. |
| `ask-waiting` | `ask_user_question` waits for user input | Clarification answers without `taskId` | Pending ask input is recovered and pipeline completes; VSwitch evidence exists. |
| `image-initial` | Initial user message is the static `initial.png` image fixture | Candidate selection text | The image starts the pipeline, reaches step4 selection, completes, and produces VSwitch evidence. |
Expand Down
86 changes: 83 additions & 3 deletions scripts/a2a/e2e/README.zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,85 @@
# A2A 会话恢复与脱敏 E2E

## 真实 StartChat 权限等待矩阵

`run_start_chat_permission_wait.py` 是本功能可重复执行、受凭证开关保护的真实链路:Qoder 真实 LLM
→ 安装后的 `alicloud-ros-agent` Skill → Python bridge → 原生 `aliyun` CLI → 只暴露 StartChat/StopChat
的本地 HTTPS relay → 本地 iac-code A2A server → 真实 iac-code LLM 和云调用。Runner 会先在调用方指定的
`--source-config-dir` 中原地刷新 OAuth STS,再把最新凭证复制到权限受限的独立 config dir。这样可避免只在
一次性副本中刷新并轮换 OAuth refresh token,导致源配置在下一个场景失效。随后把服务端策略固定为
`300 / 300 / 30`,启用共享备份提交协议,使用唯一的 Stack/VSwitch 名称,并在结束时仅按精确名称做兜底清理。

Qoder 的 host 权限绕过只用于允许测试驱动执行本地 Bash/文件操作,不会批准 ROS Agent 权限。隔离的 iac-code
配置使用默认权限模式,显式允许辅助工具并要求云资源变更工具确认;A2A server 同时保持
`auto_approve_permissions: false`,非只读云操作仍必须通过带完整关联字段的 StartChat 权限回答。

每个场景使用新的目录:

真实 headless Qoder 的单轮超时默认是 900 秒,保证双 candidate Pipeline 不会被测试驱动过早终止;
provider 更慢时可通过 `--qoder-turn-timeout` 显式覆盖,不会削减 Pipeline 场景。

```bash
uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-normal-before \
--mode normal

uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-pipeline-before \
--mode pipeline

# resident 300 秒到期后,在 30 秒 grace 内回答。
uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-normal-grace \
--mode normal \
--answer-delay-seconds 305

# 等非失败挂起完成后再回答。
uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-pipeline-suspended \
--mode pipeline \
--answer-delay-seconds 335

# 在首个权限等待点只终止当前模式的本地 A2A 进程,用同一 config/persistence 目录重启后回答。
uv run python scripts/a2a/e2e/permission_wait/run_start_chat_permission_wait.py \
--allow-real-cloud \
--run-dir /tmp/iac-pwait-normal-restart \
--mode normal \
--restart-at-first-permission
```

Normal 和 Pipeline 都要分别运行 grace、挂起后恢复和进程重启变体。仓库内 prompt 是
`permission_wait/permission_wait_start_chat_prompt.md`。每次运行只保存有界 Qoder turn 摘要、权限观察、relay metrics、
安全结果清单和本地服务日志;退出前会先提取有界只读证据,再删除复制的凭证文件和完整会话 transcript。
发现只读权限弹窗或范围外云写入时,Runner 会直接失败,
不会替用户批准。最终断言要求:真实非只读权限、Normal/顶层 Pipeline 本地与共享 checkpoint、Sub Pipeline 无
checkpoint、确实经过原生 StartChat、精确清理本次资源,以及全部原有 VPC 仍存在。

受控 Sub Pipeline fixture 使用真实 `PipelineRunner` 和两个真实 `AgentLoop` candidate:一个 candidate
停在真实权限 Future,另一个自然完成;到达配置的硬超时后,父 Pipeline 聚合两个 conclusion、进入 candidate
选择并自然完成。fixture 同时安装生产 A2A 备份 hook,证明 Sub 权限本身不会触发权限关键备份。验收运行使用
生产环境的 300 秒配置:

```bash
uv run python scripts/a2a/e2e/permission_wait/run_sub_pipeline_permission_timeout.py \
--run-dir /tmp/iac-pwait-sub-pipeline-300 \
--timeout-seconds 300
```

对应的加速回归和无真实凭证的快速进程重启矩阵为:

```bash
uv run pytest -q tests/a2a_e2e/test_sub_pipeline_permission_timeout.py
uv run pytest -q tests/a2a_e2e/test_permission_wait_restart.py
```

进程重启矩阵覆盖 Normal/Pipeline × allow/deny。Sub Pipeline fixture 断言只生成一次拒绝 ToolResult、
Agent loop 继续、父 Pipeline 进入 candidate 选择并完成,且全程没有 grace、持久化 permission checkpoint
或权限关键备份。

本目录包含用于 A2A pipeline 会话恢复和脱敏回归的 headless 端到端检查。Runner 会驱动公开的
A2A JSON-RPC streaming endpoint 并记录 SSE 事件和 pipeline snapshot。恢复场景会用 `SIGKILL`
杀掉 A2A server,再用相同持久化目录重启;`redaction-step4` 则在候选方案选择处停止,不重启、
Expand Down Expand Up @@ -126,8 +206,8 @@ uv run python scripts/a2a/e2e/run_recovery_scenarios.py \
--scenario selection-during-backup
```

该场景通过仅注入 server 子进程的 E2E fixture,把 step4 `input_required` backup 至少阻塞 10 秒;
客户端收到候选选择事件后立即提交方案。场景会验证选择请求确实在 backup 的 started/finished 窗口内发出、
该场景会在首轮请求前 arm 仅注入 server 子进程的 E2E fixture,把 step4 `input_required` backup 至少阻塞 10 秒;
runner 收到 backup started 标记后立即提交方案,同时继续核验候选选择事件。场景会验证选择请求确实在 backup 的 started/finished 窗口内发出、
最终被消费为 candidate selection,并且没有产生 `interrupt_received` / `interrupt_classified`。

如果要跑完整真实 E2E 矩阵:
Expand Down Expand Up @@ -186,7 +266,7 @@ provider、tool、真实云调用场景默认会被保护住。只有确认要
| `redaction-step4` | 强制 A2A safe mode,真实小程序后端/数据库需求到达 step4 候选方案选择即停止 | 无;不提交方案选择 | canonical 密码参数不是脱敏占位符;A2A 密码值与 canonical 一致;存在的 token 统计仍为数字;已知服务器路径只在 A2A 副本中变成 `[PATH]`;不进入部署。 |
| `scenario1` | pipeline 完成并完成一轮 normal-chat follow-up 后 | 询问上一条 normal-chat 问题是什么 | normal-chat 历史重启后仍可用;存在 VSwitch 证据。 |
| `scenario1-performance-backup` | 完整 `scenario1`,并强制 `IAC_CODE_A2A_EXTREME_PERFORMANCE=true`、`IAC_CODE_CONFIG_BACKUP_DIR=<run-dir>/session-backup` | step4 backup 落盘后停服并删除主 `projects` 下对应 session,重启后不带 `taskId` 选择;后续 normal-chat 恢复也不带 `taskId` | 重启前只有 backup session;重启本身不会重建主 session;省略 `taskId` 的选择会从 backup restore 主 session 并 hydrate 到恢复 task;完整 scenario1 通过。 |
| `selection-during-backup` | step4 `input_required` 已发出,E2E fixture 将随后执行的 backup 至少阻塞 10 | backup 仍在执行时,携带原 task 的 `taskId` 立即发送 `你随便选一个方案。` | 请求时间落在 backup started/finished 窗口内;选择被排队并作为 candidate input 消费;不产生 interrupt 事件;pipeline 完成。 |
| `selection-during-backup` | 首轮请求前 arm E2E fixture;step4 `input_required` backup 开始后至少阻塞 10 秒并写出 started 标记 | backup 仍在执行时,携带原 task 的 `taskId` 立即发送 `你随便选一个方案。`,随后核验 step4 事件 | 请求时间落在 backup started/finished 窗口内;选择被排队并作为 candidate input 消费;不产生 interrupt 事件;pipeline 完成。 |
| `selection-waiting` | step4 等待候选方案选择时 | 不带 `taskId` 发送 `你随便选一个方案。` | 能恢复等待中的 step4 task 并完成选择;存在 VSwitch 证据。 |
| `ask-waiting` | `ask_user_question` 等待用户输入时 | 不带 `taskId` 发送澄清回答 | 能恢复 pending ask 输入并完成 pipeline;存在 VSwitch 证据。 |
| `image-initial` | 首轮用户消息就是静态 `initial.png` 图片 fixture | 文本选择候选方案 | 图片能启动 pipeline,进入 step4 选择,最终完成并产生 VSwitch 证据。 |
Expand Down
Loading
Loading