feat(tests): SDK compatibility E2E test framework and quality gates (#864)#883
Closed
sparkzky wants to merge 4 commits into
Closed
feat(tests): SDK compatibility E2E test framework and quality gates (#864)#883sparkzky wants to merge 4 commits into
sparkzky wants to merge 4 commits into
Conversation
Add a first-class pytest-based SDK compatibility E2E test framework under tests/e2e/sdk_compat/. The same backend-neutral test cases run against both the CubeSandbox Python SDK and E2B-compatible Python SDKs. Framework components: - Adapter layer (SandboxAdapter ABC) abstracts SDK differences into a shared interface: create, info, commands, files, run_code, pause/resume, set_timeout, kill - Capability markers skip unsupported backends without branching in tests - Preflight checks validate API health + template readiness before runs - JSONL structured reporting for diagnostics - Best-effort sandbox cleanup with REST fallback - Default PR gate runs cubesandbox backend; SDK_E2E_BACKENDS for dual-SDK Coverage domains (38 tests across 7 domains): - lifecycle: create, info, kill, set_timeout, pause/resume - commands: stdout/stderr/exit_code, env, cwd, special chars, multiline, large output, missing binary - filesystem: read/write, overwrite, multiline, exists, list, make_dir, remove, cross-API interoperability - run_code: expression text, stdout, kernel state, multiline blocks, string/list output, imports, error reporting - metadata: env var propagation, cross-API metadata interoperability - errors: info-after-kill, double-kill idempotency - concurrency: sequential reuse, parallel sandbox isolation CI workflow (.github/workflows/sdk-compat-e2e.yml) validates test collection on PRs touching sdk_compat or the Python SDK; optional live E2E job runs when deployment secrets are configured. Autonomously-by: omp:zai/glm-5.2 Signed-off-by: sparkzky <sparkhhhhhhhhhh@outlook.com>
The CubeAPI /connect endpoint requires a `timeout` field in the request
body. The SDK was sending `{}`, causing a 400 "missing field `timeout`"
on every connect-after-pause call.
Sandbox.connect() now accepts an optional `timeout` parameter (defaults
to config.timeout) and includes it in the POST body.
Discovered by the SDK compat E2E pause/resume tests against a live
CubeSandbox deployment — all 38 E2E tests pass after this fix.
Autonomously-by: omp:zai/glm-5.2
Signed-off-by: sparkzky <sparkhhhhhhhhhh@outlook.com>
Complete the capability domain coverage listed in issue TencentCloud#864: - cases/network/: DNS resolution, HTTP/HTTPS egress (requires_internet) - cases/proxy/: CubeProxy routing for filesystem, commands, run_code (requires_capability proxy_url, requires_cubeproxy) 44 tests across all 9 domains, all passing against live deployment. Autonomously-by: omp:zai/glm-5.2 Signed-off-by: sparkzky <sparkhhhhhhhhhh@outlook.com>
E2B SDK 2.30+ validates the API key format client-side (requires e2b_ prefix + hex). Self-hosted CubeSandbox deployments with auth disabled accept any key at the server level, but the SDK rejects non-conforming keys before sending the request. Use a format-valid placeholder key so dual-backend E2E tests work against local deployments without a real E2B account. Verified: 82 passed, 6 skipped (cubesandbox-only capabilities) across both backends. Autonomously-by: omp:zai/glm-5.2 Signed-off-by: sparkzky <sparkhhhhhhhhhh@outlook.com>
sparkzky
requested review from
chenhengqi,
fslongjin,
ls-ggg and
tinklone
as code owners
Member
|
duplicated with: #835 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Build a first-class SDK compatibility E2E test framework for CubeSandbox, enabling the same user-facing scenarios to be validated across both the
cubesandboxnative Python SDK ande2b-compatible Python SDKs.Resolves #864.
Changes
1. SDK Compatibility E2E Framework
Added
tests/e2e/sdk_compat/with a complete pytest-based framework:adapters/):SandboxAdapterABC with 16 backend-neutral methods.CubeSandboxAdapterandE2BAdapterwrap each SDK, normalizing return types and behavioral differences (e.g., E2B raises exceptions on non-zero exit codes vs CubeSandbox returns results).framework/capabilities.py):requires_capabilitymarker auto-skips unsupported tests — E2B backend correctly skipsset_timeout,pause_resume, andproxy_urlwith zero branching inside test bodies.framework/preflight.py): session-scoped validation of CubeAPI health and template readiness, with early exit to avoid wasting resources.framework/reporting.py): JSONL events for preflight, sandbox create/cleanup, and test results (backend, sandbox ID, duration, outcome, error diagnostics).framework/cleanup.py):safe_killnever raises — REST DELETE fallback, original test failure always preserved.2. Nine Capability Domains, 44 Backend-Neutral Test Cases
3. SDK Bug Fix
E2E tests discovered
Sandbox.connect()sent an empty JSON body{}, but the CubeAPI/connectendpoint requires atimeoutfield. Fixed to send{"timeout": <value>}.4. CI Workflow
.github/workflows/sdk-compat-e2e.yml:--run-e2erun when deployment secrets are configuredVerification
Dual-backend E2E run against a local PVM deployment:
Usage
Autonomously-by: omp:zai/glm-5.2