system-e2e-matrix.md

System E2E matrix (HTTP + MatrixOne)

This document maps user-visible capabilities to HTTP routes, persistence (MatrixOne tables or in-process stores), and the integration tests that assert them. It complements router_builder.rs unit tests (route registration only).

Layout (system E2E plan): the integration binary is crates/runtime/tests/system_matrix_http_e2e/main.rs with shared harness.rs (env gate, HTTP, sqlx, bootstrap) and journey modules such as journey_full.rs, journey_tasks_runs.rs, journey_extended.rs, journey_delegate_http_matrix.rs, journey_admin_smoke_matrix.rs, and other journey_*.rs files. Every active chat admission in this matrix uses the Server-owned execution path; stream callbacks are correlated back to that live run. Provider fixtures exercise actual HTTP requests, response parsing, inference admission, and durable settlement. Request-context mock rounds and their separate host loop are retired. Cargo.toml names the test target system_matrix_http_e2e and enables e2e-hooks.

How to run

ASTRA_TEST_DB_IT=1 \
ASTRA_TEST_E2E_SECRET=system-matrix-e2e-secret \
ASTRA_BACKEND_SERVICE_KEY=test-service-key-e2e \
ASTRA_LLM_RETRY_BASE_MS=10 ASTRA_DEFAULT_RETRY_AFTER_MS=10 ASTRA_BCRYPT_COST=4 \
RUST_MIN_STACK=16777216 \
cargo test -p astra-runtime --test system_matrix_http_e2e --features e2e-hooks -- \
  --ignored --nocapture

Requires the same environment as astra-server: MATRIXONE_*, ASTRA_JWT_SECRET, ASTRA_TOKEN_ENCRYPTION_KEY, Memoria, and related settings parsed by astra_core::AppSettings::from_env. Use a local .env if you use one for development.

Environment variables (对照表)

Variable	Role	Notes
ASTRA_TEST_DB_IT	Gate	Must be 1 or ignored tests panic in require_system_e2e_env
ASTRA_TEST_E2E_SECRET	Explicit system-lane gate	Required by the test runner; lifecycle timing barriers are compile-time e2e-hooks fixtures. Provider responses come from strict loopback HTTP fixtures
ASTRA_BACKEND_SERVICE_KEY	Service-edge fixture	Non-empty test-only key for /service/edges/status authentication coverage
ASTRA_LLM_RETRY_BASE_MS, ASTRA_DEFAULT_RETRY_AFTER_MS, ASTRA_BCRYPT_COST	Deterministic test timing	Runner-owned low-latency values; the E2E harness never changes process environment
RUST_MIN_STACK	Tokio test worker stack	Set to 16777216 so the E2E runtime matches Astra's production process runtime
MATRIXONE_HOST	DB	Default localhost
MATRIXONE_PORT	DB	Default 6001
MATRIXONE_USER	DB	Default root
MATRIXONE_PASSWORD	DB	Default dev password in astra_core::runtime_limits if unset
ASTRA_DATABASE	DB	Base name; default astra_runtime
ASTRA_DATABASE_PREFIX	DB	Optional; effective DB = prefix + ASTRA_DATABASE (e.g. test_ + astra_runtime)
ASTRA_JWT_SECRET	Auth tokens	Default dev string if unset (not for production)
ASTRA_TOKEN_ENCRYPTION_KEY	Token encryption	Default dev string if unset
MEMORIA_EMBEDDING_*	Embeddings config	MEMORIA_EMBEDDING_DIM may be required for unknown models
ASTRA_RUNTIME_ROOT_SECRET	Runtime signature root	Required by the production-shaped server state; artifact and execution-grant keys are purpose-derived from it
ASTRA_TEST_DB_IT_TEST_THREADS	Makefile and CI	Set to 1 to run online integration binaries serially (-j 1)

Evaluation read routes in the full journey use x-user-id without bearer (see journey_full). Other authenticated calls use the JWT from bootstrap.

Test binaries (ignored by default)

Ignored tests in system_matrix_http_e2e avoid overlap with the full journey (e.g. no separate “basic session” test that repeats the same list/get/close/resume steps).

Related (separate crate / gate): ASTRA_TEST_DB_IT=1 runs cargo test -p astra-services --test services_db_integration -- --ignored (MatrixOne): pagination clamps, skills_registry list/index, cross-session audit paths, session restore, and owner-bound sync behavior (see that test file’s module doc).

Test name	File / module	Scope
product_matrix_api_journey_hits_multiple_tables	journey_full.rs	Full journey: sessions (list/get/put, close/resume, activity, platform snapshot), agents, events, context, decisions, memory proxy, edge, jobs, sandbox, triggers, skills, introspection, reflect/decision-trace, evaluation reads, marketplace probe, server-owned chat/stream SSE + agent_events, audit ownership, logout
e2e_matrix_chat_run_pause_resume_http	journey_tasks_runs.rs	POST /chat (background run), POST .../pause, GET /chat/runs/{id}, POST .../resume
e2e_matrix_session_cancel_delete	journey_extended.rs	POST /sessions/{id}/cancel + agent_sessions.status, DELETE /sessions/{id}
e2e_matrix_chat_stream_session_info	journey_extended.rs	POST /chat/stream SSE → session_info + run_id
e2e_matrix_approval_respond_invalid_session_id	journey_extended.rs	POST /approval/respond with an unsafe session_id; assert 400 rejects invalid journal path components
e2e_matrix_edge_callback_http_boundary_failures	journey_extended.rs	POST /tools/result and /approval/respond without auth or with malformed payloads; assert auth/client errors at the HTTP boundary
e2e_matrix_duplicate_tool_result_idempotency	journey_extended.rs	Start one server-owned POST /chat/stream, then POST /tools/result twice for the same request_id; assert one request, one durable result, final model text, and exactly one terminal with continuation_owner=server
e2e_matrix_duplicate_approval_response_idempotency	journey_extended.rs	POST /approval/respond twice for the same request_id + session_id; assert one persisted approval_decision in the session journal
e2e_matrix_server_stream_partial_batch_failure	journey_extended.rs	Start one server-owned stream that emits two tool_requests; reply with one success and one failure, then assert the same stream reaches final model text and exactly one server terminal
e2e_matrix_server_stream_out_of_order_tool_results	journey_extended.rs	Start one server-owned stream that emits two callbacks; send the second result before the first, assert request-identity correlation, one durable effect per callback, final model text, and one server terminal
e2e_matrix_auth_session_negative_paths	journey_extended.rs	GET /sessions without auth (401); duplicate register + bad login + successful login after negative calls
e2e_matrix_memory_proxy_user_isolation	journey_extended.rs	Unauthenticated POST /memory/store (401); spoofed user_id/session_id in body → forwarder receives JWT user_id for both fields
e2e_matrix_models_admin_crud	journey_extended.rs	SQL astra_admin role grant + POST/PUT/DELETE /models with DB checks
e2e_matrix_saas_events_and_audit_cross_user_isolation	journey_saas_platform_matrix.rs	Foreign-user GET /events, session audit, and activity return 404; filtered global events do not leak the foreign session
e2e_matrix_meta_health	journey_meta_matrix.rs	GET /, GET /health (root metadata, DB connected, persist counters)
e2e_matrix_session_http_db	journey_session_http_db_matrix.rs	GET/PUT /sessions/{id} vs agent_sessions (title, user_id)
e2e_matrix_evaluation_reads	journey_evaluation_reads_matrix.rs	Evaluation GET smoke (x-user-id), seed agent for trust/SLO/observability
e2e_matrix_context_decision_chain	journey_context_decision_chain_matrix.rs	Event → context → decision chain + ctx_snapshots / ctx_decision_audits SQL
e2e_matrix_models	journey_models_matrix.rs	paginated GET /models + cursor continuation, global-count GET /model-access, stable revision/default semantics
e2e_matrix_delegate_http_boundaries	journey_delegate_http_matrix.rs	POST /chat → run_id; GET /chat/runs/{id}/delegations; POST .../delegate validation 400
e2e_matrix_saas_admin_tokens_rbac_smoke	journey_saas_platform_matrix.rs	GET /admin/tokens: 403 → grant astra_admin → 200 JSON array

Shared helpers: crates/runtime/tests/system_matrix_http_e2e/harness.rs (bootstrap, grant_astra_admin_role, revoke_astra_admin_role, HTTP helpers, cleanup_*, row getters, SSE helpers, wait_for_agent_event_types — polls agent_events after server-owned chat/stream instead of a fixed sleep).

Database isolation

• Shared database: All tests use the same MatrixOne database from AppSettings (typically astra_runtime). There is no separate schema per test.
• Row isolation: Each bootstrap() registers a new user (prod_matrix_{uuid}), creates a new session_id, and uses an edge_agent_id / suffix unique to that run. API state and SQL assertions are scoped by those IDs.
• Parallel runs: Tests are safe to run in parallel by default (cargo / make test-online without ASTRA_TEST_DB_IT_TEST_THREADS=1). The full journey uses a suffix-scoped marketplace skill name (e2e_matrix_mkt_{suffix}) so concurrent runs do not fight over the same global marketplace stats key.
• Opt-in serial: If database connection limits or another shared dependency makes the lane flaky, run with ASTRA_TEST_DB_IT_TEST_THREADS=1. The Makefile passes -j 1 to the online nextest invocations.

API groups vs coverage (P0 / P1)

Legend: DB = SQL assertion on MatrixOne; HTTP = response-only; — = not covered by system E2E yet.

Group	P	Representative routes	Persistence check	Test(s)
Meta	P0	GET /health, GET /	—	product_matrix_*, e2e_matrix_meta_health
Auth	P0	/auth/register, /login, /refresh, /me, /logout	auth_users	Every test uses bootstrap (register/login); product_matrix_* also hits /auth/refresh and /logout
Sessions	P0	/sessions, .../close, .../resume, .../cancel, DELETE ..., .../activity	agent_sessions	product_matrix_* + e2e_matrix_session_cancel_delete + e2e_matrix_session_http_db
Session audit	P0	/sessions/{id}/audit/*, /audit/*	agent_events + HTTP cross-check	product_matrix_* (summary/turn counts and token totals); e2e_matrix_saas_events_and_audit_cross_user_isolation (foreign-user events/audit/activity isolation)
Agents	P0	/agents CRUD	agent_agents	product_matrix_*
Models	P1	GET /models, admin POST/PUT/DELETE /models	infra_llm_models	product_matrix_* (list), e2e_matrix_models (list/access); e2e_matrix_models_admin_crud (admin CRUD + DB)
Events	P0	/events, causal chain, session events	agent_events	product_matrix_*
Context	P0	/context	ctx_snapshots	product_matrix_*, e2e_matrix_context_decision_chain
Decisions	P0	/decisions, audit	ctx_decision_audits	product_matrix_*, e2e_matrix_context_decision_chain
Memory proxy	P1	/memory/*	Memoria stub calls	product_matrix_*
Edge §5.5	P0	/agents/edge, /tools/result, /approval/respond	edge_agent_registry, durable run events	product_matrix_*, e2e_matrix_approval_respond_invalid_session_id, e2e_matrix_edge_callback_http_boundary_failures, e2e_matrix_saas_edge_tool_result_success_path, duplicate/mixed/out-of-order server-stream callback journeys, e2e_matrix_duplicate_approval_response_idempotency; legacy Task Lease claim/renew/release routes are not registered by runtime, so no live E2E is claimed
Sandbox	P1	/sandbox	infra_sandbox_metadata	product_matrix_*
Skills / introspection	P1	/skills, /introspection/*	mixed	product_matrix_*
Evaluation (writes)	P1	POST gate/validate, drift/run, loop	—	— (no system E2E; add when implementations return success)
Marketplace	P1	quality report, stats, search	marketplace stats tables	product_matrix_*
Chat (server-owned SSE)	P0	POST /chat/stream	agent_events, session_transcript_items, audit/session projections	product_matrix_*, e2e_matrix_chat_stream_session_info, e2e_matrix_edge_callback_http_boundary_failures, e2e_matrix_saas_edge_tool_result_success_path, duplicate/mixed/out-of-order callback journeys
Chat / runs	P0	POST /chat, /chat/stream, /chat/runs/*	In-memory run store in build_server_state (not Matrix table today)	e2e_matrix_chat_run_pause_resume_http, e2e_matrix_chat_stream_session_info
Platform	P1	GET /platform/snapshot	—	product_matrix_*
Data versioning	P1	lineage GETs	—	product_matrix_*
Admin	P1	GET /admin/tokens	—	e2e_matrix_saas_admin_tokens_rbac_smoke
WebSocket	—	/chat/ws	—	—
Delegation	P1	GET .../delegations, POST .../delegate (validation-only path)	In-memory tracker	e2e_matrix_delegate_http_boundaries

CI

• PR (.github/workflows/test.yml): offline tests are sharded by package; two online lanes start MatrixOne and Memoria through make dev-deps-up, set ASTRA_TEST_DB_IT=1 and ASTRA_TEST_DB_IT_TEST_THREADS=1, and run the core and integration groups separately. See also coverage-matrix.md.
• Manual / nightly: .github/workflows/e2e-matrix-nightly.yml — workflow_dispatch with optional test name filter (substring) to run a subset (e.g. e2e_matrix_chat_run_pause_resume_http) or leave empty for all ignored tests in the binary.

Router groups alignment

Same prefixes as router_builder all_api_groups_have_routes (integration tests only check registration; this table tracks system E2E).

Group (router_builder)	Prefix	System E2E	Notes
auth	/auth/	Yes	auth_users in bootstrap / product_matrix_*
chat	/chat	Partial	Server-owned /chat/stream SSE + agent_events in product_matrix_*, callback HTTP-boundary and duplicate/mixed/out-of-order callback journeys, SaaS success-path correlation, POST /chat + run pause/resume in e2e_matrix_chat_run_pause_resume_http, and delegation list + POST .../delegate validation boundary in e2e_matrix_delegate_http_boundaries; no /chat/ws E2E
sessions	/sessions	Yes	CRUD/close/resume/activity + DB
admin	/admin/	Partial	GET /admin/tokens smoke in e2e_matrix_saas_admin_tokens_rbac_smoke
agents	/agents	Yes	Includes edge register path
events	/events	Yes
skills	/skills	Partial	List/status; not publish/config/resources E2E
introspection	/introspection/	Yes
marketplace	/marketplace/	Partial	Quality report / stats / search; not full install/upgrade/rollback/credentials
sandbox	/sandbox	Yes
platform	/platform/	Partial	GET /platform/snapshot in product_matrix_*
runs	/runs	Partial	List in product_matrix_*; lifecycle in e2e_matrix_chat_run_pause_resume_http

Additional route families in router_builder not named above: memory (/memory/*), context (/context), decisions (/decisions), models (/models), data-versioning (/data-versioning), reflect (/chat/session/.../reflect), completions (/v1/chat/completions) — see the P0/P1 table above for E2E status.

Explicit coverage gaps

• Client-disconnect behavior for streamed tool-call artifacts is intentionally not registered in this matrix; the retained server_loop_* artifact tests cover block-parse, transport, idle-timeout, and rate-limit outcomes only.
• Malformed streamed tool-call artifact handling is not registered here. Keep that provider/parser contract in focused server-loop tests until a stable streamed fixture can prove durable partial state without a second admission.

Future work

• Runs + DB: when RunStateStore is backed by Matrix for build_server_state, add SQL assertions alongside e2e_matrix_chat_run_pause_resume_http.
• Evaluation writes: add a focused test when validate_gate / run_drift_pipeline return 200 with stable response shapes.
• Branches / admin HTTP (beyond cost estimate + token list): optional deeper journeys when routes stabilize.
• /chat/ws, successful delegation execute (long-running): optional fixtures; validation-only delegation is in e2e_matrix_delegate_http_boundaries.
• Real Memoria: optional second target with a Memoria test double URL instead of the stub forwarder.