Chunked transfer for large workspace delivery (Archive Transport)

[EricSanchezok]

Problem

When using Archive Transport, the entire workspace is ZIP-compressed, base64-encoded, and embedded in the START message JSON body. This causes 413 Payload Too Large errors for workspaces exceeding ~75KB (after base64 inflation hits Express's default 100KB JSON limit).

Original Size	After ZIP	After Base64 (+33%)
1 MB	~0.8 MB	~1.07 MB
10 MB	~8 MB	~10.7 MB
100 MB	~80 MB	~107 MB

Current Code Path

Delegator.prepare()
  → createArchive(exportPath, archivePath)
  → buffer = readFile(archivePath)
  → handle = { workspaceBase64: buffer.toString('base64'), checksum }
  → START message = { transportHandle: handle }
  → POST to Executor (JSON body with embedded archive)

Executor HttpListener
  → json()  // Express default 100KB limit
  → 413 if body > limit

Prior Implementation (removed)

Chunked transfer was fully implemented in commits 0e0b760 and 4cde2ec (Feb 9, 2026), then removed in commit 24304d6 (Feb 11, 2026) during the transport delegator/executor split refactor.

What existed:

• ChunkReceiver (executor-side, 193 lines) — idempotent receives, dual checksum validation, timeout, streaming assembly
• ChunkUploader (delegator-side, 216 lines) — streaming reads, parallel upload, per-chunk retry with backoff, resume support
• 3 HTTP endpoints: POST /chunks/:id, GET /chunks/:id/status, POST /chunks/:id/complete
• Auto mode selection: < 2MB → inline base64, ≥ 2MB → chunked
• Documentation: English + Chinese (~2100 lines)
• Integration test scenario with 100 images (~6MB workspace)

Why it was removed:

The refactor in 24304d6 split the monolithic ArchiveTransport into ArchiveDelegatorTransport + ArchiveExecutorTransport. The chunked transfer logic was not ported to the new architecture. The code is fully recoverable from git history at commit 4cde2ec.

Design Discussion

1. Generic chunking type (decided)

ChunkedTransferInfo should be a generic transport concept in @awcp/core, not archive-specific. Any HTTP-based transport (archive, storage, future ones) could use chunked transfer. Snapshot transfers could also benefit from this in the future.

// packages/core/src/types/transport.ts
export interface ChunkedTransferInfo {
  totalSize: number;
  chunkSize: number;
  chunkCount: number;
  totalChecksum: string;
  chunkChecksums: string[];
}

Referenced by ArchiveTransportHandle (and potentially other handles later):

// packages/core/src/types/messages.ts
export interface ArchiveTransportHandle {
  transport: 'archive';
  workspaceBase64?: string;       // small files: inline
  chunked?: ChunkedTransferInfo;  // large files: metadata only
  checksum: string;
}

2. Where ChunkReceiver/ChunkUploader live (open question)

The dependency graph constrains placement:

core ← transport-archive ← transport-storage
core ← sdk ← mcp
        (transport-archive and sdk are siblings, no cross-dependency)

Approach A: SDK-managed chunking (cleaner long-term)

SDK handles all chunk transfer transparently. Transports don't know about chunking.

• DelegatorService detects chunked in handle, creates ChunkUploader, uploads chunks to executor
• ExecutorService manages ChunkReceiver, assembles file, passes local path to transport.setup()
• Requires adding assembledArchivePath?: string to TransportSetupParams so SDK can hand the assembled file to the transport for extraction
• ChunkReceiver/ChunkUploader live in SDK, reusable for future snapshot chunking
• Tradeoff: Changes TransportSetupParams interface (affects all transports), more complex handle/file-path plumbing between SDK and transport

Approach B: Transport-managed chunking (simpler now)

Chunk code lives in transport-archive. SDK only provides HTTP endpoint routing.

• afterStart() hook on DelegatorTransportAdapter — archive transport uploads chunks itself
• receiveChunk() / getChunkStatus() / completeChunks() optional methods on ExecutorTransportAdapter
• SDK's HttpListener adds 3 chunk endpoints, proxies to transport via ExecutorService
• ChunkReceiver/ChunkUploader are internal to transport-archive (no cross-dependency issues)
• Tradeoff: Chunk code is in transport-archive, needs extraction to SDK later if snapshot chunking is needed

3. Protocol flow (both approaches)

Delegator                                    Executor
  │── INVITE ──────────────────────────────►   │
  │◄── ACCEPT ─────────────────────────────   │
  │  prepare(): create ZIP, decide mode        │
  │── START { chunked: metadata } ─────────►   │  → setup() waits for chunks
  │── GET  /chunks/:id/status ─────────────►   │  → resume support
  │── POST /chunks/:id  (chunk 0..N) ──────►   │  → receive, validate, store
  │── POST /chunks/:id/complete ───────────►   │  → assemble → extract → run task
  │── GET  /tasks/:id/events (SSE) ────────►   │
  │◄── event: done ────────────────────────   │

Small files (< chunkThreshold) use existing inline base64 — zero overhead, no chunking.

4. Configuration

// Delegator config
chunkThreshold?: number;      // default 2MB — below this, inline base64
chunkSize?: number;           // default 512KB per chunk
uploadConcurrency?: number;   // default 3 (0 = serial)
chunkRetries?: number;        // default 3 per chunk
chunkTimeoutMs?: number;      // default 30s per chunk

// Executor config
chunkReceiveTimeoutMs?: number;  // default 5 minutes total

5. Key features from prior implementation (to preserve)

• Auto mode selection: threshold-based, transparent to callers
• Streaming reads: ChunkUploader reads byte ranges, doesn't load entire file into memory
• Parallel upload: configurable concurrency with worker pool
• Per-chunk retry: exponential backoff, configurable attempts
• Resume support: query status endpoint, skip already-received chunks
• Triple checksum validation: per-chunk (provided vs computed), per-chunk (computed vs pre-computed), total (assembled vs expected)
• Idempotent receives: duplicate chunks silently ignored
• Timeout protection: ChunkReceiver auto-cleans if chunks stop arriving
• Out-of-order support: chunks can arrive in any order

Estimated Changes

Package	New Files	Modified Files	~Lines
@awcp/core	—	messages.ts, transport.ts	~25
transport-archive	chunk-uploader.ts, chunk-receiver.ts	types.ts, delegator/transport.ts, executor/transport.ts, index.ts	~350
@awcp/sdk	—	http-listener.ts, executor/service.ts, delegator/service.ts, listener/types.ts	~60
Total	2	9	~435

Core chunk logic (~300 lines) recoverable from commit 4cde2ec.

Decision Needed

Approach A vs B — see section 2 above. Approach B is recommended for initial implementation; Approach A can be done as a follow-up refactor when snapshot chunking is needed.