feat: add durable vector store batches and owned workers

[franciscojavierarceo]

Summary

Synchronize this layer with its updated parent after main's AgentPipeline refactor (#274), preserving the immediate-base review boundary.

Stack layer 6 of 7: #34 foundation → #278 pgvector/CI → #280 contextual retrieval/reranking/configuration → #281 streaming Files → #282 store lifecycle/content → #283 durable batches → #284 streaming/SDK compatibility.

Base: codex/vector-store-lifecycle (#282). Next: #284. Review against the immediate base and land from the bottom of the stack.

Adds durable Vector Store file batches: create, retrieve, cancel, and list files with per-file attributes/chunking, filtered pagination, and durable status/counts. Multiple server instances claim bounded jobs with renewable leases and generation fences; cancellation, detach, deletion, expiration, and lost claims prevent stale publication. Completed and failed members remain visible in batch history.

The server now owns and joins the ingestion workers across normal and error exits. Workers resume queued work after restart and run the existing file/store expiration and durable blob cleanup. Shutdown preserves resumability and cooperatively drains owned work; model requests may repeat after recovery, while publication is fenced.

Native batch policy: duplicate IDs and existing non-completed attachments reject atomically; existing completed attachments are reused with their original options. Shared database replicas require a shared Files mount.

Test Plan

Layer-specific implementation verification:

• Focused all-feature core integration: models 18, service 41, tool 6, batches 6, lifecycle 10 passed, including real PostgreSQL 17 / pgvector 0.8.6.
• Storage tests: 80 passed; focused batch fencing tests: 6 passed, including competing claims, stale leases, cancellation, parent mutations, and contended PostgreSQL publication/deletion.
• File-search units: 41 passed; explicit runtime/parser ownership and cleanup tests passed.
• Server HTTP 13, OpenAPI 8, and process lifecycle 11 tests passed.
• Four strict OpenAI Python SDK 3.13.0 lifecycle/batch cases passed against the rebuilt server and loopback model fixtures, including blocked-ingestion cancellation.
• All-feature clippy for both crates/all targets, formatting, diff checks, and all-files pre-commit hooks passed.

Independent task review and scoped fix review passed. Review regressions reproduce a PostgreSQL post-commit lock timeout and slow-heartbeat starvation; both fixes pass, along with 18 covering SQLite/PostgreSQL batch tests and repeated clippy/pre-commit gates.

After synchronization with #274: This updated layer passed locked all-feature workspace compilation, diff checks, and all-files pre-commit hooks. The combined stack in #284 passed 1,389 workspace tests, 31 PostgreSQL/pgvector tests, and 17 maintained SDK cases, plus the Rust 1.88 all-feature check and all-target workspace clippy. The pipeline integration review is complete. GitHub CI is rerunning for the updated heads; previous green checks apply to the prior revisions.