release: v26.5.10 — v1 endpoints + RetentionWorker + worker CLI + repo extraction + SDK helpers

Headlines
- v1 history/billing/stats: GET /queries, /queries/{id}, /queries/{id}/result, /billing, /stats
- RetentionWorker (TTL sweep + stuck-RUNNING reaper + orphan-PENDING republisher)
- Worker CLI: flyquery worker {ingest|retention|all}
- Repository extraction complete: 8 new repos, 0 raw SQL in controllers, CI gate enforces
- Paginated[T] uniform envelope across 14 list endpoints
- /datasets/{id}/files:async endpoint with Stage-1-only sync path
- /datasets/{id}:purge endpoint with object-store walk
- Agent surface: 8 → 28 paths
- Idempotency-Key wired on 5 endpoints
- OpenAPI spec ships header + security + SSE content surface
- Java FlyqueryClient + Python FlyqueryClient with v1 helpers
- 311 unit tests (+53 vs session start), lint clean, lockstep clean
- Two new deep-dive docs: schema-detection.md (6.2k words), workers.md (4.3k words)
- 16 existing docs refreshed for the release
- Release runbook at docs/release.md

Fixes
- Production bug in IngestWorker._drain_inflight (with asyncio.timeout → wait_for)
- Missing return-type annotations on 8 controllers (dropped schemas from spec)
- Docker label was flycanon string
- 4 documented endpoints (/audit, /billing, /stats, /queries) now live

Spec: 64 → 95 paths · 51 → 85 schemas · 22 → 32 tags
Routes: 129 → 142 mounted
Tests: 258 → 311

Author: ancongui

.github/workflows/docker-publish.yaml

@@ -112,7 +112,7 @@ jobs:
             type=raw,value=manual-${{ github.run_id }},enable=${{ github.event_name == 'workflow_dispatch' }}
           labels: |
             org.opencontainers.image.title=flyquery
-            org.opencontainers.image.description=flyquery -- Operational Knowledge Repository. Universal ingestion (any file format), hybrid retrieval (Postgres BM25 + pgvector dense + RRF), grounded RAG answers with citations, append-only version chain + provenance graph + EDA outbox. Part of Firefly OperationOS, platform-agnostic.
+            org.opencontainers.image.description=flyquery -- Operational Structured-Data Intelligence (upload-driven). Multi-tenant ingestion + Text-to-SQL service over user-uploaded structured files (CSV / TSV / XLSX / XLS / ODS / JSON / JSONL / Parquet / Avro / ORC / Arrow / Feather + compression variants). Materialises uploads to Parquet on object storage; indexes a long-lived schema knowledge base; answers natural-language questions via a multi-agent pipeline. Part of Firefly OperationOS.
             org.opencontainers.image.source=https://github.com/${{ github.repository }}
             org.opencontainers.image.licenses=Proprietary
             org.opencontainers.image.vendor=Firefly Software Solutions Inc

CHANGELOG.md

@@ -25,7 +25,7 @@ JSONL, Parquet, Avro, ORC, Arrow and Feather, plus their `.gz`,
 [![OpenAPI](https://img.shields.io/badge/api-openapi%203.1-green)](docs/api-reference.md)
 [![DuckDB](https://img.shields.io/badge/query%20engine-duckdb-yellow)](docs/architecture.md)
 [![pgvector](https://img.shields.io/badge/vector--store-pgvector-336791)](docs/architecture.md)
-[![Version](https://img.shields.io/badge/version-26.5.4-green.svg)](#)
+[![Version](https://img.shields.io/badge/version-26.5.10-green.svg)](#)
 [![License](https://img.shields.io/badge/license-Proprietary-lightgrey.svg)](LICENSE)
 
 </div>

README.md

@@ -87,27 +87,51 @@ tasks:
     cmds:
       - uv run python scripts/openapi_snapshot.py
 
+  sdk:all:
+    desc: Refresh openapi.json snapshot and regenerate both SDKs
+    cmds:
+      - task: openapi-snapshot
+      - task: sdk:python
+      - task: sdk:java
+
   sdk:python:
     desc: Regenerate the Python SDK from openapi.json
     cmds:
       # NOTE: pyproject.toml metadata (authors, description, urls) is
       # preserved across regeneration by .openapi-generator-ignore.
       # Only packageVersion + library are passed as additional-properties;
       # multi-word values are hand-curated in pyproject.toml.
+      #
+      # IMPORTANT: prune the generated api/ + models/ + docs/ trees
+      # FIRST so files for OpenAPI tags that have since been split or
+      # renamed (eg. SchemaApi -> SchemaChangesApi + SchemaObjectsApi)
+      # don't accumulate. .openapi-generator-ignore preserves the
+      # hand-curated README + pyproject + LICENSE + examples/.
+      - rm -rf sdks/python/flyquery_sdk/api
+      - rm -rf sdks/python/flyquery_sdk/models
+      - rm -rf sdks/python/docs
       - npx -y @openapitools/openapi-generator-cli generate
           -i openapi.json
           -g python
           -o sdks/python
           --package-name flyquery_sdk
-          --additional-properties=packageVersion=26.5.6,projectName=flyquery-sdk,library=asyncio
+          --additional-properties=packageVersion=26.5.10,projectName=flyquery-sdk,library=asyncio
           --skip-validate-spec
       - cd sdks/python && uv pip install --system --quiet . || true
 
   sdk:java:
     desc: Regenerate the Java SDK from openapi.json (Java 25 + Spring Boot 3.5.9 + WebFlux)
     cmds:
-      # Remove stale package trees from earlier generator runs.
+      # Remove stale package trees from earlier generator runs (older
+      # versions used org.* / io.* package roots) AND the current
+      # api/ + model/ + docs/ trees so renamed/split tags don't leave
+      # zombie classes behind.
       - rm -rf sdks/java/src/main/java/org sdks/java/src/main/java/io
+      - rm -rf sdks/java/src/main/java/com/firefly/flyquery/api
+      - rm -rf sdks/java/src/main/java/com/firefly/flyquery/model
+      - rm -rf sdks/java/src/test/java/com/firefly/flyquery/api
+      - rm -rf sdks/java/src/test/java/com/firefly/flyquery/model
+      - rm -rf sdks/java/docs sdks/java/api
       - npx -y @openapitools/openapi-generator-cli generate
           -i openapi.json
           -g java
@@ -116,5 +140,5 @@ tasks:
           --invoker-package com.firefly.flyquery
           --api-package com.firefly.flyquery.api
           --model-package com.firefly.flyquery.model
-          --additional-properties=artifactId=flyquery-sdk,groupId=com.firefly,artifactVersion=26.5.6,library=webclient,dateLibrary=java8,java8=true
+          --additional-properties=artifactId=flyquery-sdk,groupId=com.firefly,artifactVersion=26.5.10,library=webclient,dateLibrary=java8,java8=true,useJakartaEe=true
           --skip-validate-spec

Taskfile.yml

@@ -40,7 +40,9 @@ Pick the entry point that matches what you're trying to do.
    durable Postgres outbox default.
 2. [**async-ingest.md**](async-ingest.md) — how `IngestWorker` consumes the
    `flyquery.ingest` topic, job lifecycle, cooperative cancel, retry + dead-letter.
-3. [**architecture.md**](architecture.md) — how `pyfly.eda.EventPublisher` is
+3. [**workers.md**](workers.md) — the worker fleet topology (IngestWorker +
+   RetentionWorker), CLI subcommands, scaling, recovery, and deployment forms.
+4. [**architecture.md**](architecture.md) — how `pyfly.eda.EventPublisher` is
    injected into the ingestion and query services.
 
 ### "I want to understand how it works"
@@ -52,7 +54,10 @@ Pick the entry point that matches what you're trying to do.
    diagrams; mode coupling (sync vs async via EDA).
 3. [**ingestion.md**](ingestion.md) — stage-by-stage ingestion detail: each
    stage's inputs, outputs, skip policy, and SSE events.
-4. [**prompts.md**](prompts.md) — catalog of the 7 agent prompts: instruction
+4. [**schema-detection.md**](schema-detection.md) — XLSX/CSV/JSON parsing
+   deep dive, type inference, drift detection, PII tagging, and the
+   column-name proposer agent.
+5. [**prompts.md**](prompts.md) — catalog of the 7 agent prompts: instruction
    text, input shape, output schema.
 
 ### "I'm running this in production"
@@ -138,8 +143,10 @@ Pick the entry point that matches what you're trying to do.
 | [integration-with-firefly-os.md](integration-with-firefly-os.md) | You're wiring flyquery with flycanon or flyradar. |
 | [glossary.md](glossary.md) | You need a precise definition for a term the API or docs use. |
 | [scale-and-performance.md](scale-and-performance.md) | You're capacity-planning or tuning FLYQUERY_* knobs. |
-| [billing.md](billing.md) | You're tracking per-query and per-ingest LLM cost via `flyquery_cost_events`. |
+| [billing.md](billing.md) | You're tracking per-query and per-ingest LLM cost via `flyquery_cost_events` and `GET /api/v1/billing`. |
 | [stats.md](stats.md) | You're rendering a workspace dashboard via `GET /api/v1/stats`. |
+| [schema-detection.md](schema-detection.md) | You're debugging XLSX/CSV/JSON parsing, type inference, drift detection, or the column-name proposer. |
+| [workers.md](workers.md) | You're running, scaling, or recovering the `IngestWorker` + `RetentionWorker` fleet. |
 | [quality.md](quality.md) | You're running or extending the test suite — unit, integration, parser fixtures, pipeline tests. |
 | [firefly-intelligence-system.md](firefly-intelligence-system.md) | You want the three-pillar narrative (flycanon + flyradar + flyquery). |
 | [../sdks/python/README.md](../sdks/python/README.md) | You're integrating from Python (async-first SDK, Pydantic types). |
@@ -164,14 +171,16 @@ Pick the entry point that matches what you're trying to do.
 | Agent-token surface | [security-model.md § 4](security-model.md) | [api-reference.md](api-reference.md) |
 | EDA / typed events | [eda-events.md](eda-events.md) | [async-ingest.md](async-ingest.md) |
 | Snapshot versioning + pinning | [ingestion.md](ingestion.md) | [conversations.md](conversations.md) |
-| Schema drift + rename detection | [ingestion.md § RENAMED_CANDIDATE](ingestion.md) | [pipeline.md](pipeline.md) |
+| Schema drift + rename detection | [schema-detection.md](schema-detection.md) | [ingestion.md § RENAMED_CANDIDATE](ingestion.md) |
+| Format detection + parser dispatch | [schema-detection.md](schema-detection.md) | [file-formats.md](file-formats.md) |
+| Worker fleet + scaling | [workers.md](workers.md) | [async-ingest.md](async-ingest.md), [deployment-topology.md](deployment-topology.md) |
 | Object-store layout | [architecture.md § Storage architecture](architecture.md) | [deployment.md](deployment.md) |
-| Billing + cost stream | [billing.md](billing.md) | [api-reference.md](api-reference.md) |
+| Billing + cost stream | [billing.md](billing.md) | [api-reference.md](api-reference.md), [cost-tracking.md](cost-tracking.md) |
 | Workspace stats | [stats.md](stats.md) | [api-reference.md](api-reference.md) |
 | Lock-step modules | [architecture.md § Lock-step](architecture.md) | [cicd.md](cicd.md) |
 | GDPR purge | [operations-runbook.md](operations-runbook.md) | [security-model.md § Upload security](security-model.md) |
 | RFC 7807 error envelope | [payload-reference.md](payload-reference.md) | [api-reference.md](api-reference.md) |
-| Multi-replica concurrency | [concurrency.md](concurrency.md) | [async-ingest.md](async-ingest.md) |
+| Multi-replica concurrency | [concurrency.md](concurrency.md) | [workers.md](workers.md), [async-ingest.md](async-ingest.md) |
 
 ---
 

docs/README.md

@@ -292,6 +292,39 @@ SDK helpers:
 
 ---
 
+#### `POST /api/v1/datasets/{id}/files:async` — async upload
+
+Like `POST /datasets/{id}/files` but the response is a `202 Accepted`
+with a job id. Stage 1 (receive: caps check, hash, format detect,
+store bytes, write `flyquery_files` row, track workspace storage)
+runs **synchronously** before responding — `file_id` is final and the
+content hash provides natural per-file dedup. Stages 2-10 of the
+pipeline are queued as a `PARSE_AND_INGEST` ingest job that the
+`IngestWorker` consumes (the worker honours an `already_received`
+flag on the job so it does NOT re-run Stage 1; see
+[async-ingest.md](async-ingest.md) and
+[workers.md](workers.md) for the worker side).
+
+**Request** `Content-Type: multipart/form-data` — one `file` part.
+
+**Response** `202 Accepted` with `Location: /api/v1/ingest-jobs/{job_id}`
+```json
+{
+  "job_id": "01906f2c-...",
+  "file_id": "01906f2b-...",
+  "dataset_id": "01906f2a-...",
+  "status": "PENDING"
+}
+```
+
+Use in place of the synchronous endpoint when a single file exceeds
+the request-timeout budget of the deployment (typically anything
+above a few MB once cold-cache LLM describe calls land).
+
+Source: `files_controller.py:249` (upload_file_async).
+
+---
+
 #### `GET /api/v1/datasets/{id}/files`
 
 List files for a dataset.
@@ -883,60 +916,163 @@ Add a drill-down turn. The prior turn's `executed_sql`, `table_qnames`, and
 
 ### 5.12 History and ops
 
+#### `GET /api/v1/audit-events`
+
+Paginated audit-event ledger. Filters: `event_type`, `actor`,
+`resource_kind`, `date_from`, `date_to`, `limit`, `offset`. Response
+is the standard `Paginated[AuditEventRead]` envelope. Newest first.
+
+Today's writers (more callsites land over time):
+- Dataset CRUD (`dataset.created`, `dataset.updated`, `dataset.archived`, `dataset.purged`)
+- Workspace CRUD (`workspace.created`, `workspace.updated`)
+- Agent-token mint / revoke (`agent_token.minted`, `agent_token.revoked`)
+
+Future scope gate: `flyquery.audit:read`.
+
+---
+
+#### `GET /api/v1/cost-events`
+
+Paginated per-call LLM cost ledger. Filters: `actor`, `model`,
+`operation`, `date_from`, `date_to`, `limit`, `offset`. Response is
+`Paginated[CostEventRead]`. One row per LLM-bearing operation.
+
+Today's writers cover the query pipeline (grounding / generation /
+critic / explainer) and the ingest pipeline (describe / column-name
+proposer / relation proposer / rename detector).
+
+Future scope gate: `flyquery.billing:read`. The `GET /api/v1/billing`
+rollup below is the aggregation on top of this raw stream — see
+[billing.md](billing.md).
+
+---
+
+> **Live as of 26.5.10.** The endpoints below were placeholders in
+> earlier releases; the read-side query history + billing rollup +
+> stats endpoints are now shipped on top of the pre-existing
+> `flyquery_queries`, `flyquery_cost_events`, and workspace inventory
+> tables. See [`src/flyquery/web/controllers/queries_controller.py`](../src/flyquery/web/controllers/queries_controller.py),
+> [`billing_controller.py`](../src/flyquery/web/controllers/billing_controller.py),
+> [`stats_controller.py`](../src/flyquery/web/controllers/stats_controller.py).
+
 #### `GET /api/v1/queries`
 
-Paginated query history. Optional: `dataset_id`, `execution_status`,
-`semantic_path_taken`, `date_from`, `date_to`.
+Paginated query history scoped to the caller's workspace, newest first.
+
+**Query params** — `dataset_id`, `execution_status`,
+`semantic_path_taken`, `date_from` (inclusive), `date_to` (exclusive),
+`limit` (clamped to `[1, 200]`), `offset`.
+
+Each row is the compact `QueryHistoryItem` shape — heavy JSONB columns
+(candidates, clarification, PII findings) are omitted; use the detail
+endpoint below for the full payload. Response is the standard
+`Paginated[QueryHistoryItem]` envelope.
+
+Source: `queries_controller.py:52` (list_queries).
 
 ---
 
 #### `GET /api/v1/queries/{id}`
 
-Full query detail including all candidates, retries, cost, model identifiers.
+Full single-query payload including every candidate proposal, the AST
+classification, every model identifier used
+(`model_grounding` / `model_generation` / `model_critic` /
+`model_explainer`), PII findings, clarification frame, retries, and the
+final error envelope if any. Returns `QueryDetailRead`.
+
+Cross-tenant probing returns 404 (the same response as a missing query),
+so the endpoint never leaks the existence of another tenant's row.
+
+Source: `queries_controller.py:100` (get_query).
 
 ---
 
 #### `GET /api/v1/queries/{id}/result`
 
+Re-download the preview + a fresh presigned Parquet URL for a previously
+executed query.
+
 ```json
 {
+  "query_id": "01906f40-...",
   "preview_json": [...],
   "parquet_presigned_url": "https://...",
   "result_byte_size": 204800,
   "ttl_expires_at": "2026-05-24T12:00:00Z"
 }
 ```
 
----
-
-#### `GET /api/v1/audit`
+`parquet_presigned_url` is `None` once `ttl_expires_at` has elapsed
+(default 24h, controlled by `FLYQUERY_RESULT_TTL_HOURS`) — the consumer
+must rerun the query to materialise a fresh URL. A presign-time failure
+(object reclaimed, backend transient outage) also collapses the URL to
+`None` while still returning the inline preview.
 
-Paginated audit events. Optional: `event_type`, `actor`, `date_from`, `date_to`.
+Source: `queries_controller.py:124` (get_query_result).
 
 ---
 
 #### `GET /api/v1/billing`
 
-Cost rollup by period.
+Aggregates `flyquery_cost_events` into day / week / month buckets
+scoped to the caller's workspace. Consume
+[`GET /api/v1/cost-events`](#cost-events) for the raw per-call ledger;
+this endpoint just rolls it up.
 
-**Query params**: `period` (day|week|month), `date_from`, `date_to`.
+**Query params**: `period` = `day` (default) | `week` | `month`,
+`date_from` (inclusive), `date_to` (exclusive).
+
+Returns `BillingRollup` with the split `ingest_cost_cents` /
+`query_cost_cents` / `other_cost_cents` per bucket plus the
+`total_cost_cents` sum. Buckets with zero cost are omitted (no empty
+days). An invalid `period` returns 400 `invalid_request`.
 
-**Response**
 ```json
 {
+  "period": "day",
+  "date_from": "2026-05-01T00:00:00Z",
+  "date_to": "2026-05-24T00:00:00Z",
   "total_cost_cents": 14230,
   "breakdown": [
-    {"date": "2026-05-23", "ingest_cost_cents": 200, "query_cost_cents": 14030}
+    {
+      "date": "2026-05-23T00:00:00Z",
+      "ingest_cost_cents": 200,
+      "query_cost_cents": 14030,
+      "other_cost_cents": 0,
+      "total_cost_cents": 14230
+    }
   ]
 }
 ```
 
+See [billing.md](billing.md) and [cost-tracking.md](cost-tracking.md)
+for the cost model and which call sites currently write to the ledger.
+Scope: `flyquery.billing:read`.
+
+Source: `billing_controller.py:32` (rollup).
+
 ---
 
 #### `GET /api/v1/stats`
 
-Workspace usage summary: `storage_used_bytes`, `dataset_count`, `table_count`,
-`query_count_last_30d`, `token_count_last_30d`.
+Compact workspace summary, returned as `WorkspaceStats`. Six fields:
+
+```json
+{
+  "storage_used_bytes": 524288000,
+  "dataset_count": 5,
+  "table_count": 23,
+  "query_count_last_30d": 312,
+  "token_count_last_30d": 412380,
+  "ingest_job_count_pending": 0
+}
+```
+
+Operator traffic, not hot-path — backed by a small set of `COUNT(*)`
+queries on demand without caching. See [stats.md](stats.md) for field
+semantics. Scope: `flyquery.billing:read` (reused).
+
+Source: `stats_controller.py:32` (workspace_summary).
 
 ---
 
@@ -946,6 +1082,27 @@ Returns `{version, git_sha, build_time}`.
 
 ---
 
+#### Health probes — `GET /actuator/health` and siblings
+
+Pyfly's actuator is enabled and exposes Spring-Boot-style health
+probes. **Use these instead of ad-hoc `/healthz` / `/readyz` paths**:
+
+| Path | Purpose |
+|---|---|
+| `GET /actuator/health` | Aggregate of every registered `HealthIndicator`. Returns `{"status": "UP" \| "DOWN"}`. |
+| `GET /actuator/health/liveness` | Kubernetes liveness probe (process alive?). |
+| `GET /actuator/health/readiness` | Kubernetes readiness probe (deps reachable?). |
+| `GET /actuator/info` | `{"app": {"name", "version", "description"}}` from `pyfly.yaml`. |
+| `GET /actuator/metrics` | Prometheus-compatible counters. |
+| `GET /actuator/env` | Runtime configuration view (gate behind auth in production). |
+| `GET /admin/*` | Full admin UI surface for development. |
+
+These are **not** under `/api/v1/*` (deliberate — they're operator
+endpoints, not tenant-scoped business endpoints) and therefore don't
+require `X-Tenant-Id` / `X-Workspace-Id`.
+
+---
+
 ### 5.13 Agent tokens
 
 Operator-only. Creates long-lived bearer tokens for agent-tier access.