The combined narrative for flycanon (operational knowledge repository), flyradar (operations discovery and diagnostic intelligence), and flyquery (structured-data Text-to-SQL). These three services form the intelligence backbone of Firefly OperationOS.
flycanon is the long-lived knowledge boundary for unstructured and semi-structured content:
- Universal ingestion: documents, process artefacts, reports, observations.
- Hybrid retrieval: pgvector HNSW + Postgres BM25 with Reciprocal Rank Fusion.
- RAG answers with citations.
- Candidate → Accept workflow for human-validated knowledge items.
- Per-workspace isolation enforced via Postgres RLS.
flycanon owns the truth across ingested knowledge. It does not produce structured query results or process-discovery graphs.
flyradar turns heterogeneous corpus (logs, system signals, documents, observations) into:
- An AS-IS process graph (BPMN-shaped).
- A dependency DAG across systems.
- Findings: bottlenecks, exceptions, deviations.
flyradar is stateless about long-lived knowledge. Each discovery is a job; results are emitted and optionally handed off to flycanon for retention.
flyquery is the counterpart for structured data the operator brings in as uploaded files:
- Multi-format ingestion (CSV/TSV/XLSX/XLS/ODS/JSON/JSONL/Parquet/Avro/ORC/ Arrow/Feather + compression variants), materialised as Parquet on object storage.
- A schema knowledge base with embeddings, samples, profiles, PII tags, and AI-curated descriptions.
- A relation graph (heuristic + AI-proposed joins).
- A semantic layer (MetricFlow-shape YAML metrics and dimensions).
- A Vanna-style example store of
(question, SQL)pairs. - A multi-agent pipeline (Grounding → Generation → DuckDB Executor → Critic → Explainer) for natural-language Text-to-SQL.
- Conversation drill-down memory: prior
executed_sql + table_qnames + snapshot_pinscarry across turns.
flyquery does not own documents, process graphs, or database connections.
+------------+ POST /agent/canon/handoff +------------+
| | ------------------------------> | |
| flyradar | POST /agent/query (RAG) | flycanon |
| | <------------------------------ | |
+-----+------+ +-----+------+
| |
| workspace events (EDA) |
| <------------------------------------------ |
| |
| +-----+------+
| POST /agent/query (NL→SQL) | |
+---------------------------------------->| flyquery |
| |
+------------+
All three services use an identical wire contract:
| Field | Meaning | Where |
|---|---|---|
tenant_id |
Customer tenant slug | DTO field + X-Tenant-Id header + DB column |
workspace_id |
Workspace slug within a tenant | DTO field + X-Workspace-Id header + DB column |
correlation_id |
Trace identifier | X-Correlation-Id header (autogenerated if absent) |
actor |
User or agent identity | Audit metadata; agent:<token-prefix> for agent calls |
agent_token |
Long-lived bearer token | X-Agent-Token header on /api/v1/agent/* |
No service-specific synonyms. No mapping layers. Same field name = same meaning everywhere.
All three services emit the same error shape:
{
"type": "https://firefly.dev/problems/<code>",
"title": "...",
"status": 404,
"code": "resource_not_found",
"detail": "...",
"errors": [{"path": "field", "message": "..."}]
}Branch on code (stable machine-readable), not title (human-readable).
Operator → POST /api/v1/workspaces (flycanon, workspace authority)
→ flycanon: insert canon_workspaces, emit canon.workspaces.v1 event
flyradar → consume canon.workspaces.v1 → WorkspaceCacheClient entry
flyquery → consume canon.workspaces.v1 → WorkspaceCacheClient entry
flycanon is the authoritative source of workspace identity. flyradar and flyquery are consumers; they cache workspace rows for RLS binding but do not have an independent workspace API.
Operator → POST /api/v1/agent/discovery-jobs (flyradar)
→ flyradar: run discovery, produce process graph + findings
→ POST /api/v1/agent/canon/handoff (flyradar → flycanon)
→ flycanon: ingest findings as knowledge items
→ flycanon: return source_id to flyradar
flyradar discovery stage → POST /api/v1/agent/query (flycanon)
→ flycanon: hybrid retrieval + RAG answer
→ flyradar: incorporate answer into discovery context
A flyradar discovery might need structured-data context — for example: "What is the order count in the dataset for the region flagged as bottleneck?"
flyradar discovery stage → POST /api/v1/agent/query (flyquery)
Request: {
"question": "Count of orders in dataset 'sales-2026' for region 'EMEA' in Q1",
"dataset_id": "...",
"X-Agent-Token": "<flyradar-flyquery-token>"
}
→ flyquery: grounding + SQL generation + DuckDB execution
→ flyquery: return AnswerResponse with row_count + result_preview
→ flyradar: incorporate numeric result into finding
This flow is forward-looking. In v0, flyradar does not ship a built-in
flyquery integration; the connection is opt-in via an agent token configured
in FLYRADAR_FLYQUERY_AGENT_TOKEN.
AI-curated schema descriptions from flyquery's DescribeAgent could be
promoted to flycanon knowledge items as authoritative data dictionaries. This
handoff is not implemented in v0 but follows the same
POST /api/v1/agent/sources pattern flyradar uses.
All three services use the same two-role Postgres pattern:
| Role | BYPASSRLS | Used for |
|---|---|---|
<service>_admin |
Yes | Migrations + cross-workspace workers |
<service>_app |
No | Request-scoped queries |
All three services apply the same GUC-based isolation:
SET LOCAL app.tenant_id = '<slug>';
SET LOCAL app.workspace_id = '<slug>';Per-table RLS policies on (tenant_id, workspace_id) ensure one service's
app role cannot read another workspace's data even in a shared Postgres cluster.
All three services use identical token infrastructure (lock-step modules):
- Format:
agt_<8hex>_<32hex> - Verification: constant-time SHA-256 compare
- Scopes: service-specific catalog (flycanon scopes ≠ flyradar scopes ≠ flyquery scopes; namespaced to avoid confusion)
- Rate limiting: per-token sliding 60-second window
- Workspace allowlist: enforced at verify time
Cross-service agent tokens (e.g., a flyradar token that can call flyquery) must be minted explicitly in flyquery with the appropriate flyquery scopes. There is no cross-service token federation in v0.
All three services inherit pyfly's observability stack:
- Health:
/actuator/health→{status: UP, components: {db, redis, ...}} - Metrics:
/actuator/metrics(Prometheus text format) - Tracing: OpenTelemetry if
<SERVICE>_OTEL_ENDPOINTis set - Logs: structured JSON to stdout;
<SERVICE>_LOG_LEVEL
Cross-service correlation: X-Correlation-Id is passed through agent-tier
calls, enabling distributed tracing across a flyradar → flyquery → flycanon
request chain.
| Service | Port | OpenAPI | SDK |
|---|---|---|---|
| flycanon | 8500 | flycanon/openapi.json |
sdks/python, sdks/java |
| flyradar | 8510 | flyradar/openapi.json |
sdks/python, sdks/java |
| flyquery | 8520 | flyquery/openapi.json |
sdks/python, sdks/java |
docs/architecture.mddocs/api-reference.mddocs/deployment.mddocs/eda-events.md
docs/architecture.mddocs/api-reference.mddocs/deployment.md
docs/architecture.md— system position, pipelines, hexagonal adaptersdocs/api-reference.md— full REST surface, scopes, error envelopedocs/ingestion.md— 10-stage pipeline + job kindsdocs/file-formats.md— per-format reader matrixdocs/semantic-layer.md— MetricFlow YAML + compilationdocs/payload-reference.md— DTO catalogdocs/security-model.md— RLS, token model, AST firewall, PIIdocs/deployment.md— ops runbook
Unified across 12 plans: conventions, workspace rename, signal rename, conventions adoption, agent surface, tenant lockdown (6 flyradar plans), conventions, multitenancy, embeddings hardening, conventions adoption, agent surface, tenant lockdown (6 flycanon plans). Closure batch: middleware fix, idempotency replay, OpenAPI snapshot gate, SDK propagation, cross-service handoff, Docker e2e harness.
4 plans from foundation to packaging: Postgres + RLS + object storage + ingestion pipeline + query pipeline + semantic layer + SDKs + full docs. flyquery joins the intelligence system as the third pillar.
See each service's CHANGELOG.md for the per-commit history.