GAPS.md

Stayquery Production Readiness Gaps

Reviewed on 2026-06-23. This register summarizes missing or insufficient implementations needed to move Stayquery from MVP toward a production-ready application. It is based on the current repository state and explicit follow-up plans in PROGRESS.md.

Response Quality Evaluation

• Expand objective regression gates for stayquery eval; the optional quality rubric now has a fail-under threshold, but default CI still uses offline deterministic checks.
• Calibrate the new answer-quality judge workflow against a curated private Novigrad benchmark before treating rubric scores as production release gates.
• Persist evaluation results separately from ad hoc query runs so quality trends can be compared across commits, datasets, model versions, prompts, and retrieval changes.
• Add live or contract tests for OpenAI planning and answer generation so prompt/schema changes are validated beyond monkeypatched unit tests.
• Expand evaluation fixtures to cover ambiguity, unsupported questions, multilingual source text, thin evidence, conflicting evidence, citation quality, and hotel-versus-restaurant intent.
• Define acceptance criteria for hallucination resistance, citation faithfulness, uncertainty wording, and source-grounded refusal behavior.
• Add a repeatable review process for manually inspecting failed or low-confidence answers and turning them into regression cases.

Retrieval Design

• Add deterministic ordering to SQL retrieval so equal or weak matches produce stable results and stable evaluation output.
• Define and test the intended SQL relevance model for metadata filters, ILIKE text matches, extracted query terms, and amenities/dietary/time constraints.
• Improve SQL retrieval scalability for %term% matching over document text and JSON metadata, or document why the current approach is acceptable for the target corpus size.
• Define whether Qdrant payloads missing content_hash are acceptable legacy hits or stale hits, then add tests for that behavior.
• Add an explicit Qdrant rebuild or reconciliation command that can repair the derived vector index from authoritative Postgres documents.
• Delete or reconcile Qdrant points when Postgres documents are removed or replaced during ingest.
• Add recovery behavior for partial ingest failures where Postgres writes succeed but Qdrant indexing fails.
• Add live or local integration tests for Qdrant collection creation, indexing, querying, rehydration, stale-hit skipping, and fallback behavior.
• Split orchestration into narrower planning, retrieval, answer-generation, and persistence phases so retrieval behavior can be tested and evolved without broad service coupling.

Token, Resource, And Latency Management

• Add request-level budgets for OpenAI-backed operations, including maximum query length, maximum retrieved context size, maximum answer tokens, and per-request cost expectations.
• Add rate limiting or equivalent abuse controls for API calls that can trigger OpenAI usage and query-run persistence.
• Add full ask-flow timeout handling in addition to the current OpenAI SDK timeout setting.
• Move external OpenAI and Qdrant calls out of long-lived database transaction scopes, or define partial-run persistence semantics for failures.
• Add structured failure handling for OpenAI planning failures, embedding failures, answer-generation failures, and Qdrant retrieval failures.
• Decide whether Qdrant failures should degrade gracefully to SQL-only evidence, fail the request, or return a typed partial-result warning.
• Track token usage and latency by operation over time, not only in individual query responses.
• Add configurable concurrency, backpressure, or worker limits for deployed API use.
• Define cost budgets and ownership for OpenAI usage, including alert thresholds and rotation or shutdown procedures if budgets are exceeded.

Deployment Readiness And Monitoring Systems

• Define the production runtime target, such as container, VM, managed platform service, or local-only tool, and add the corresponding deployment artifact.
• Add authentication, authorization, or documented deployment-level access controls for POST /ask.
• Add production-safe networking assumptions, TLS termination, CORS policy, and API exposure boundaries.
• Define production secret storage, rotation ownership, and incident response for OpenAI, Postgres, and Qdrant credentials.
• Define production Postgres and Qdrant provisioning, backups, restore testing, and data retention.
• Add structured application logging for CLI and API request paths, including correlation IDs for query runs.
• Add metrics for request counts, failures, latency, OpenAI calls, token usage, Qdrant failures, stale-hit skips, and hybrid fallback reasons.
• Add tracing across planning, SQL retrieval, vector retrieval, hybrid fusion, answer generation, and persistence.
• Add dashboards, uptime checks, and alerting for service health, dependency health, latency, failure rate, and budget usage.
• Add CI/CD expectations for tests, static checks, dependency locking, vulnerability scanning, migrations, deployment, and rollback.
• Define migration rollback procedures and operator responsibilities before schema changes are applied in production.
• Define retention, minimization, and redaction policies for user queries, generated answers, raw source JSON, query telemetry, and Qdrant payload metadata.

Cross-Cutting Production Gaps

• Add static check support or remove stale Ruff configuration, then document the supported command.
• Add dependency and supply-chain review, including whether production installs must use uv.lock.
• Add API validation coverage for empty, oversized, malformed, unauthenticated, unauthorized, and rate-limited requests after those behaviors are implemented.
• Add security tests for error mapping so API responses do not leak sensitive internals.
• Define ownership for roadmap items in PROGRESS.md, including priority, target milestone, and whether each item blocks production exposure.