close review gaps, fix schema-DDL splitter, improve test coverage

CHANGELOG.md

@@ -1,5 +1,60 @@
 # Changelog
 
+## v0.14.0 — community PR hardening: NEO4J_DATABASE, NAMS cypher runtime, --ontology-file (unreleased)
+
+Integrates five community PRs merged since v0.13.1 (#52, #56, #58, #59, #60), closes the gaps found while reviewing them, and adds test coverage at every level: +88 fast-suite tests, +3 Neo4j integration tests, a new bolt combo in the slow generated-suite runner, and 2 new tests inside every generated project (+122 collected overall — fast suite now 1,437 passing / 1,697 collected). The new integration coverage immediately exposed — and this release fixes — a long-standing schema-DDL splitter bug that had been silently dropping five indexes/constraints from every seeded database.
+
+### Features (community PRs)
+
+- **`NEO4J_DATABASE` support end-to-end** (#60, @henrardo). New `--neo4j-database` flag / `NEO4J_DATABASE` env var for the self-hosted backend, threaded through the CLI, wizard (new prompt on manual credential entry), Aura `.env` import (previously read and silently discarded), generated `.env`, the generated app's `Settings`, memory-layer `MemorySettings`, and the raw-driver session in `execute_cypher()`. Blank defers to the driver default (`neo4j`); set it for Aura instances provisioned via the Aura API/CLI, which commonly name the database after the instance id.
+- **Memory-layer failures now surface in `/health`** (#60, @henrardo). `store_message()` failures are recorded into the same classified error state used at startup connect time; the bolt `/health` response gains `memory`, `memory_error`, and `memory_error_detail` fields, and the startup lifecycle checks the memory client instead of assuming success once the raw driver connects. Previously a wrong database name meant the app reported "ok" while every memory write failed with only a log line.
+- **`execute_cypher()` works on NAMS** (#56, @benmyrgorod). New `_execute_nams_cypher()` routes through `client.query.cypher` with result-shape coercion (`_coerce_nams_records`), so domain agent tools and `POST /cypher` execute read queries on the hosted backend instead of dying on the never-connected bolt driver. `_require_neo4j()` now returns 503 when the NAMS client is missing, for every route.
+- **`--ontology-file` wired up** (#58, @irene221b). The flag was documented (and tracked as #50) but had no code path. Scaffolds directly from a hand-written domain YAML — no LLM call — using the domain id declared in the file.
+- **Custom domains resolve by id** (#52, @ecsricktorzynski, issue #30). `load_domain()` now searches `~/.create-context-graph/custom-domains/` (with a declared-`domain.id` fallback for renamed files), matching what `list_available_domains()` advertises. Bundled domains shadow same-id custom files.
+- **`demo_scenarios` optional in practice** (#59, @irene221b). The generated Playwright spec indexed `demo_scenarios[0].prompts[0]` unconditionally, crashing scaffold generation for any domain that omits scenarios. Falls back to a generic prompt.
+
+### Bug Fixes (gaps found reviewing the PRs)
+
+- **`--ontology-file` scaffolds now include `data/ontology.yaml`.** The renderer only wrote the ontology copy for `--custom-domain` (YAML string) or bundled domains (copy by id); hand-written ontologies matched neither branch, silently producing a scaffold without the file the docs promise. The raw YAML is now carried into the scaffold verbatim, and the flag also works when the wizard collects the remaining settings. `--ontology-file` + `--custom-domain` is now an explicit error instead of a silent precedence pick. (`cli.py`)
+- **`NEO4J_DATABASE` reaches scaffold-time ingest and the generated import script.** #60 threaded the database through the generated app but not through `ingest.py` (`_ingest_with_memory_client`, `_ingest_with_driver`, `reset_neo4j`) or the scaffolded `import_data.py` bolt path — so `--demo` / `--ingest` / `--reset-database` / `make import` targeted the *default* database while the app read from the configured one, producing an empty graph on non-default-database instances. All bolt sessions now honor the setting; `validate_connection()` gains an optional `database` parameter. (`ingest.py`, `neo4j_validator.py`, `templates/backend/connectors/import_data.py.j2`)
+- **`.env.example` documents `NEO4J_DATABASE`.** #60 updated `dot_env.j2` only; the example file now carries the same commented block. (`templates/base/dot_env_example.j2`)
+- **Playwright spec survives a scenario with an empty `prompts` list.** #59's guard handled a missing `demo_scenarios` but still crashed rendering when the first scenario had `prompts: []` (legal per the Pydantic model). (`templates/frontend/e2e/app.spec.ts.j2`)
+- **Schema DDL splitter no longer eats statements or executes comment tails.** Every consumer of `generate_cypher_schema()` output split on `;` and skipped fragments starting with `//` — which (a) silently dropped the 5 real statements that sit behind comment headers (`person_name`, `document_title`, `document_domain`, `document_name_unique` and the `local_file_fulltext` index were never created by `make seed`/ingest), and (b) executed the tail of the "dimensions must match your embed model" comment as Cypher, raising `CypherSyntaxError` on every schema apply. New shared `split_cypher_statements()` strips comment lines before splitting; used by `ingest.py` (both bolt paths), the generated `generate_data.py`, and the integration suite. (`ontology.py`, `ingest.py`, `templates/backend/shared/generate_data.py.j2`)
+- **Test suite is hermetic against `~/.create-context-graph/custom-domains/`.** With #52, `load_domain()` joins `list_available_domains()` in scanning the user-local directory — so a contributor's saved custom domains leaked into every domain-iterating test (the "football-intelligence" failures reported while developing #60). An autouse conftest fixture isolates the path; tests that need custom domains patch it explicitly. (`tests/conftest.py`)
+
+### Live NAMS verification (and the bugs it caught)
+
+The full flow — scaffold → ingest → boot → API — was exercised against the production NAMS service (`memory.neo4jlabs.com`) with a real API key and `neo4j-agent-memory` 0.5.0 (the version fresh scaffolds install). That surfaced five breaks the mocked suites couldn't see, all fixed and now pinned by tests; 19/19 live API checks pass afterward:
+
+- **Conversation memory silently failed on every message.** The NAMS service only accepts messages addressed to conversation ids *it* minted at create time — client-chosen session ids 404 with "conversation not found," and `MemoryIntegration` (0.5.x) both posts the client id straight through and swallows the failure into an `{"error": ...}` return value. The generated `memory.py` now creates the conversation on first use per session and addresses the server-assigned id (`_resolve_nams_conversation()`, process-local cache — a restart starts a fresh conversation instead of erroring), and `store_message()` treats `{"error": ...}` returns as failures so they surface in `/health` rather than reading as success.
+- **Every document/body ingest write failed** (`role must be user, assistant, or system` + the same conversation-404). All three NAMS ingest implementations (`run_nams_ingest`, the scaffolded `import_data.py`, and `make seed`'s `ingest_fixtures_nams`) now create their message channels via `create_conversation` and send `role="user"` with a `metadata.kind` marker (`"document"` / `"entity-body"`). Before: 0/25 documents ingested; after: clean.
+- **`/api/documents` and `/api/schema/visualization` returned nothing.** The live service coerces unknown `entity_type` values (`OBJECT`, `EVENT`) to `custom`, so the server-side OBJECT filter matched nothing — and it rejects empty search queries (`query is required`), so the schema view's enumerate-everything search 400'd. Both adapters are now cypher-first (using the scaffold's own `_pole_type: OBJECT_` description marker for documents, and a type-count aggregation for the schema view), with the search-based flows kept as fallbacks.
+- **Graph expand / entity connections were dead.** `long_term.get_entity(id)` doesn't exist in `neo4j-agent-memory` 0.5.x; `expand_node_nams` and `get_entity_detail_nams` now resolve id-addressed lookups and neighbor edges through the cypher API (which also surfaces the server-created `SAME_AS` resolution edges), keeping the old REST flow as a fallback.
+- **NAMS reset never deleted anything.** `long_term.delete_entity` doesn't exist in 0.5.x (the old code swallowed the `AttributeError` and reported "0 entities removed"), the REST API has no delete endpoint, and the cypher API is read-only. `--reset-database` and the scaffold's `make reset` now say so honestly (with the current entity count) and point at the NAMS dashboard, instead of pretending. Docs updated to match.
+
+Verified unchanged live: `client.query.cypher` read queries (the PR #56 dispatch) work and write queries are rejected; the reasoning trace ingest (`start_trace`/`add_step`/`complete_trace`) succeeds; `add_relationship`/`add_fact`/`add_preference` exist in 0.5.0 but raise `NotSupportedError` against NAMS, so the `ccg-edges` encoding remains the correct design; `reasoning.list_traces` is NotSupported client-side, so `/api/traces` degrades to an empty list on NAMS. Upstream issues worth filing against `neo4j-agent-memory`: `MemoryIntegration` should create conversations (or the service should honor client ids), `add_message` shouldn't silently target nonexistent conversations, and `add_entity` throws a client-side validation error when the server responds with a dedup/merge result.
+
+### Testing
+
+- **`tests/test_generated_client_runtime.py`** (new, 31 tests) — executes the rendered `context_graph_client.py` and `memory.py` against doubles: NAMS dispatch + result coercion + tool-event collection + not-connected error; bolt session `database=` threading; `MemorySettings` database pass-through; `store_message()` error recording/clearing/`NotSupportedError` handling; the `_classify_memory_error` buckets `/health` reports; the NAMS conversation-id translation (created once per session, server id targeted, bolt untouched, create-failure fallback) and the swallowed-`{"error"}` failure path.
+- **`test_routes_integration.py`** (+6) — mounts the generated FastAPI app: `POST /cypher` on NAMS dispatches through `execute_cypher` (and maps errors to 400), API routes 503 when the NAMS client is missing while `/health` reports degraded, bolt `/cypher` still injects the `$domain` parameter, and a live `store_message` failure flips `/health` to degraded with classified fields — then a successful write clears it Three more pin the cypher-first adapters: documents enumerated by description marker (search fallback for custom-typed entities), schema visualization aggregated via cypher.
+- **`test_cli.py`** (+12) — `TestNeo4jDatabaseFlag` (flag→`.env`, blank default, Aura import, flag-beats-file precedence, dry-run display) and `TestOntologyFileFlag` (scaffold, `data/ontology.yaml` copy, static demo data, invalid YAML exit 1, missing file exit 2, `--custom-domain` conflict, auto-slug).
+- **`test_wizard.py`** (+5) — `_parse_aura_env` four-tuple contract: database read/absent/quoted, missing URI/password aborts.
+- **`test_generated_project.py`** (+17) — template pins for the database threading (config/client/memory/import script/`.env`/`.env.example`, NAMS `.env` exclusion), memory-error surfacing in `main.py`, the NAMS cypher branch, and scenario-fallback rendering (none / empty-prompts / real prompts).
+- **`test_ontology.py`** (+9) — `split_cypher_statements` unit tests (semicolon-in-comment, comment-header recovery, commented-out DDL dropped, all-domains executable-statement sweep, proof the old pattern dropped `person_name`) and custom-domain isolation meta-tests.
+- **`test_custom_domain.py`** (+3) — resolution precedence: bundled shadows same-id custom, corrupt custom YAML skipped during the id scan, underscore files ignored.
+- **`test_bolt_ingest_parity.py`** (+1) — the scaffolded bolt import session must target `settings.neo4j_database`.
+- **`test_integration.py`** (+3, `--integration`) — explicit-database ingest lands data (via `ProjectConfig.neo4j_database`), `validate_connection` accepts a database name and rejects an unknown one. `TestSchemaCreation` now applies DDL through the shared splitter and asserts the previously-skipped indexes exist.
+- **`test_generated_tests.py`** — the slow generated-suite runner now also scaffolds one **bolt** project (5 combos), and asserts the new backend-specific health tests actually ran. The generated `test_routes.py` gains two tests per backend: NAMS degraded-client reporting, bolt live-write-failure surfacing.
+- **`scripts/e2e_smoke_test.py`** — asserts the `/health` contract shape per backend on startup (bolt: `neo4j` + `memory` fields, logging a warning with the classified error when degraded; NAMS: `nams` field).
+
+### Docs
+
+- `reference/cli-options.md` — `--neo4j-database` and `--ontology-file` rows, `NEO4J_DATABASE` in the env-var table.
+- `how-to/use-neo4j-aura.md` — caution block on Aura API/CLI-provisioned database names and how the failure presents in `/health`.
+- `how-to/add-custom-domain.md` — `--ontology-file` semantics: domain id from the file, `data/ontology.yaml` copy, mutual exclusion with `--custom-domain`.
+- `reference/generated-project-structure.md` — `.env` listing includes `NEO4J_DATABASE`.
+
 ## v0.13.1 — v0.13.0 feedback report triage (unreleased)
 
 Addresses the May 20, 2026 v0.13.0 feedback report. The report mixed verified issues with claims that don't match the current codebase; each claim was verified before scoping work. This release closes every real issue, makes the generated `app.models` module load-bearing, and adds regression tests so the v0.12.0/v0.13.0 fixes can't silently come back.