Search Intelligence phase 2 — keyword gap, rank tracking, share of search

THIRD_PARTY_NOTICES.md

@@ -0,0 +1,60 @@
+# Third-party notices
+
+This repository contains, and is derived from, third-party open-source software.
+The original licence and copyright are preserved in full.
+
+## OpenSEO
+
+- **Project**: OpenSEO — "Open source alternative to Semrush and Ahrefs"
+- **Upstream repository**: https://github.com/every-app/open-seo
+- **Homepage**: https://openseo.so
+- **Licence**: MIT
+- **Copyright**: Copyright (c) 2026 Ben Senescu
+- **Integrated release**: `v0.1.3`
+- **Integrated commit**: `9d19e439905a9a954ccdefe22d9270d7c389695d`
+- **Integration date**: 2026-08-05
+
+The complete MIT licence text is retained unmodified in [`LICENSE`](./LICENSE)
+at the root of this repository, and applies to all substantial portions of
+OpenSEO reused here — which is to say, essentially the whole of this tree apart
+from the Morgana modifications listed below.
+
+### Modifications by CheckSig
+
+This fork is operated internally as the **Morgana Search Intelligence Engine**.
+Modifications are limited to the following, and are documented in detail in
+[`UPSTREAM.md`](./UPSTREAM.md):
+
+1. An isolated staging deployment configuration (`wrangler.staging.jsonc`) and a
+   one-line build hook allowing it to be selected (`vite.config.ts`).
+2. Three additive operational endpoints — `/healthz`, `/readyz`,
+   `/internal/status` — under `src/server/morgana/`, plus a small dispatch hook
+   in `src/server.ts`.
+3. A DataForSEO credential alias in `src/server/lib/runtime-env.ts`, so this
+   deployment uses its own dedicated credential rather than a shared one.
+4. Zero-spend guards, structured logging with redaction, a separate cost centre,
+   an MCP kill-switch, a scheduled-run refusal, and a resource isolation guard.
+5. Provenance and attribution documentation (this file, `UPSTREAM.md`,
+   `scripts/check-upstream.mjs`).
+
+No upstream file was renamed, restructured, rebranded or deleted. No upstream
+attribution was removed.
+
+### Trademarks
+
+"OpenSEO" and any associated logos or marks remain the property of their
+respective owners. CheckSig uses the name here solely to identify the upstream
+project from which this software is derived, as required by the attribution
+terms of the MIT licence. Nothing in this repository is intended to suggest
+sponsorship or endorsement by the OpenSEO project or its authors.
+
+## Dependencies
+
+This project's runtime and build dependencies carry their own licences,
+recorded in `pnpm-lock.yaml` and installed under `node_modules/`. The dependency
+tree is installed reproducibly with `pnpm install --frozen-lockfile`.
+
+Note that upstream's `pnpm-workspace.yaml` pins a number of `overrides` to raise
+transitive dependencies above known security advisories, and sets a
+`minimumReleaseAge` delay as a supply-chain control. Both are retained
+unmodified.

UPSTREAM.md

@@ -0,0 +1,137 @@
+# Upstream pin — OpenSEO
+
+This repository is a fork of an upstream open-source project, operated by CheckSig
+as the **Morgana Search Intelligence Engine**. It is not a rewrite and not a
+vendored copy: upstream history is intact and upstream remains the source of
+truth for everything except the small, enumerated patch series below.
+
+## Pin
+
+| Field                 | Value                                                     |
+| --------------------- | --------------------------------------------------------- |
+| `upstream_repository` | https://github.com/every-app/open-seo                     |
+| `upstream_release`    | `v0.1.3`                                                  |
+| `upstream_commit`     | `9d19e439905a9a954ccdefe22d9270d7c389695d`                |
+| `integration_date`    | 2026-08-05                                                |
+| `origin`              | https://github.com/checksig-custody/open-seo              |
+| `working_branch`      | `morgana/phase-0`                                         |
+| `licence`             | MIT (unchanged — see `LICENSE`, `THIRD_PARTY_NOTICES.md`) |
+
+### Why this commit
+
+`v0.1.3` was the latest release at integration time (published 2026-07-30) and
+`git compare v0.1.3...main` reported **identical** — zero commits ahead, zero
+behind. Pinning the release therefore costs nothing relative to `main` while
+giving a stable, human-announced reference point. All releases are `0.x`; there
+is no `1.0` line, and the cadence is roughly weekly.
+
+The pin is a **commit SHA**, not a tag: a tag can be moved, a SHA cannot.
+
+## Remotes
+
+```
+origin    https://github.com/checksig-custody/open-seo      (fetch + push)
+upstream  https://github.com/every-app/open-seo             (fetch)
+upstream  DISABLED_no_push_to_upstream                      (push)
+```
+
+The upstream push URL is deliberately set to an invalid value. Nothing in this
+repository should ever be pushed to the public upstream project, and a typo in a
+remote name should fail loudly rather than open a pull request by accident.
+
+`origin/main` is kept as a clean mirror of upstream for diffing. Morgana work
+happens on `morgana/phase-0` and later branches.
+
+## Verification
+
+Confirm at any time that the tree still descends from the pinned commit and that
+the patch series is the only divergence:
+
+```bash
+git merge-base --is-ancestor 9d19e439905a9a954ccdefe22d9270d7c389695d HEAD && echo "pin intact"
+git diff --stat 9d19e439905a9a954ccdefe22d9270d7c389695d..HEAD
+node scripts/check-upstream.mjs        # read-only: reports distance from upstream
+node scripts/assert-isolation.mjs wrangler.staging.jsonc
+```
+
+## Local patch series
+
+The patch series is kept deliberately small, isolated and reviewable. No
+upstream file is renamed, no directory is restructured, no component is
+rebranded, and no upstream code is deleted. Capabilities are disabled through
+configuration and guards, never by removing the code that implements them —
+deletion would guarantee a conflict on every future upstream merge.
+
+| #      | Files                                                                                                                                                                                    | Purpose                                                                                                                                                                                                                                               |
+| ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **P1** | `wrangler.staging.jsonc` (new), `vite.config.ts` (1 line)                                                                                                                                | Isolated staging deploy config. The `vite.config.ts` change adds an env-driven `configPath` to the Cloudflare plugin; unset everywhere else, so upstream behaviour is byte-identical.                                                                 |
+| **P2** | `src/server/morgana/phase0-routes.ts` (new), `src/server.ts` (+8 lines)                                                                                                                  | `/healthz`, `/readyz`, `/internal/status`. Handled before upstream dispatch so a probe performs no other work. Returns `null` for every other path.                                                                                                   |
+| **P3** | `src/server/lib/runtime-env.ts` (+alias map)                                                                                                                                             | Reads `DATAFORSEO_SEARCH_INTELLIGENCE_API_KEY` in preference to `DATAFORSEO_API_KEY`, so this engine can never use Morgana Brand Monitoring's DataForSEO credential or balance. Falls back to the upstream name, so unpatched behaviour is preserved. |
+| **P4** | `src/server/morgana/phase0-{env,guard,cost,logging}.ts` (new), `src/server/morgana/phase0.test.ts` (new), `src/server.ts` (MCP + scheduled guards), `scripts/assert-isolation.mjs` (new) | Zero-spend enforcement, structured logging with redaction, cost centre, MCP kill-switch, scheduled-run refusal, resource isolation guard.                                                                                                             |
+| **P5** | `UPSTREAM.md`, `THIRD_PARTY_NOTICES.md`, `scripts/check-upstream.mjs` (new)                                                                                                              | Provenance, attribution, upstream tracking.                                                                                                                                                                                                           |
+
+Everything Morgana-specific lives under `src/server/morgana/` or in a file that
+did not exist upstream, except for four small edits to three upstream files
+(`src/server.ts`, `src/server/lib/runtime-env.ts`, `vite.config.ts`), each
+marked in place with a `MORGANA LOCAL PATCH` comment referencing this document.
+
+### Deliberate non-changes
+
+- **Alchemy is not used or removed.** Upstream deploys via `alchemy.run.ts`
+  (pinned prerelease `2.0.0-beta.61`, own state store, `--adopt` used elsewhere
+  in the repo, auto-provisions Cloudflare Access). On an account that also holds
+  Morgana production, an IaC tool that can adopt existing resources is the wrong
+  risk. We deploy with a hand-authored wrangler config instead and leave the
+  Alchemy files untouched so upstream merges stay clean.
+- **Upstream CI is untouched.** The GitHub token available to this integration
+  lacks the `workflow` scope, so `.github/workflows/**` cannot be modified. See
+  the Morgana repo's `docs/search-intelligence/OPERATIONS.md`.
+- **Branding is unchanged.** The only Morgana identification is the staging
+  marker string in the status payload.
+
+## Update procedure
+
+Upstream is **never** merged automatically, and the pin is never advanced by a
+script. The procedure is:
+
+```bash
+node scripts/check-upstream.mjs          # 1. how far behind are we?
+git fetch upstream --tags                # 2. fetch, do not merge
+git checkout -b morgana/upstream-<ver>   # 3. dedicated branch
+git merge <new-tag>                      # 4. controlled merge, resolve by hand
+pnpm install --frozen-lockfile           # 5. reproducible install
+pnpm run ci:check && pnpm run test:ci    # 6. full gate
+# 7. security review: re-run the checklist in the Morgana repo's SECURITY.md,
+#    paying attention to new external calls, new env vars, new telemetry,
+#    changes to auth, and anything that could spend money.
+node scripts/assert-isolation.mjs wrangler.staging.jsonc
+# 8. deploy to staging and verify /healthz, /readyz, /internal/status
+# 9. record the new pin in this file, then release.
+```
+
+Step 7 is not optional. Upstream is a fast-moving commercial product with
+billing, telemetry and AI paths that Phase 0 deliberately disables; a routine
+version bump can re-enable any of them.
+
+### What to re-check on every upgrade
+
+- Does `wrangler.staging.jsonc` still declare every Durable Object and Workflow
+  class the Worker exports? A new class breaks the upload.
+- Did upstream add a cron trigger, a new external host, or new telemetry?
+- Did `AUTH_MODE` handling change?
+- Did the DataForSEO credential resolution move away from `getEnvValueSync`?
+  Patch P3 depends on that single chokepoint.
+- Do the migrations still apply cleanly to a **fresh** D1?
+
+## Rollback
+
+Rolling back the engine never deletes data. See
+`docs/search-intelligence/OPERATIONS.md` in the Morgana repository for the full
+procedure; in short: redeploy the previous Worker version, leave D1 and R2
+intact, and leave Morgana production untouched.
+
+## Known conflicts
+
+None at `v0.1.3`. The four edited upstream files are the expected conflict
+surface on a future merge; each edit is small and marked with a
+`MORGANA LOCAL PATCH` comment so a conflict is easy to resolve deliberately.

drizzle-pg/0014_search_intelligence.sql

@@ -0,0 +1,132 @@
+CREATE TABLE "domain_refresh_jobs" (
+	"id" text PRIMARY KEY NOT NULL,
+	"entity_id" text NOT NULL,
+	"status" text NOT NULL,
+	"trigger" text NOT NULL,
+	"requested_by" text,
+	"dedupe_key" text NOT NULL,
+	"attempts" integer DEFAULT 0 NOT NULL,
+	"last_error" text,
+	"skip_reason" text,
+	"estimated_cost_micros" integer DEFAULT 0 NOT NULL,
+	"actual_cost_micros" integer DEFAULT 0 NOT NULL,
+	"snapshot_id" text,
+	"created_at" text DEFAULT to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') NOT NULL,
+	"started_at" text,
+	"finished_at" text
+);
+--> statement-breakpoint
+CREATE TABLE "domain_snapshot_keywords" (
+	"id" text PRIMARY KEY NOT NULL,
+	"snapshot_id" text NOT NULL,
+	"keyword" text NOT NULL,
+	"rank_group" integer,
+	"rank_absolute" integer,
+	"search_volume" integer,
+	"estimated_traffic" real,
+	"cpc" real,
+	"keyword_difficulty" integer,
+	"search_intent" text,
+	"ranking_url" text,
+	"serp_updated_at" text,
+	"position" integer NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "domain_snapshot_pages" (
+	"id" text PRIMARY KEY NOT NULL,
+	"snapshot_id" text NOT NULL,
+	"url" text NOT NULL,
+	"normalized_url" text NOT NULL,
+	"estimated_traffic" real,
+	"keyword_count" integer,
+	"top_keyword" text,
+	"top_keyword_position" integer,
+	"page_title" text,
+	"last_seen_at" text,
+	"position" integer NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "domain_snapshots" (
+	"id" text PRIMARY KEY NOT NULL,
+	"entity_id" text NOT NULL,
+	"organic_traffic_estimate" real,
+	"organic_keyword_count" integer,
+	"backlink_count" integer,
+	"referring_domain_count" integer,
+	"rank_signal" integer,
+	"location_code" integer NOT NULL,
+	"language_code" text NOT NULL,
+	"source" text NOT NULL,
+	"provider_request_id" text,
+	"fetched_at" text NOT NULL,
+	"snapshot_date" text NOT NULL,
+	"estimated_cost_micros" integer DEFAULT 0 NOT NULL,
+	"actual_cost_micros" integer DEFAULT 0 NOT NULL,
+	"dedupe_key" text NOT NULL,
+	"created_at" text DEFAULT to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "search_budget_state" (
+	"month" text PRIMARY KEY NOT NULL,
+	"monthly_cost_micros" integer DEFAULT 0 NOT NULL,
+	"current_day" text,
+	"daily_cost_micros" integer DEFAULT 0 NOT NULL,
+	"consecutive_failures" integer DEFAULT 0 NOT NULL,
+	"circuit_opened_at" text,
+	"last_alert_threshold" integer,
+	"updated_at" text DEFAULT to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "search_entities" (
+	"id" text PRIMARY KEY NOT NULL,
+	"display_name" text NOT NULL,
+	"canonical_domain" text NOT NULL,
+	"normalized_domain" text NOT NULL,
+	"entity_type" text NOT NULL,
+	"enabled" boolean DEFAULT true NOT NULL,
+	"priority" text DEFAULT 'normal' NOT NULL,
+	"include_subdomains" boolean DEFAULT false NOT NULL,
+	"location_code" integer DEFAULT 2380 NOT NULL,
+	"language_code" text DEFAULT 'it' NOT NULL,
+	"refresh_interval_hours" integer DEFAULT 24 NOT NULL,
+	"backlink_interval_hours" integer DEFAULT 168 NOT NULL,
+	"last_refreshed_at" text,
+	"last_backlink_refreshed_at" text,
+	"created_at" text DEFAULT to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') NOT NULL,
+	"updated_at" text DEFAULT to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') NOT NULL,
+	"disabled_at" text
+);
+--> statement-breakpoint
+CREATE TABLE "search_usage_ledger" (
+	"id" text PRIMARY KEY NOT NULL,
+	"day" text NOT NULL,
+	"entity_id" text,
+	"endpoint_path" text NOT NULL,
+	"metering_class" text NOT NULL,
+	"requests" integer DEFAULT 0 NOT NULL,
+	"metered_requests" integer DEFAULT 0 NOT NULL,
+	"failed_requests" integer DEFAULT 0 NOT NULL,
+	"retry_requests" integer DEFAULT 0 NOT NULL,
+	"estimated_cost_micros" integer DEFAULT 0 NOT NULL,
+	"actual_cost_micros" integer DEFAULT 0 NOT NULL,
+	"cache_hits" integer DEFAULT 0 NOT NULL,
+	"cache_misses" integer DEFAULT 0 NOT NULL,
+	"blocked_by_budget" integer DEFAULT 0 NOT NULL,
+	"updated_at" text DEFAULT to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') NOT NULL
+);
+--> statement-breakpoint
+ALTER TABLE "domain_refresh_jobs" ADD CONSTRAINT "domain_refresh_jobs_entity_id_search_entities_id_fk" FOREIGN KEY ("entity_id") REFERENCES "public"."search_entities"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "domain_snapshot_keywords" ADD CONSTRAINT "domain_snapshot_keywords_snapshot_id_domain_snapshots_id_fk" FOREIGN KEY ("snapshot_id") REFERENCES "public"."domain_snapshots"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "domain_snapshot_pages" ADD CONSTRAINT "domain_snapshot_pages_snapshot_id_domain_snapshots_id_fk" FOREIGN KEY ("snapshot_id") REFERENCES "public"."domain_snapshots"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "domain_snapshots" ADD CONSTRAINT "domain_snapshots_entity_id_search_entities_id_fk" FOREIGN KEY ("entity_id") REFERENCES "public"."search_entities"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+CREATE UNIQUE INDEX "domain_refresh_jobs_dedupe_idx" ON "domain_refresh_jobs" USING btree ("dedupe_key");--> statement-breakpoint
+CREATE INDEX "domain_refresh_jobs_status_idx" ON "domain_refresh_jobs" USING btree ("status","created_at");--> statement-breakpoint
+CREATE INDEX "domain_refresh_jobs_entity_idx" ON "domain_refresh_jobs" USING btree ("entity_id","created_at");--> statement-breakpoint
+CREATE INDEX "domain_snapshot_keywords_snapshot_idx" ON "domain_snapshot_keywords" USING btree ("snapshot_id","position");--> statement-breakpoint
+CREATE INDEX "domain_snapshot_pages_snapshot_idx" ON "domain_snapshot_pages" USING btree ("snapshot_id","position");--> statement-breakpoint
+CREATE UNIQUE INDEX "domain_snapshots_dedupe_idx" ON "domain_snapshots" USING btree ("dedupe_key");--> statement-breakpoint
+CREATE INDEX "domain_snapshots_entity_date_idx" ON "domain_snapshots" USING btree ("entity_id","snapshot_date");--> statement-breakpoint
+CREATE UNIQUE INDEX "search_entities_domain_market_idx" ON "search_entities" USING btree ("normalized_domain","location_code","language_code");--> statement-breakpoint
+CREATE INDEX "search_entities_enabled_idx" ON "search_entities" USING btree ("enabled","entity_type");--> statement-breakpoint
+CREATE UNIQUE INDEX "search_usage_ledger_day_endpoint_idx" ON "search_usage_ledger" USING btree ("day","endpoint_path","metering_class");--> statement-breakpoint
+CREATE INDEX "search_usage_ledger_day_idx" ON "search_usage_ledger" USING btree ("day");