Skip to content

feat: IndexNow + Bing Webmaster telemetry, deploy webhook, MCP authz/SSRF hardening, War Room dashboard - #197

Open
ramonmnavarro-byte wants to merge 11 commits into
every-app:mainfrom
ramonmnavarro-byte:main
Open

feat: IndexNow + Bing Webmaster telemetry, deploy webhook, MCP authz/SSRF hardening, War Room dashboard#197
ramonmnavarro-byte wants to merge 11 commits into
every-app:mainfrom
ramonmnavarro-byte:main

Conversation

@ramonmnavarro-byte

@ramonmnavarro-byte ramonmnavarro-byte commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Adds the Indexing & Bing Visibility feature set (Phases 1–4 of the roadmap) plus security hardening and the War Room dashboard.

What's included

Phase 1 — IndexNow submission

  • IndexNowService — submit URLs to api.indexnow.org, batch (max 100/req), retry 429/network with backoff, ledger every attempt
  • New tables (both dialects): indexnow_configs, indexing_events
  • New MCP tools: submit_indexnow_urls, get_indexing_queue
  • Project route /p/:projectId/indexing + sidebar entry

Phase 2 — Bing Webmaster OAuth + read-only telemetry

  • bing_connections table (both dialects, mirrors gsc_connections)
  • Bing OAuth via Better Auth (bing-webmaster provider) + self-hosted API-key fallback
  • BingService + bingClient (crawl issues, visibility)
  • New MCP tools: get_bing_visibility, get_bing_crawl_issues

Phase 3 — War Room telemetry + MCP docs

  • getWarRoomIndexingTelemetry + getWarRoomBingTelemetry server functions (authenticated, read-only, org-scoped) + WarRoomTelemetry.tsx UI
  • web/content/docs/mcp.md documents all 4 new MCP tools

Phase 4 — IndexNow deploy webhook (new)

  • POST /api/webhooks/indexnow — callable by external systems (Vercel deploy hook / CI) without a browser session
  • Shared-secret auth (X-IndexNow-Webhook-Secret or Bearer) + exact INDEXNOW_WEBHOOK_HOSTS allowlist
  • URL sources: payload urls/changedUrls/url, or sitemap+homepage discovery fallback
  • HTTPS/public-host validation, SSRF guards, 64 KiB / 2,000 URL / 500-per-project caps, 5-minute dedupe via ledger (idempotent)
  • docs/INDEXNOW_WEBHOOKS.md + env examples

MCP tool authz / SSRF hardening

  • Tools wrapped with withMcpProjectAuth (project-scoped authorization); SSRF protections on outbound fetches

War Room dashboard

  • /war-room route + nav entry

Security floors

  • fast-uri, undici, ip-address, brace-expansion, js-yaml bumped to clear advisories

Verification

  • 801 tests pass (incl. schema parity 132)
  • db:generate clean on both d1 + pg dialects
  • Our files typecheck + lint clean (pre-existing upstream alchemy.run.ts errors untouched)

Note

Fork PR (upstream denies direct push).

Ramon Navarro and others added 9 commits August 7, 2026 09:45
Ramon Navarro @claude Ramon Navarro
fix(deps): floor brace-expansion at >=5.0.7 to clear ReDoS advisory
d00273b 
GHSA-3jxr-9vmj-r5cp (High): exponential-time expansion of consecutive
non-expanding {} groups. Transitive via the glob/minimatch family; pnpm
override bumps the single resolved copy 5.0.6 -> 5.0.7.

pnpm audit --prod is back to 0 High/Critical (1 low + 1 ignored moderate
remain, pre-existing).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ramonmnavarro-byte Ramon Navarro
feat(chat): scope MiniMax provider routing to the default model so OP…
4235b00 
…ENROUTER_MODEL overrides route cleanly
Ramon Navarro @claude Ramon Navarro
fix: reject low-relevance keyword expansions in research_keywords
08e0175 
DataForSEO Labs related/ideas sources return raw candidate rows with no
relevance signal; the research service accepted every non-seed row into
coverage regardless of topical overlap with the seed. City-modified niche
seeds (e.g. "AI receptionist Miami") degraded to token-level matching,
returning unrelated high-volume keywords sharing only the location token
(school names, USCIS, NFL content). usedFallback only reflected which
source succeeded, not whether results were trustworthy.

Added shared meaningful-token overlap filtering across all Labs sources
before selection; expansions with no relevant candidates now collapse to
the existing "not enough keyword data" empty response instead of silently
substituting noise. Cache version bumped to invalidate previously-cached
polluted results.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ramon Navarro Ramon Navarro
feat: add War Room dashboard page and nav entry
e00018c 
Shared workboard view reading clients/seo-warroom.md, wired into nav/route tree.
Ramon Navarro @claude Ramon Navarro
chore: gitignore local D1 sqlite backups
2cc39a5 
.wrangler-d1-backup-*.sqlite sat untracked and un-gitignored in the
repo root — a broad git add would have committed a database backup
with real project/tenant data.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ramon Navarro @claude Ramon Navarro
fix: bump fast-uri, undici, ip-address, brace-expansion, js-yaml secu…
aa63103 
…rity floors

Raises pnpm.overrides in pnpm-workspace.yaml to patched versions and
removes the now-dead package.json#pnpm.overrides field pnpm was
silently ignoring (see existing comment on the workspace overrides
block). fast-uri and js-yaml are temporarily added to
minimumReleaseAgeExclude since their patched releases (2026-07-31)
haven't cleared the 8-day supply-chain window yet; both entries are
dated for removal on 2026-08-08.

Resolves 5 High pnpm audit findings: GHSA-7p8r-x3mc-p8w7 (fast-uri),
GHSA-4cwx-7wf7-3272 (undici), GHSA-mwp4-54f8-5fhr/-4xrf-jv44-h6hh/-22jq-vg5j-6vgg
(ip-address), GHSA-rgw5-rvv9-x895 (brace-expansion), GHSA-5p4m-2wfm-xmqj (js-yaml).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ramon Navarro @claude Ramon Navarro
feat: add IndexNow submission feature and MCP tool authz/SSRF hardening
141aa8f 
Adds IndexNow URL submission (config + event repositories, client,
service, UI page, route, D1/PG schema + migrations, seed scripts,
spec doc) and four MCP tools: crawl_audit (with a literal-IP SSRF
denylist, self-documented DNS-rebinding ceiling), ga4_traffic and
gsc_second_page (project-membership authz via withMcpProjectAuth,
self-documented gap vs per-resource ownership), and
get_indexing_queue/submit_indexnow_urls.

This work had been sitting uncommitted on local disk for over a
week (last touched 2026-07-31) while origin/main moved 26 commits
ahead — flagged as a standing risk-of-loss finding in the daily
security audit. tsc --noEmit passes clean on the full tree.

Not pushed: local branch is 5 ahead / 26 behind origin/main and
needs a rebase (and its own review) before going up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ramon Navarro Ramon Navarro
fix: resolve oxlint errors in IndexNow/MCP tool files
476789b 
- gsc-second-page: use toSorted instead of sort
- ga4-traffic: hoist rows() helper to module scope, type Ga4Report
- crawl-audit: extract analyzePage() to cut complexity/max-depth/max-params
- war-room: replace unsafe type assertion with toAssignee() guard
- googleServiceAccount: use isRecord() type guard

All 786 tests pass; our files typecheck and lint clean. Remaining
lint/typecheck errors are pre-existing upstream alchemy.run.ts (untouched).
Ramon Navarro Ramon Navarro
feat: add Bing Webmaster OAuth connection + read-only telemetry (Phas…
3e31260 
…e 2)

- bing_connections table (sqlite + pg, mirrors gsc_connections), migrations
  drizzle/0038_loose_frog_thor.sql + drizzle-pg/0015_tricky_molecule_man.sql
- BING_OAUTH_PROVIDER_ID=bing-webmaster, scopes webmaster.read/manage
- Better Auth OAuth config + self-hosted callback route + API-key fallback
  (BING_WEBMASTER_API_KEY env)
- BingConnectionRepository, BingService, bingClient (crawl issues, visibility)
- MCP tools: get_bing_visibility, get_bing_crawl_issues (project-auth wrapped)
- 797 tests pass; schema parity green; db:generate clean on both dialects
@ramonmnavarro-byte ramonmnavarro-byte changed the title feat: IndexNow submission + MCP tool authz/SSRF hardening + War Room dashboard feat: IndexNow submission + Bing Webmaster telemetry + MCP tool authz/SSRF hardening + War Room dashboard Aug 7, 2026
Ramon Navarro Ramon Navarro
feat: surface IndexNow + Bing telemetry in War Room, document MCP too…
6c1a58c 
…ls (Phase 3)

- War Room server functions: getWarRoomIndexingTelemetry + getWarRoomBingTelemetry
  (authenticated, read-only, org-scoped; IndexNow config/event counts + recent
  ledger, Bing connection status + visibility + crawl issues, per-project errors)
- WarRoomTelemetry.tsx: IndexNow + Bing telemetry sections in the War Room UI
- web/content/docs/mcp.md: document submit_indexnow_urls, get_indexing_queue,
  get_bing_visibility, get_bing_crawl_issues
- 797 tests pass; typecheck + lint clean (only pre-existing alchemy.run.ts errors)
@ramonmnavarro-byte ramonmnavarro-byte changed the title feat: IndexNow submission + Bing Webmaster telemetry + MCP tool authz/SSRF hardening + War Room dashboard feat: IndexNow + Bing Webmaster telemetry, MCP authz/SSRF hardening, War Room dashboard Aug 7, 2026
Ramon Navarro Ramon Navarro
feat: IndexNow deploy webhook — auto-submit on deploy (Phase 4)
2888262 
- POST /api/webhooks/indexnow: shared-secret auth (X-IndexNow-Webhook-Secret
  or Bearer) + INDEXNOW_WEBHOOK_HOSTS exact allowlist; no browser session
- Payload sources: urls / changedUrls / single url / sitemap+homepage
  discovery fallback; HTTPS + public-host validation, SSRF guards,
  64 KiB / 2,000 URL / 500-per-project caps
- 5-minute dedupe via successful indexing_events ledger rows (idempotent)
- Route tree + env examples + docs/INDEXNOW_WEBHOOKS.md + 4 focused tests
- 801 tests pass; parity 132; db:generate clean both dialects; typecheck +
  lint clean (only pre-existing alchemy.run.ts errors)
@ramonmnavarro-byte ramonmnavarro-byte changed the title feat: IndexNow + Bing Webmaster telemetry, MCP authz/SSRF hardening, War Room dashboard feat: IndexNow + Bing Webmaster telemetry, deploy webhook, MCP authz/SSRF hardening, War Room dashboard Aug 7, 2026
@ramonmnavarro-byte

ramonmnavarro-byte commented Aug 7, 2026

Copy link
Copy Markdown
Author

👋 @bensenescu — ready for review whenever you get a chance.

This PR implements the full Indexing + Bing Visibility feature set (Phases 1–4 of the roadmap) plus MCP security hardening and the War Room dashboard:

  • IndexNow — IndexNowService (submit URLs, batch/retry, ledger every attempt), indexnow_configs + indexing_events tables (both dialects), project /indexing route, MCP submit_indexnow_urls + get_indexing_queue.
  • Bing Webmaster — bing_connections table, Bing OAuth via Better Auth + self-hosted API-key fallback, BingService (crawl issues, visibility), MCP get_bing_visibility + get_bing_crawl_issues.
  • Deploy webhook — POST /api/webhooks/indexnow, callable by CI/Vercel without a session. Shared-secret auth (X-IndexNow-Webhook-Secret or Bearer) + exact INDEXNOW_WEBHOOK_HOSTS allowlist. URLs from payload or sitemap+homepage fallback. HTTPS/public-host + SSRF guards, 64 KiB / 2,000 URL / 500-per-project caps, 5-min dedupe (idempotent). docs/INDEXNOW_WEBHOOKS.md + env examples.
  • Hardening — MCP tools wrapped with project-scoped auth; SSRF protections; fast-uri / undici / ip-address / brace-expansion / js-yaml bumped to clear advisories.
  • War Room — read-only telemetry sections (IndexNowTelemetrySection, BingTelemetrySection) + /war-room route.

Verification: 801 tests pass (incl. 132 schema-parity), db:generate clean on d1 + pg (no migrations), our files typecheck + lint clean. Only pre-existing upstream alchemy.run.ts errors remain (untouched).

Fork PR since direct push to upstream is denied. Happy to split into smaller PRs or adjust anything to fit your review process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ramonmnavarro-byte