feat(seo): announce robosystems.ai's new URLs to IndexNow

[jfrench9]

Depends on robosystems#1462, which adds the shared action this references at @main. Merge that one first.

Why

Bing has crawled nothing under /docs, three days after the docs launched. Its Site Explorer folder tree for this site holds research, pages and blog — no docs node — and the newest crawl date on any root page is 15 Sep. Its sitemap was read on 9/19 at 295 URLs against a live file of 375.

The sitemap is the passive half: it says what exists and waits. IndexNow is the half that pushes — one POST and Bing, and through it Copilot and DuckDuckGo, plus Yandex, Seznam, Naver and Yep know within minutes. Google ignores the protocol.

What's here

The work is in the shared action. This is a key file, a schedule and the host.

The schedule is deliberate, not a deploy hook. This site's pages come from three places and only one is a deploy of this app: the guides and technical docs publish from the robosystems repo to S3 and arrive through an hourly revalidate, and the API and GraphQL references follow the live spec the same way. A workflow_run hook on Deploy Production would catch one of the three. Diffing the live sitemap every three hours catches all of them, and a run with nothing new submits nothing.

The key is public by design — the same value is served at the host root, which is how IndexNow verifies we own the host. So it is a repository variable (INDEXNOW_KEY, already set), not a secret, and the file is committed rather than templated. public/<key>.txt ships in the image; the Dockerfile already copies public/.

A test asserts the filename and the contents match. They are the same string in two places and nothing at runtime notices when they drift — the submissions just start failing key validation while the workflow stays green. The action checks the served file before submitting; this checks the file we ship, so a typo fails in CI rather than three hours after a deploy.

Verified

• Action logic run against the live sitemap: 375 URLs parsed, 0 malformed, 67 GraphQL pages; a diff against a snapshot without them returns exactly 66.
• POST https://api.indexnow.org/indexnow with this host and key returns HTTP 202 — accepted, key validation pending, which is correct for a key not yet served. It validates once this deploys.
• npm run test:all green: 71 files, 583 tests, format, lint, typecheck, cf-lint.

After merge

The first run has no snapshot, so it announces all 375 URLs — which is the intent right now. Every run after that sends only what appeared. Worth watching the first scheduled run's summary to confirm the key verifies.

Not done here: the other four hosts. Each needs its own key, since the key file proves ownership of the host serving it, so porting is a key file plus this workflow per repo.

🤖 Generated with Claude Code

[github-actions]

Change classification: normal — platform or application code changed.

A normal change carries a documented automated review on this pull request before merge, in place of a second human approver. The change-classification job requests that review and passes once it has been posted. Recorded by the change-classification job.

[jfrench9]

@claude please review this PR

[claude]

Claude finished @jfrench9's task in 1m 24s —— View job

---

Review

• Gather context (PR body, diff, files)
• Review .github/workflows/indexnow.yml
• Review public/c2029278e9c11a6a3d0ebb2f3c7af661.txt
• Review src/app/__tests__/indexnow-key.test.ts
• Post review feedback

This is a clean, well-scoped change — three files, no surprises, and the PR description does the hard work of explaining why each piece exists.

What's solid:

• public/c2029278e9c11a6a3d0ebb2f3c7af661.txt — single line, 32 bytes, no trailing newline, filename matches contents exactly (verified against the Dockerfile's COPY --from=builder .../app/public ./public at line 96, so it does ship in the image).
• src/app/__tests__/indexnow-key.test.ts — good coverage: exactly one key file, filename/content match, no stray whitespace. Only one .txt file currently lives under public/ that matches the [a-f0-9]{8,128} pattern, so no false positives today.
• .github/workflows/indexnow.yml — scoped permissions (contents: read, actions: write for the cache), concurrency with cancel-in-progress: false to avoid a mid-flight run clobbering the snapshot, and the reasoning for polling the sitemap on a schedule instead of hooking the deploy is sound given pages arrive from three different sources.

Minor, non-blocking notes:

1. uses: RoboFinSystems/robosystems/.github/actions/indexnow-ping@main (indexnow.yml:40) pins to a mutable branch rather than a tag/SHA. Since this is a same-org shared action that's still landing (per the PR's stated dependency on robosystems#1462), that's a reasonable stopgap — but once that action stabilizes, consider pinning to a release tag or commit SHA so a future breaking change in the shared action doesn't silently break this scheduled job without a corresponding version bump here.
2. In the test file, if keyFiles were ever empty, the second and third it blocks (const [file] = keyFiles) would throw on undefined.replace(...) rather than fail with a clean assertion message — the first test (toHaveLength(1)) would already flag the real problem, so this is cosmetic only.

Nothing here blocks merge. Nice attention to the "silent failure" mode with the filename/content test — that's the right thing to guard in CI rather than discover three hours after a deploy.

Note: I wasn't able to execute the test suite or fetch origin/main in this sandboxed session (git fetch and test-runner commands required approval that wasn't available), so this review is based on static reading of the three changed files plus the Dockerfile. The PR body states npm run test:all was run green (71 files, 583 tests) by the author.