chore: improve llms and seo discoverability

[nibzard]

Summary

Improve docs discoverability for both search engines and coding agents.

This PR:

• adds a standard alternate head link for /llms.txt
• improves dynamic docs metadata with canonical, Open Graph, and Twitter fields
• updates the root llms.txt generator with a stronger intro, top-level section ordering, and an optional link to /llms-full.txt
• adds an Instructions for AI Agents card to the overview resources section
• fixes robots.txt host/sitemap entries for docs.steel.dev

Why

The docs site had a few missed discoverability signals:

• docs pages only returned title and description
• robots.txt still pointed at an old Hiro host
• llms.txt existed, but was not surfaced as clearly as it could be from metadata, docs navigation, or its generated contents

Impact

• Better canonical and social metadata coverage across doc pages
• Clearer agent-facing discovery via /llms.txt and /llms-full.txt
• Better overview-page entry point for agents landing on the docs UI
• Correct crawler metadata in robots.txt

Validation

• bun run check
• bun run generate-llms

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview, Comment	Mar 27, 2026 5:48pm