Skip to content

Move search api to Worker, and search indexes to R2#100

Merged
kamilkisiela merged 23 commits into
mainfrom
kamil-r2-search
May 4, 2026
Merged

Move search api to Worker, and search indexes to R2#100
kamilkisiela merged 23 commits into
mainfrom
kamil-r2-search

Conversation

@kamilkisiela
Copy link
Copy Markdown
Contributor

@kamilkisiela kamilkisiela commented Apr 29, 2026

During the docs build, /api/search.json is prerendered and exported as the search index.
The website workflow uploads that index to R2 under search/<github-sha>.json, then deploys the docs Worker with SEARCH_INDEX_KEY=<github-sha>.

At runtime, the docs app still calls /api/search, but the docs Worker intercepts that request and proxies it through the SEARCH_API service binding to the new hive-search-api Worker.
The docs Worker forwards the active SEARCH_INDEX_KEY as x-search-index-key header, and the Search API Worker loads the matching R2 object, creates the Fumadocs search server, and caches it in Worker memory.

PR previews reuse the same Search API Worker but upload and reference their own SHA-specific index.

The generated search index is 3MB raw.


Current main ships 25MB file with search related data, that is sent to every user as 6-7mb gzipped content.
Slow 4G network = 26s to load the file.

Now this file is smaller (3MB raw) and sent from R2 to Worker (and cached per cf location), so in case of cold starts (that I try to eliminate with perwarming the cf location), the network has no impact on search performance.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 29, 2026

@kamilkisiela
Copy link
Copy Markdown
Contributor Author

@copilot how to fix this error:

$ bun --bun ./tools/export-search-indexes.ts ./.output/search/latest.json
60 | lastModified?: Date;
61 | },
62 | }
63 | }>({"doc":{"passthroughs":["extractedReferences","_headerImage","_mobileImage","_ogImage","lastModified"]}});
64 |
65 | export const blog = await create.docLazy("blog", "content/blog", import.meta.glob([".//*.{mdx,md}"], {
^
TypeError: import.meta.glob is not a function. (In 'import.meta.glob(["./
/*.{mdx,md}"], {
base: "./../content/blog",
query: {
collection: "blog",
only: "frontmatter"
},
import: "frontmatter",
eager: !0
})', 'import.meta.glob' is undefined)
at /home/runner/work/docs/docs/packages/documentation/.source/server.ts:65:78

Bun v1.3.5 (Linux x64)
error: script "search:export" exited with code 1

options: {
api: withBasePath("/api/search"),
type: "static",
delayMs: 500,
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default was so low that when I typed "router" it ended end with ro, rout and router http calls, first two were not even cancelled... :D

@kamilkisiela kamilkisiela temporarily deployed to storybook-preview April 30, 2026 16:09 — with GitHub Actions Inactive
@kamilkisiela kamilkisiela changed the title WIP: lightweight search (at some point...) Move search api to Worker, and search indexes to R2 Apr 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant