feat: M7 — Firecrawl web search and article reader within the free tier - #7
Merged
Conversation
- /proxy/firecrawl/search (2 credits, 6h cache) and /scrape (1 credit, 7d cache), optional FIRECRAWL_API_KEY (503 absent) - proxy_usage table (migration 0002): guarded monthly counter, cache hits never charge, 429 past the ceiling, new month starts clean - FakeD1 now applies every migration; migrate tests track latest version Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ted) - worker: /proxy/firecrawl/search (2 credits, 6h cache) and /scrape (1 credit, 7d cache); optional FIRECRAWL_API_KEY; monthly counter in d1 (proxy_usage) fails closed with 429, cache hits never charge - connectors: 'web' source (article kind) via the authed proxy - pwa: web filter chip + calm 503/429 notes, /library/read clean reader (minimal markdown render, 65ch, Literata), web content routed to the reader from topics and global search - e2e web-content-loop.cy.ts (4 specs); deploy/readme/architecture docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the web as a content source via Firecrawl, engineered around the free plan (1,000 credits/month, no rollover; search = 2 credits per 10 results, scrape = 1 credit/page, 2 concurrent):
GET /proxy/firecrawl/search?q=andGET /proxy/firecrawl/scrape?url=(bearer-auth'd, optionalFIRECRAWL_API_KEY→ 503 without it, same https-only/private-host rules as the RSS proxy).proxy_usage, migration0002): guarded atomic increment that fails closed — past the ceiling the endpoints answer429without touching the API; cache hits never charge; a new month starts clean. Ceiling configurable viaFIRECRAWL_MONTHLY_CREDITS.webin the registry (article kind, fetch-injectable, failures → empty).webfilter chip in the library with calm notes ('busca na web não configurada.' / 'limite mensal de busca atingido — renova no próximo mês.'); in-app reader at/library/read(minimal markdown rendering, 65ch measure, Literata) with 'abrir original →'; web content attached to topics and global-search hits route to the reader.Tests
web-content-loop.cy.ts(4): stubbed search → attach → in-app reader renders markdown → calm over-budget note🤖 Generated with Claude Code