Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/credit-costs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -321,11 +321,11 @@ <h2>Resources</h2>
</tr>
<tr>
<td><code>search</code></td>
<td>1 credit</td>
<td>2 credits</td>
</tr>
<tr>
<td><code>search</code> + scrape</td>
<td>1 credit + 1 per scraped result</td>
<td>2 credits + 1 per scraped result</td>
</tr>
<tr>
<td><code>monitor</code> create / list / get</td>
Expand All @@ -349,7 +349,7 @@ <h2>Resources</h2>
<h2 id="top-up-credits">Top-up credits</h2>
<p>Purchased top-up credits never expire and are not tied to your current billing cycle. Cancelling your subscription zeroes only the plan&#39;s included/monthly credit allowance; any purchased top-up balance survives cancellation and stays spendable.</p>
<h2 id="free-tier">Free tier</h2>
<p>The FREE plan grants 1000 credits <strong>once, for the lifetime of the account</strong>, not monthly, and it never resets. No card is required to get them. Once they are spent, a FREE account either upgrades to a paid plan or stops.</p>
<p>The FREE plan grants 500 credits <strong>once, for the lifetime of the account</strong>, not monthly, and it never resets, plus up to 500 more from onboarding tasks. No card is required to get them. Once they are spent, a FREE account either upgrades to a paid plan or stops.</p>
<h2 id="why-crawl-billing-looks-different">Why crawl billing looks different</h2>
<p>The crawl start reserves the job. Subsequent polls charge only for newly materialized pages, not for the total accumulated page count each time.</p>
<p>That prevents the same already-seen pages from being charged again and again just because you are checking progress.</p>
Expand Down Expand Up @@ -383,15 +383,15 @@ <h2 id="simple-examples">Simple Examples</h2>
</tr>
<tr>
<td>Search for &quot;AI tools&quot; with 5 results</td>
<td>1 credit</td>
<td>2 credits</td>
</tr>
<tr>
<td>Search + scrape 3 results</td>
<td>1 + 3 = 4 credits</td>
<td>2 + 3 = 5 credits</td>
</tr>
<tr>
<td>Search + scrape, 1 scrape fails</td>
<td>1 + 2 = 3 credits (failed scrape refunded)</td>
<td>2 + 2 = 4 credits (failed scrape refunded)</td>
</tr>
</tbody></table>
<h2 id="what-usually-does-not-consume-permanent-credits">What Usually Does Not Consume Permanent Credits</h2>
Expand Down
3 changes: 2 additions & 1 deletion docs/docker/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,8 @@ <h2 id="pre-built-image">Pre-built Image</h2>
docker run -p 3000:3000 ghcr.io/us/crw:0.16.0</code></pre>
<p>Available tags: <code>latest</code>, <code>0.16</code> (tracks the current minor), <code>0.16.0</code> (pinned).
Use a pinned tag in production — <code>latest</code> rolls forward on every release.</p>
<h2 id="docker-compose">Docker Compose</h2>
<div class="callout callout-note"><div class="callout-icon"></div><div class="callout-content">500 credits to start, up to 500 more from onboarding tasks, no card.
<a href="https://fastcrw.com/register?ref=docs-docker-inline" target="_blank" rel="noopener">Get your API key</a></div></div><h2 id="docker-compose">Docker Compose</h2>
<pre data-lang="bash"><code class="language-bash">git clone https://github.com/us/crw.git
cd crw
docker compose up</code></pre>
Expand Down
12 changes: 6 additions & 6 deletions docs/docs/credit-costs.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ Cloud only (fastcrw.com) -- self-hosted instances do not have credit-based billi
| `map` | 1 credit |
| `crawl` start | 1 credit |
| `crawl` polling | New pages discovered since the previous poll |
| `search` | 1 credit |
| `search` + scrape | 1 credit + 1 per scraped result |
| `search` | 2 credits |
| `search` + scrape | 2 credits + 1 per scraped result |
| `monitor` create / list / get | 0 credits |
| `monitor` check (per run) | 1 credit per scraped page, plus 1 more credit per page judged changed |
| `browse` | Not a billed cloud endpoint. It is a local CLI/companion capability, free either way |
Expand All @@ -43,7 +43,7 @@ Purchased top-up credits never expire and are not tied to your current billing c

## Free tier

The FREE plan grants 1000 credits **once, for the lifetime of the account**, not monthly, and it never resets. No card is required to get them. Once they are spent, a FREE account either upgrades to a paid plan or stops.
The FREE plan grants 500 credits **once, for the lifetime of the account**, not monthly, and it never resets, plus up to 500 more from onboarding tasks. No card is required to get them. Once they are spent, a FREE account either upgrades to a paid plan or stops.

## Why crawl billing looks different

Expand All @@ -60,9 +60,9 @@ That prevents the same already-seen pages from being charged again and again jus
| Start one crawl job | 1 credit |
| Poll a crawl and receive 7 new pages | 7 additional credits |
| Poll again with no new pages | No new page credits |
| Search for "AI tools" with 5 results | 1 credit |
| Search + scrape 3 results | 1 + 3 = 4 credits |
| Search + scrape, 1 scrape fails | 1 + 2 = 3 credits (failed scrape refunded) |
| Search for "AI tools" with 5 results | 2 credits |
| Search + scrape 3 results | 2 + 3 = 5 credits |
| Search + scrape, 1 scrape fails | 2 + 2 = 4 credits (failed scrape refunded) |

## What Usually Does Not Consume Permanent Credits

Expand Down
5 changes: 5 additions & 0 deletions docs/docs/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ docker run -p 3000:3000 ghcr.io/us/crw:0.16.0
Available tags: `latest`, `0.16` (tracks the current minor), `0.16.0` (pinned).
Use a pinned tag in production — `latest` rolls forward on every release.

:::note
500 credits to start, up to 500 more from onboarding tasks, no card.
<a href="https://fastcrw.com/register?ref=docs-docker-inline" target="_blank" rel="noopener">Get your API key</a>
:::

## Docker Compose

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Chrome DevTools Protocol is the WebSocket-based protocol that lets external prog

## credit

A credit is the billing unit for the hosted cloud at `fastcrw.com`, worth $0.001. Every scrape costs 1 credit regardless of renderer (HTTP, lightpanda, or Chrome); there is no renderer or proxy surcharge. LLM-backed work (`POST /v1/extract`, or `formats: ["json"]` / `summary` on scrape) costs that 1-credit base render plus the actual LLM cost of the call, dynamically metered: there is no flat credit number for it. See the [Credit Costs](/docs/credit-costs) table for current billing. Map and search start-calls each cost 1 credit, and crawl jobs charge 1 credit at start plus one additional credit per page as pages are discovered during polling. The FREE plan grants 1000 credits once, for the lifetime of the account, not monthly, and they never reset. Purchased top-up credits never expire and survive plan cancellation. Self-hosted deployments have no billing layer and are unaffected by credit costs. Check your balance with `GET /api/v1/account/balance` on the SaaS control-plane. See [Credit Costs](/docs/credit-costs) for the full table.
A credit is the billing unit for the hosted cloud at `fastcrw.com`, worth $0.001. Every scrape costs 1 credit regardless of renderer (HTTP, lightpanda, or Chrome); there is no renderer or proxy surcharge. LLM-backed work (`POST /v1/extract`, or `formats: ["json"]` / `summary` on scrape) costs that 1-credit base render plus the actual LLM cost of the call, dynamically metered: there is no flat credit number for it. See the [Credit Costs](/docs/credit-costs) table for current billing. A map call costs 1 credit, a search call costs 2 credits, and crawl jobs charge 1 credit at start plus one additional credit per page as pages are discovered during polling. The FREE plan grants 500 credits once, for the lifetime of the account, not monthly, and they never reset, plus up to 500 more from onboarding tasks. Purchased top-up credits never expire and survive plan cancellation. Self-hosted deployments have no billing layer and are unaffected by credit costs. Check your balance with `GET /api/v1/account/balance` on the SaaS control-plane. See [Credit Costs](/docs/credit-costs) for the full table.

---

Expand Down
9 changes: 8 additions & 1 deletion docs/docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@

## Cloud (no installation needed)

Sign up at [fastcrw.com](https://fastcrw.com) and start using the API immediately.
Skip installation and get a free API key for fastCRW Cloud, or install the self-hosted engine as a single binary, Docker container, or Homebrew formula.

Sign up at [fastcrw.com/register](https://fastcrw.com/register?ref=docs-installation-inline) and start using the API immediately.
Native `/v1` endpoints for new CRW integrations, with `/firecrawl/v2` compatibility available for Firecrawl migrations.

```bash
Expand All @@ -12,6 +14,11 @@ curl -X POST https://api.fastcrw.com/v1/scrape \
-d '{"url": "https://example.com"}'
```

:::note
500 credits to start, up to 500 more from onboarding tasks, no card.
<a href="https://fastcrw.com/register?ref=docs-installation-inline" target="_blank" rel="noopener">Get your API key</a>
:::

## One-Line Install (Recommended)

The install script auto-detects your OS and architecture, downloads the latest binary, and installs it:
Expand Down
25 changes: 24 additions & 1 deletion docs/docs/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<div class="page-intro">
<div class="page-kicker">Get Started</div>
<h1>CRW Docs</h1>
<p class="page-subtitle">Turn websites into usable data with one API. Start with a single <code>scrape</code> request, then move into <code>search</code>, <code>map</code>, <code>crawl</code>, <code>extract</code>, or MCP only when your workflow actually needs them. Interactive browser automation is handled by the companion <code>crw-browse</code> service.</p>
<p class="page-subtitle">Open-source web scraping API for agents and RAG. Scrape, crawl, search, map, and extract data from any site with one Rust binary, self-hosted or on fastCRW Cloud.</p>
<div class="page-capabilities">
<div class="page-capability"><strong>Fastest first win:</strong> one URL, one markdown response</div>
<div class="page-capability"><strong>Works for:</strong> agents, ETL, RAG, structured extraction</div>
Expand All @@ -15,6 +15,27 @@
</div>
</div>

## Setting up an AI coding agent?

Paste this into Claude Code, Cursor, Codex, Windsurf, or any other coding agent's chat:

:::note
Install fastCRW for web access: run <code>npx crw-mcp@latest install</code> to add the MCP server and skill, then use the <code>crw_scrape</code>, <code>crw_search</code>, <code>crw_map</code>, and <code>crw_crawl</code> tools (or the native <code>/v1</code> REST API) to fetch, search, and crawl the web with the <code>CRW_API_KEY</code> in my environment.
:::

Full reference written for autonomous agents: [Agent Onboarding Guide](/agent-onboarding).

## Pick your endpoint

:::cards
::card{icon="code" title="Scrape" href="#scraping" description="One known URL in, clean markdown or JSON out."}
::card{icon="search" title="Search" href="#search" description="No URL yet? Search the web and get ranked results with content."}
::card{icon="map" title="Map" href="#map" description="Discover every URL under a domain before you scrape or crawl."}
::card{icon="layers" title="Crawl" href="#crawling" description="Scrape every page under a URL in one background job."}
::card{icon="key" title="Extract" href="#extract" description="Pull structured JSON fields from a page with an LLM."}
::card{icon="plug" title="MCP" href="#mcp" description="Give Claude, Cursor, Codex, and other agents live web access."}
:::

> **New to CRW? Use `/v1`.** The `/v1` routes are the native fastCRW API for new integrations. Use `/firecrawl/v2` when migrating existing Firecrawl v2 SDK code or when you need compatibility-only routes such as batch scrape or PDF parse.

<div class="playground-panel">
Expand All @@ -23,6 +44,8 @@
<div class="playground-copy">If this request works, you already understand the core CRW model: known URL in, clean content out. Everything else in the docs builds on that.</div>
</div>

**Try it without a key:** run this exact request in the <a href="https://fastcrw.com/playground?url=https%3A%2F%2Fexample.com&mode=scrape" target="_blank" rel="noopener">playground</a> (rate-limited preview, no signup).

## Does this work for me?

| I want to… | Use |
Expand Down
5 changes: 5 additions & 0 deletions docs/docs/js-rendering.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,11 @@ Override the rendering mode per request using `renderJs`:
| `true` | Force CDP rendering |
| `false` | HTTP only |

:::note
500 credits to start, up to 500 more from onboarding tasks, no card.
<a href="https://fastcrw.com/register?ref=docs-js-rendering-inline" target="_blank" rel="noopener">Get your API key</a>
:::

## Global default

To force JS rendering for every request that doesn't specify `renderJs` explicitly, set `render_js_default` in your config:
Expand Down
2 changes: 2 additions & 0 deletions docs/docs/map.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@
</div>
</div>

**Try it without a key:** run this exact request in the <a href="https://fastcrw.com/playground?url=https%3A%2F%2Fexample.com&mode=map" target="_blank" rel="noopener">playground</a> (rate-limited preview, no signup).

## Mapping a site with CRW

### /v1/map
Expand Down
5 changes: 5 additions & 0 deletions docs/docs/mcp-clients.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,11 @@ If you only remember one rule, remember this one: local embedded mode is the eas
claude mcp add crw -- npx -y crw-mcp
```

:::note
500 credits to start, up to 500 more from onboarding tasks, no card.
<a href="https://fastcrw.com/register?ref=docs-mcp-clients-inline" target="_blank" rel="noopener">Get your API key</a>
:::

### fastcrw.com cloud

```bash
Expand Down
5 changes: 5 additions & 0 deletions docs/docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ That's it. The agent starts `crw-mcp`, which contains the full scraping engine.

If you want host-by-host config files instead of one-liners, jump to [MCP Client Setup](#mcp-clients).

:::note
500 credits to start, up to 500 more from onboarding tasks, no card.
<a href="https://fastcrw.com/register?ref=docs-mcp-inline" target="_blank" rel="noopener">Get your API key</a>
:::

### With CDP rendering (LightPanda/Chrome)

If you have a CDP-compatible browser, pass it via env vars:
Expand Down
6 changes: 3 additions & 3 deletions docs/docs/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<div class="page-capabilities">
<div class="page-capability"><strong>Goal:</strong> first success in under 3 minutes</div>
<div class="page-capability"><strong>Base URL:</strong> <code>https://api.fastcrw.com</code></div>
<div class="page-capability"><strong>Free tier:</strong> 1000 credits, no card required</div>
<div class="page-capability"><strong>Free tier:</strong> 500 credits to start, up to 500 more from onboarding tasks, no card required</div>
</div>
<div class="page-actions">
<a class="page-btn primary" href="https://fastcrw.com/register?ref=docs-quick-start-inline" target="_blank" rel="noopener">Get API key</a>
Expand All @@ -22,13 +22,13 @@

- **Terminal** — macOS Terminal, Linux shell, or Windows [WSL](https://learn.microsoft.com/en-us/windows/wsl/install)
- **`curl`** — ships with macOS 10.15+ and most Linux distros; Windows users can use WSL or [download curl](https://curl.se/windows/)
- **A free account** at [fastcrw.com/register](https://fastcrw.com/register?ref=docs-quick-start-inline) — 1000 credits, one-time, no card required
- **A free account** at [fastcrw.com/register](https://fastcrw.com/register?ref=docs-quick-start-inline): 500 credits to start, and up to 500 more from onboarding tasks, no card required
- **Node.js 18+** — only for the MCP path (optional, described at the bottom of this page)
:::

## Get a key

Register at [fastcrw.com/register](https://fastcrw.com/register?ref=docs-quick-start-inline). Once you confirm your email, your API key appears on the dashboard. Your account starts with **1000 free credits** — one credit equals one basic scrape request, so you have plenty to explore.
Register at [fastcrw.com/register](https://fastcrw.com/register?ref=docs-quick-start-inline). Once you confirm your email, your API key appears on the dashboard. Your account starts with **500 credits, and up to 500 more from onboarding tasks**. one credit equals one basic scrape request, so you have plenty to explore.

Copy the key and keep it somewhere safe. You will paste it into the `Authorization` header below.

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/rate-limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ You already have as many requests in flight as your plan allows. The response in

### Cause 3: Credits exhausted

Once your available balance reaches zero, the API returns `429`. This covers a FREE account that has spent its lifetime 1000 credits, and a paid account that is out of credits without an active auto-recharge attempt in progress. The response includes `X-FASTCRW-Credits-Available`, `X-FASTCRW-Included-Remaining`, `X-FASTCRW-Purchased-Remaining`, and `X-FASTCRW-Upgrade-Url` headers. There is no reset header: a FREE account's 1000 credits are a one-time lifetime grant, not a monthly allowance, so there is nothing to reset.
Once your available balance reaches zero, the API returns `429`. This covers a FREE account that has spent its lifetime 500 credits (plus up to 500 more from onboarding tasks), and a paid account that is out of credits without an active auto-recharge attempt in progress. The response includes `X-FASTCRW-Credits-Available`, `X-FASTCRW-Included-Remaining`, `X-FASTCRW-Purchased-Remaining`, and `X-FASTCRW-Upgrade-Url` headers. There is no reset header: a FREE account's credits are a one-time lifetime grant, not a monthly allowance, so there is nothing to reset.

**Action:** do NOT retry. Retrying burns no credits (the request is rejected before processing) but creates noise in your logs and may hide the real cause. Alert or pause your pipeline and top up your balance or upgrade your plan.

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/recipe-mcp-5min.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Claude Code writes this into your project `.claude/mcp.json` automatically. You

### Option B: Cloud mode (all 9 tools, including `crw_search`)

Get a free API key at [fastcrw.com](https://fastcrw.com) — 1000 one-time lifetime credits, no monthly reset.
Get a free API key at [fastcrw.com](https://fastcrw.com): 500 credits to start, and up to 500 more from onboarding tasks, no monthly reset.

```bash
claude mcp add crw \
Expand Down
7 changes: 7 additions & 0 deletions docs/docs/scraping.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@
</div>
</div>

**Try it without a key:** run this exact request in the <a href="https://fastcrw.com/playground?url=https%3A%2F%2Fexample.com&mode=scrape" target="_blank" rel="noopener">playground</a> (rate-limited preview, no signup).

## Scraping a URL with CRW

### /v1/scrape
Expand Down Expand Up @@ -104,6 +106,11 @@ curl -X POST https://api.fastcrw.com/v1/scrape \
```
:::

:::note
500 credits to start, up to 500 more from onboarding tasks, no card.
<a href="https://fastcrw.com/register?ref=docs-scraping-inline" target="_blank" rel="noopener">Get your API key</a>
:::

### Response

```json
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/sdk-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ const crw = new CrwClient();
const crw = new CrwClient({ apiKey: "crw_live_..." });
```

Sign up for **1000 free credits** (no payment card, no monthly reset) at
Sign up for **500 free credits, and up to 500 more from onboarding tasks** (no payment card, no monthly reset) at
<https://fastcrw.com/dashboard> and then set `CRW_API_KEY` in your environment.

### Self-hosted HTTP server
Expand Down
7 changes: 7 additions & 0 deletions docs/docs/search.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@
</div>
</div>

**Try it without a key:** run this exact request in the <a href="https://fastcrw.com/playground?query=web+scraping+tools&mode=search" target="_blank" rel="noopener">playground</a> (rate-limited preview, no signup).

:::note
**Self-hosted users**: `docker compose up` boots the search sidecar automatically (reachable inside the Compose network as `searxng:8080`). `/v1/search` is live on `http://localhost:3000` with no extra setup. To point at a search backend you already run instead, set `CRW_SEARCH__SEARCH_BACKEND_URL=http://your-host:8080` and remove the `searxng` service from your compose file. To disable search entirely, set `[search].enabled = false` — the route returns a clear `search_disabled` error (HTTP 503). See the [Docker → Search backend](/docker) section for the full setup, the `SEARXNG_BASE_URL` vs `search_backend_url` distinction, and cold-start timing.
:::
Expand Down Expand Up @@ -116,6 +118,11 @@ curl -X POST https://api.fastcrw.com/v1/search \
```
:::

:::note
500 credits to start, up to 500 more from onboarding tasks, no card.
<a href="https://fastcrw.com/register?ref=docs-search-inline" target="_blank" rel="noopener">Get your API key</a>
:::

### Response

```json
Expand Down
5 changes: 5 additions & 0 deletions docs/docs/self-hosting.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ curl -X POST http://localhost:3000/v1/scrape \
-d '{"url":"https://example.com","formats":["markdown"]}'
```

:::note
500 credits to start, up to 500 more from onboarding tasks, no card.
<a href="https://fastcrw.com/register?ref=docs-self-hosting-inline" target="_blank" rel="noopener">Get your API key</a>
:::

## What You Get

- the same core self-hosted routes: `scrape`, `crawl`, `map`, `mcp`, `health`
Expand Down
Loading
Loading