This OMK skill packages documentation, routing, and examples for the crawling/scraping/ extraction toolbox below. No upstream library is vendored here — each is installed separately from PyPI / npm / native packages. Only docs and examples live in this directory. Respect each upstream license when you use the tool.
| Tool | Version | License | Upstream |
|---|---|---|---|
| crawl4ai | 0.9.2 | Apache-2.0 (+ attribution addendum) | https://github.com/unclecode/crawl4ai |
| scrapy | 2.17.0 | BSD-3-Clause | https://github.com/scrapy/scrapy |
| crawlee (JS) | 3.17.0 | Apache-2.0 | https://github.com/apify/crawlee |
| crawlee (Python) | 1.8.3 | Apache-2.0 | https://github.com/apify/crawlee-python |
| browser-use | 0.13.8 | MIT | https://github.com/browser-use/browser-use |
| curl-impersonate | native (C) | MIT | https://github.com/lwthiker/curl-impersonate |
| curl_cffi (binding) | 0.15.0 | MIT | https://github.com/lexiforest/curl_cffi |
| autoscraper | 1.1.14 | MIT | https://github.com/alirezamika/autoscraper |
| markitdown | 0.1.6 | MIT | https://github.com/microsoft/markitdown |
| scrcpy | 4.1 | Apache-2.0 | https://github.com/Genymobile/scrcpy |
| scrapling | 0.4.11 | BSD-3-Clause | https://github.com/d4vinci/Scrapling |
omk-crawling exists because these 11 projects did the hard work first. Every tool below is referenced, documented, and routed by this skill. Thank you to every maintainer and contributor.
1. Crawl4AI — unclecode/crawl4ai
The LLM-first web crawler. Markdown output, deep crawling, MCP server, adaptive crawling. By UncleCode (@unclecode). Apache-2.0 + attribution.
"This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai)."
2. Scrapy — scrapy/scrapy
The mature, battle-tested crawling framework. Spiders, pipelines, middleware, 10+ years of ecosystem. By Zyte and the Scrapy community. BSD-3-Clause.
3. Crawlee — apify/crawlee
Production-grade crawling infrastructure. Queues, auto-scaling, proxy rotation, unified storage. By Apify. Apache-2.0. JS/TS first-class + official Python port.
4. Browser-Use — browser-use/browser-use
LLM agents that drive a real browser. Login, multi-step navigation, form filling, clicking. By Browser Use team. MIT.
5. curl-impersonate — lwthiker/curl-impersonate
Browser TLS/JA3/HTTP2 fingerprint impersonation. The lightest way past fingerprint-based 403s. By lwthiker. MIT.
6. curl_cffi — lexiforest/curl_cffi
Python bindings for curl-impersonate.
requests-like API withimpersonate="chrome124". By lexiforest. MIT.
7. Autoscraper — alirezamika/autoscraper
Give it examples, it learns extraction rules. No browser, no selectors, ultra-lightweight. By Alireza Mika. MIT.
8. MarkItDown — microsoft/markitdown
Convert PDF, Office, images, audio, HTML → Markdown for LLM input. By Microsoft. MIT.
9. scrcpy — Genymobile/scrcpy
Android screen mirroring and control. The bridge when data lives only in a mobile app. By Genymobile (Romain Vimont). Apache-2.0.
10. Scrapling — d4vinci/Scrapling
Stealth scraping framework. Cloudflare Turnstile bypass, adaptive selectors, spider framework. By d4vinci. BSD-3-Clause.
11. insane-search — fivetaku/gptaku_plugins
Exhaustive multi-source search & auto-bypass for blocked websites. rg → ast-grep → knowledge graph → git history → MCP → WAF grid → Playwright. By fivetaku (@fivetaku) via the GPTaku Plugins marketplace. The "single blocked URL" breaker and WAF escalation engine in this toolbox. Huge shoutout to fivetaku for building the most insane search plugin in the ecosystem. 🙏
crawl4ai's license (Apache-2.0) carries a mandatory attribution addendum (see
LICENSE.txt). Any distribution, publication, or public use of crawl4ai or
derivative works must display:
"This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai)."
Short form for agent/tool output:
This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction.
| License | Tools | Key obligation |
|---|---|---|
| Apache-2.0 | crawl4ai, crawlee, scrcpy | Attribution + NOTICE retention. crawl4ai has extra addendum. |
| MIT | browser-use, curl-impersonate, curl_cffi, autoscraper, markitdown | Retain copyright + license notice. |
| BSD-3-Clause | scrapy, scrapling | Retain copyright + license. No endorsement use. |
| GPTaku Plugins | insane-search | fivetaku/gptaku_plugins marketplace. |
- MIT / BSD-3-Clause / Apache-2.0 all permit use with attribution; retain upstream copyright and license notices when you redistribute any of these libraries' code.
- Only crawl4ai's full license text is bundled (
LICENSE.txt) because its addendum explicitly requires displaying attribution. For the others, this NOTICE + the upstream links satisfy attribution since no code is vendored.