Skip to content

README.md

Latest commit

 

History

History
8 lines (6 loc) · 369 Bytes

README.md

File metadata and controls

8 lines (6 loc) · 369 Bytes
Raw

Super Webcrawler AI (LLM-Only Upgrade)

LLM-native crawler using Firecrawl (scraping to markdown) + Parallel.ai-style parallelism. No RL, driven by LangChain ReAct agents.

Setup

  1. Add API keys in agent.py/crawler_env.py (Firecrawl, HF).
  2. pip install -r requirements.txt
  3. Run: python main.py --goal "your goal" --start_url "https://start.com" --parallel 3