Async Web Scraper

A high-performance asynchronous web scraper built with aiohttp and BeautifulSoup.

Features

• Async I/O: Concurrent requests with configurable rate limiting
• Crawl mode: Follow links within a domain up to a configurable page limit
• Batch mode: Scrape a list of URLs in parallel
• Export: JSON and Markdown output formats
• Smart parsing: Strips scripts, styles, nav/footer elements
• Rate limiting: Respectful throttling with configurable requests/sec
• Error resilience: Timeouts, retries, and graceful error handling

Usage

from scraper import AsyncWebScraper
import asyncio

async def main():
    scraper = AsyncWebScraper(max_concurrent=10, rate_limit=5)
    
    # Batch scrape
    pages = await scraper.scrape_many([
        "https://example.com",
        "https://httpbin.org/html",
    ])
    
    # Crawl a domain
    pages = await scraper.crawl("https://example.com", max_pages=10)
    
    scraper.export_json("output.json")

asyncio.run(main())

CLI Demo

pip install -r requirements.txt
python scraper.py