Universal News Scraper

Drop-in replacement for NewsAPI — scrapes news from 4 major sources using RSS feeds and HTML parsing.

Why this exists

NewsAPI is great but has rate limits, requires an API key, and the free tier is very restrictive. This scraper gives you the same data structure with zero API keys, zero rate limits, and zero cost.

Sources

Source	Method	Status
BBC News	RSS feed	✅
ReliefWeb (UN OCHA)	RSS feed	✅
Al Jazeera	RSS feed	✅
Reuters	HTML scraping	⚠️ (some networks block)

Quick start

from universal_news_scraper import fetch_news

# Get latest crisis news
articles = fetch_news()
print(f"Got {len(articles)} articles")

# Filter by country
sudan_news = fetch_news("Sudan")

# Each article looks like NewsAPI output
for a in articles[:3]:
    print(f"[{a['source']['name']}] {a['title']}")
    print(f"  {a['url']}")

Output format

Matches NewsAPI article schema exactly — drop-in compatible:

{
    "title": str,
    "description": str,
    "content": str,
    "url": str,
    "urlToImage": str,
    "publishedAt": str,       # ISO 8601
    "source": {"name": str},  # e.g. "BBC News"
}

Features

• ✅ No API keys needed — works out of the box
• ✅ Built-in caching — 15-minute TTL, respects source servers
• ✅ Deduplication — cross-source dedup by title hash
• ✅ Graceful degradation — if a source fails, others still work
• ✅ NewsAPI fallback — pass a fallback function for resilience
• ✅ Country filtering — keyword-based filter across all sources

Installation

pip install universal-news-scraper

Or copy scraper.py directly into your project — it's a single file.

License

MIT