Skip to content

SELF_HOSTING.md

Latest commit

 

History

History
206 lines (156 loc) · 7.17 KB

SELF_HOSTING.md

File metadata and controls

206 lines (156 loc) · 7.17 KB

🌍 Self-Hosting World Monitor

Run the full World Monitor stack locally with Docker/Podman.

📋 Prerequisites

  • Docker or Podman (rootless works fine)
  • Docker Compose or podman-compose (pip install podman-compose or uvx podman-compose)
  • Node.js 22+ (for running seed scripts on the host)

🚀 Quick Start

# 1. Clone and enter the repo
git clone https://github.com/koala73/worldmonitor.git
cd worldmonitor
npm install

# 2. Start the stack
docker compose up -d        # or: uvx podman-compose up -d

# 3. Seed data into Redis
./scripts/run-seeders.sh

# 4. Open the dashboard
open http://localhost:3000

The dashboard works out of the box with public data sources (earthquakes, weather, conflicts, etc.). API keys unlock additional data feeds.

🔑 API Keys

Create a docker-compose.override.yml to inject your keys. This file is gitignored — your secrets stay local.

services:
  worldmonitor:
    environment:
      # 🤖 LLM — pick one or both (used for intelligence assessments)
      GROQ_API_KEY: ""            # https://console.groq.com (free, 14.4K req/day)
      OPENROUTER_API_KEY: ""      # https://openrouter.ai (free, 50 req/day)

      # 📊 Markets & Economics
      FINNHUB_API_KEY: ""         # https://finnhub.io (free tier)
      FRED_API_KEY: ""            # https://fred.stlouisfed.org/docs/api/api_key.html (free)
      EIA_API_KEY: ""             # https://www.eia.gov/opendata/ (free)

      # ⚔️ Conflict & Unrest
      ACLED_ACCESS_TOKEN: ""      # https://acleddata.com (free for researchers)

      # 🛰️ Earth Observation
      NASA_FIRMS_API_KEY: ""      # https://firms.modaps.eosdis.nasa.gov (free)

      # ✈️ Aviation
      AVIATIONSTACK_API: ""       # https://aviationstack.com (free tier)

      # 🚢 Maritime
      AISSTREAM_API_KEY: ""       # https://aisstream.io (free)

      # 🌐 Internet Outages (paid)
      CLOUDFLARE_API_TOKEN: ""    # https://dash.cloudflare.com (requires Radar access)

      # 🔌 Self-hosted LLM (optional — any OpenAI-compatible endpoint)
      LLM_API_URL: ""             # e.g. http://localhost:11434/v1/chat/completions
      LLM_API_KEY: ""
      LLM_MODEL: ""

  ais-relay:
    environment:
      AISSTREAM_API_KEY: ""       # same key as above — relay needs it too

💰 Free vs Paid

Status Keys
🟢 No key needed Earthquakes, weather, natural events, UNHCR displacement, prediction markets, stablecoins, crypto, spending, climate anomalies, submarine cables, BIS data, cyber threats
🟢 Free signup GROQ, FRED, EIA, NASA FIRMS, AISSTREAM, Finnhub, AviationStack, ACLED, OpenRouter
🟡 Free (limited) OpenSky (higher rate limits with account)
🔴 Paid Cloudflare Radar (internet outages)

🌱 Seeding Data

The seed scripts fetch upstream data and write it to Redis. They run on the host (not inside the container) and need the Redis REST proxy to be running.

# Run all seeders (auto-sources API keys from docker-compose.override.yml)
./scripts/run-seeders.sh

⚠️ Important: Redis data persists across container restarts via the redis-data volume, but is lost on docker compose down -v. Re-run the seeders if you remove volumes or see stale data.

To automate, add a cron job:

# Re-seed every 30 minutes
*/30 * * * * cd /path/to/worldmonitor && ./scripts/run-seeders.sh >> /tmp/wm-seeders.log 2>&1

🔧 Manual seeder invocation

If you prefer to run seeders individually:

export UPSTASH_REDIS_REST_URL=http://localhost:8079
export UPSTASH_REDIS_REST_TOKEN=wm-local-token
node scripts/seed-earthquakes.mjs
node scripts/seed-military-flights.mjs
# ... etc

🏗️ Architecture

┌─────────────────────────────────────────────┐
│                 localhost:3000               │
│                   (nginx)                    │
├──────────────┬──────────────────────────────┤
│ Static Files │      /api/* proxy            │
│  (Vite SPA)  │         │                    │
│              │    Node.js API (:46123)       │
│              │    50+ route handlers         │
│              │         │                     │
│              │    Redis REST proxy (:8079)   │
│              │         │                     │
│              │      Redis (:6379)            │
└──────────────┴──────────────────────────────┘
         AIS Relay (WebSocket → AISStream)
Container Purpose Port
worldmonitor nginx + Node.js API (supervisord) 3000 → 8080
worldmonitor-redis Data store 6379 (internal)
worldmonitor-redis-rest Upstash-compatible REST proxy 8079
worldmonitor-ais-relay Live vessel tracking WebSocket 3004 (internal)

🔨 Building from Source

# Frontend only (for development)
npx vite build

# Full Docker image
docker build -t worldmonitor:latest -f Dockerfile .

# Rebuild and restart
docker compose down && docker compose up -d
./scripts/run-seeders.sh

⚠️ Build Notes

  • The Docker image uses Node.js 22 Alpine for both builder and runtime stages
  • Blog site build is skipped in Docker (separate dependencies)
  • The runtime stage needs gettext (Alpine package) for envsubst in the nginx config
  • If you hit npm ci sync errors in Docker, regenerate the lockfile with the container's npm version: 
    docker run --rm -v "$(pwd)":/app -w /app node:22-alpine npm install --package-lock-only

🌐 Connecting to External Infrastructure

Shared Redis (optional)

If you run other stacks that share a Redis instance, connect via an external network:

# docker-compose.override.yml
services:
  redis:
    networks:
      - infra_default

networks:
  infra_default:
    external: true

Self-Hosted LLM

Any OpenAI-compatible endpoint works (Ollama, vLLM, llama.cpp server, etc.):

# docker-compose.override.yml
services:
  worldmonitor:
    environment:
      LLM_API_URL: "http://your-host:8000/v1/chat/completions"
      LLM_API_KEY: "your-key"
      LLM_MODEL: "your-model-name"
    extra_hosts:
      - "your-host:192.168.1.100"  # if not DNS-resolvable

🐛 Troubleshooting

Issue Fix
📡 0/55 OK on health check Seeders haven't run — ./scripts/run-seeders.sh
🔴 nginx won't start Check podman logs worldmonitor — likely missing gettext package
🔑 Seeders say "Missing UPSTASH_REDIS_REST_URL" Stack isn't running, or run via ./scripts/run-seeders.sh (auto-sets env vars)
📦 npm ci fails in Docker build Lockfile mismatch — regenerate with docker run --rm -v $(pwd):/app -w /app node:22-alpine npm install --package-lock-only
🚢 No vessel data Set AISSTREAM_API_KEY in both worldmonitor and ais-relay services
🔥 No wildfire data Set NASA_FIRMS_API_KEY
🌐 No outage data Requires CLOUDFLARE_API_TOKEN (paid Radar access)