Skip to content

OUTBOUND_DATA.md

Latest commit

 

History

History
119 lines (85 loc) · 5.98 KB

OUTBOUND_DATA.md

File metadata and controls

119 lines (85 loc) · 5.98 KB
Raw

Outbound data and third-party exposure

Shadowbroker is self-hosted: each install uses its own backend egress IP. This document is the operator-facing record for GitHub audit issues #348–#366 (tg12): what contacts third parties, why, and how to opt out without losing unrelated features.

Architecture

Path Who calls third parties
Map UI → /api/* → fetchers This install’s backend
Basemap tiles / fonts Operator’s browser (CARTO, demotiles.maplibre.org)
CCTV still/video proxy Backend (Referer/Origin set per agency — see #349)

Issue disposition summary

Issue Status Approach
#351 Fixed Region dossier via backend proxy
#352 Fixed Geocode via /api/geocode only
#360 Fixed Wikipedia/Wikidata via backend
#362 Fixed DEEPSTATE_MIRROR_COMMIT optional pin
#363 Fixed Madrid KML HTTPS-first
#364 Fixed KiwiSDR HTTPS-first + validation
#348 Accepted + gated Windows UI opt-in; env override; stealth documented
#349 Accepted + documented Agency-required Referer on backend proxy only
#350 Mitigated Callsign in UA off by default; opt-in MESHTASTIC_SEND_CALLSIGN_HEADER=true
#354 Accepted + documented Default basemap CDN; optional self-hosted tiles
#361 Mitigated UA is install handle only (operator-…), not shared Shadowbroker/ token
#366 Accepted + documented Honest per-install scrape; feature degrades if blocked

Per-install User-Agent (#361)

  • Code: backend/services/network_utils.py — outbound_user_agent(), OPERATOR_HANDLE
  • Sent: operator-7f3a92 or your-handle (purpose: nominatim) — no shared app product name
  • Why: Upstreams can rate-limit one install; a block on operator-abc123 does not require blocking every Shadowbroker user
  • Override: SHADOWBROKER_USER_AGENT replaces the entire string
  • Note: The same handle across Wikipedia, Broadcastify, etc. still correlates your traffic across those sites — that is intentional per-install attribution, not anonymity

LiveUAMap scraper (#348)

  • Layer: global_incidents (LiveUAMap map pins; GDELT text still loads without LiveUAMap)
  • Code: backend/services/liveuamap_scraper.py (Playwright + stealth for Turnstile)
  • Windows: Scraper off until you enable Global Incidents and confirm the UI dialog → backend/data/liveuamap_scraper_opt_in.json
  • Linux/macOS: Scraper runs when the layer is on (unless env forces off)
  • API: GET /api/liveuamap/scraper-status, POST /api/liveuamap/scraper-opt-in
  • Env: SHADOWBROKER_ENABLE_LIVEUAMAP_SCRAPER=true|false overrides UI on all platforms
  • Honesty: Backend-only; no browser-direct LiveUAMap from end users. Stealth remains a functional tradeoff for Turnstile; disable layer or env if unacceptable

UAP sightings (NUFORC map layer)

  • Window: last 60 days (NUFORC_RECENT_DAYS, ~2 months) from live nuforc.org
  • Cadence: Weekly (Monday 12:00 UTC) per install; typical yield ~400–500 geocoded pins
  • Between weeks: backend/data/nuforc_recent_sightings.json (7-day TTL) so restarts do not wipe the layer
  • Immediate pull: admin GET /api/refresh on that install
  • Not used for map pins: stale Hugging Face mirror (frozen ~2023) unless live is down and mirror happens to have in-window rows

CCTV proxy Referer / Origin (#349)

  • Code: backend/routers/cctv.py, backend/main.py
  • Behavior: Backend proxies streams and sets Referer / Origin each agency expects (e.g. https://511ga.org/cctv, https://informo.madrid.es/)
  • Exposure: Agency sees backend IP, not each viewer’s browser
  • Not removed: Without these headers, most public DOT/city feeds return 403 — this is not end-user browser impersonation, it is the same headers a normal browser session would send to play the feed

Meshtastic map callsign (#350)

  • Layer: sigint_meshtastic must be active for fetch_meshtastic_nodes()
  • Default: MESHTASTIC_SEND_CALLSIGN_HEADER=false — callsign not sent to meshtastic.liamcottle.net unless you set true
  • Optional: MESHTASTIC_OPERATOR_CALLSIGN for local display; header only when explicitly enabled

Basemap CDN (#354)

  • Code: frontend/src/components/map/styles/mapStyles.ts, frontend/public/map-style.json
  • Hosts: *.basemaps.cartocdn.com, demotiles.maplibre.org
  • Exposure: Browser loads tiles (client IP + pan/zoom), not the backend
  • Mitigation: Self-host raster tiles and point MapLibre sources at your tile server (operator choice; not required for core features)

Broadcastify top feeds (#366)

  • Code: backend/services/radio_intercept.py
  • Behavior: Backend fetches https://www.broadcastify.com/listen/top with per-install handle UA; parses public HTML for feed metadata and CDN stream URLs
  • Exposure: Your backend IP; 5-minute cache
  • If blocked: Panel shows empty list — feature not removed from the app
  • Not: Fake Chrome UA or cloudscraper bypass (removed in Round 7a)

Ukraine frontline mirror (#362)

  • Layer: ukraine_frontline / frontlines
  • Pin: DEEPSTATE_MIRROR_COMMIT, optional DEEPSTATE_MIRROR_REPO

Madrid CCTV (#363) / KiwiSDR (#364)

  • Madrid: HTTPS-first KML catalog; image URLs unchanged
  • KiwiSDR: HTTPS-first directory fetch; shape validation + bundled fallback

Operator checklist

  1. Set OPERATOR_HANDLE if you want a recognizable name on upstream logs.
  2. Pin DEEPSTATE_MIRROR_COMMIT for reproducible frontlines (optional).
  3. Windows: enable Global Incidents in UI only if you accept LiveUAMap server contact.
  4. Set SHADOWBROKER_ENABLE_LIVEUAMAP_SCRAPER=false to forbid LiveUAMap entirely.
  5. Set MESHTASTIC_SEND_CALLSIGN_HEADER=true only if you want callsign sent upstream.
  6. Self-host map tiles if basemap CDN exposure matters.