Sentinel

Open-source all-source intelligence fusion and link-analysis platform: a Palantir-Gotham-style analyst workbench built entirely on free OSS and public OSINT sources.

Unified search, an entity graph, a map, a cross-filtering timeline, ingest connectors, cases and watchlist alerts, with provenance on every record.

Try it: dextheking1.github.io/sentinel

• a live, read-only demo running entirely in the browser against a static snapshot of the seeded corpus. No backend, nothing to install.

What it does

• Unified search across every ingested source, with facets by type, source, country, confidence band and watchlist status (SQLite FTS5 or OpenSearch).
• Link analysis on an entity/relationship graph (Cytoscape + cose-bilkent), depth-bounded cycle-safe traversal, neighbourhood focus, expand-selection.
• Map of geocoded entities with an event-density heat layer (Leaflet).
• Timeline of entity events with brush-to-cross-filter on the other canvases.
• Cases: pin entities and edges, annotate, export as JSON / CSV / Markdown.
• Watchlist alerts: stored queries evaluated against new data.
• Ingest connectors: GDELT 2.0, OpenSanctions, crt.sh, Nominatim.
• Governance: mandatory provenance, confidence scoring, retention ceiling, email redaction, per-source rate limits that respect publisher policies.

Architecture

                     +-------------------------+
   public OSINT      |  ingest connectors      |
   GDELT 2.0 ------->|  gdelt / opensanctions  |
   OpenSanctions --->|  crt.sh / nominatim     |
   crt.sh ---------->+-------------------------+
                              |  entities + relationships + events
                              v
   +-------------------------------------------------------------------+
   | FastAPI app                                     frontend (React)  |
   |  /api/search  /api/entity  /api/graph  /api/timeline             |
   |  /api/cases   /api/alerts  /api/stats   /api/ingest              |
   |        repository                   search index                 |
   |   sqlite | neo4j                  sqlite-fts5 | opensearch       |
   +-------------------------------------------------------------------+

Storage is swappable without code changes:

SENTINEL_GRAPH_BACKEND	SENTINEL_SEARCH_BACKEND	Setup
sqlite (default)	sqlite (default)	none, zero dependencies
neo4j	opensearch	docker compose up

The Neo4j repository layers on top of the SQLite one (entities are owned by SQLite, mirrored into the graph for traversal); the OpenSearch index is a flattened, denormalised mirror of the entity table for faceted search at scale.

Every entity, relationship and event carries a Provenance record: source, source identifier, URL, retrieval timestamp. require_provenance rejects any record without one.

Quick start (zero dependencies)

cd backend
uv pip install -e ".[test]"     # or: python -m venv .venv && .venv/bin/pip install -e ".[test]"
uvicorn app.main:app --reload   # http://localhost:8000

The API is live immediately. To work on the UI:

cd frontend
pnpm install
SENTINEL_API_URL=http://localhost:8000 pnpm dev   # http://localhost:5173

Build the frontend once and the FastAPI app serves it from /:

cd frontend && pnpm build        # -> frontend/dist, served by the backend

Full stack (Neo4j + OpenSearch)

docker compose up --build          # UI + API on :8000, Neo4j on :7474/:7687, OpenSearch on :9200

Optional backends are opt-in; depends_on uses required: false so the app still starts when only one is running.

Variable	Default	Purpose
SENTINEL_GRAPH_BACKEND	sqlite	sqlite or neo4j
SENTINEL_SEARCH_BACKEND	sqlite	sqlite or opensearch
SENTINEL_NEO4J_URI	bolt://neo4j:7687	Bolt endpoint
SENTINEL_OPENSEARCH_URL	https://opensearch:9200	OpenSearch endpoint
SENTINEL_RETENTION_DAYS	0 (unlimited)	ceiling on stored record age
SENTINEL_NOMINATIM_CONTACT	admin@example.com	required by the Nominatim usage policy
SENTINEL_ID_SECRET	dev key	HMAC salt for source-less entity ids
SENTINEL_REDACT_EMAILS	false	mask personal data in API responses

Seeding data

# latest 15-minute GDELT batch
curl -X POST 'http://localhost:8000/api/ingest/gdelt?limit=1500&wait=true'

# UN Security Council consolidated sanctions list
curl -X POST 'http://localhost:8000/api/ingest/opensanctions?datasets=un_sc_sanctions&wait=true'

# certificate transparency for a domain
curl -X POST 'http://localhost:8000/api/ingest/crt.sh?domain=example.com&wait=true'

# geocode pending locations (rate-limited to 1 req/sec per Nominatim policy)
curl -X POST 'http://localhost:8000/api/ingest/nominatim?limit=200&wait=true'

Or click Ingest in the UI. Every run is recorded with row-by-row counts in ingest_runs and surfaced on the dashboard.

API surface

GET  /api/health                   GET   /api/stats
POST /api/search                   GET   /api/search/facets
GET  /api/entity/{id}              GET   /api/entity/{id}/neighbors
GET  /api/entity/{id}/events       POST  /api/graph
GET  /api/timeline                 POST  /api/ingest/{source}
GET  /api/ingest/sources           POST  /api/ingest/text/enrich
GET  /api/cases                    POST  /api/cases
GET  /api/cases/{id}/export        GET   /api/alerts
POST /api/alerts                   GET   /api/docs (OpenAPI)

Ontology

Entities: Person, Organization, Location, Event, Document, Asset. Relationships: employed_by, located_at, participated_in, directed_action_toward, sanctioned_by, issued_certificate, resolved_to. Confidence is a 0-1 score with high/medium/low bands; cross-source records are deduplicated by natural key and merged, keeping the highest confidence and the union of provenance, aliases and tags.

Data sources and licensing

Sentinel only reads sources whose terms permit programmatic access. It does not scrape authenticated or paywalled services.

Source	Data	Terms
GDELT 2.0	global news events, 15-minute batches	free for research and commercial use with attribution
OpenSanctions	sanctions and PEP watchlists	CC BY 4.0 / non-commercial usage policy
crt.sh	certificate transparency log	public CT logs
Nominatim	geocoding	usage policy (identify yourself, max 1 req/sec)

Legal and ethical guardrails

• Provenance is mandatory. No record enters the store without a source, a source identifier, a URL and a retrieval timestamp. require_provenance=true enforces this at ingest time.
• Retention is configurable. SENTINEL_RETENTION_DAYS sets a hard ceiling; ingest refuses records older than it and the reaper prunes by it. The default is unlimited so the choice is deliberate, not accidental.
• Personal data is reducible. SENTINEL_REDACT_EMAILS masks personal data in API responses; documents can be stored as evidence without re-publishing them.
• Identify the crawler. The HTTP user agent and the Nominatim contact address are configurable and should point at a real operator contact before any deployment. Source rate limits are honoured per connector.
• Scope. Sentinel is a tool for analysts working on data they are entitled to hold. Watchlists and sanctions data carry accuracy caveats: a match is a lead, not a determination. Exported case reports retain their provenance so a finding can always be traced back to its primary source.

Development

cd backend && python -m pytest tests/ -q          # 18 tests
SENTINEL_LIVE_TESTS=1 python -m pytest tests/ -q  # hits the live sources
cd frontend && pnpm typecheck && pnpm build

Layout

backend/app/         config, ontology, models, schemas, db, repository,
                    repository_neo4j, search_index, services, main
backend/app/ingest/  base, gdelt, opensanctions, crtsh, geocoder, text_ner
backend/app/routers/ search, entities, ingest, cases, alerts
backend/scripts/     export_demo_snapshot.py, sync_ontology.py
backend/tests/       18 tests + a live-source test behind an env flag
frontend/public/     demo-data.json (the static snapshot the demo ships)
frontend/src/        api client, demo adapter, zustand store, ui primitives,
                    GraphCanvas, MapCanvas, TimelineCanvas, Dashboard,
                    Sidebar, EntityPanel, SearchResults, dialogs
deploy/              Dockerfile (multi-stage: builds the frontend, serves via FastAPI)
docker-compose.yml   sentinel + neo4j + opensearch
docs/                publishing-the-demo.md, dashboard.png

Status

Platform foundation, not a product. The ontology, storage abstraction, connectors, API and analyst UI are complete and seeded end-to-end against live sources; enrichment beyond the rule-based NER (spaCy models), alert delivery, and SSO are deliberately absent.

Demo

A read-only demo is published from the gh-pages branch at dextheking1.github.io/sentinel. It runs the same frontend with a static snapshot of the seeded corpus, so search, graph, map, timeline and dashboard all work with no server. Writes are rejected with an explanatory message. See docs/publishing-the-demo.md for how it is refreshed.