Releases: HarperZ9/gather
Release list
v1.6.1
gather v1.5.0
gather v1.5.0
The accountable research-intake organ, complete.
Gather pulls information in from scattered, awkward sources, and records how each piece was
obtained, so that downstream a quote is never confused with an inference. It is the one isolated
place where network access, third-party tools, and credentials are allowed to live, hidden behind a
single adapter shape, so the rest of your system stays clean. Every item it brings back carries a
provenance receipt; every run emits a witnessed digest you can re-check.
This is the 1.5.0 milestone: organic completion. Built across fifteen reviewed releases, gated by
two independent multi-lens whole-system reviews (at 1.0 and again at 1.5). Zero runtime
dependencies, fair-source, 165 tests.
What you can do with it
- Reach almost anywhere, behind one shape. Eleven adapters share a one-method
Sourceseam:
video (yt-dlp), static web pages, RSS/Atom feeds, local files, arXiv papers, PDFs (pdftotext),
authenticated JSON APIs, JavaScript-rendered pages (a headless browser), scanned images (OCR), and
audio (transcription). Awkward access is an adapter problem, not your problem. - Know how every item was obtained. Each item carries a
Provenancereceipt: a sha256 of the
content plus amethodthat records HOW it was reached (fetched, OCR'd, transcribed, synthesized).
The method ladder is enforced at construction, so a fetch can never pose as an inference, nor an
inference as a quote. - Never confuse a quote with a synthesis. Derived items go through a
Synthesizerseam; the
default compiles inputs verbatim and a real synthesis only happens when you plug a model in, with
derived_fromrecording the inputs it was built from. - Keep a corpus you can still trust. A content-addressed store dedups bodies, keeps every
receipt, andverifyre-hashes every stored body (MATCH / MISSING / CORRUPT). A gather run emits
a re-checkable record, kept in a durable run history. - Do it safely. The single network edge enforces an http(s) allowlist and blocks private and
cloud-metadata hosts on every redirect hop, strips credentials across origins, and caps response
size. Credentials live in the environment only, never in a receipt, a URL, or a log. Every
external tool is invoked without a shell and with its arguments fenced.
Install
pip install gather-engineThe distribution is gather-engine; it installs the gather command and import gather. The core
is pure standard library. An adapter calls its external tool (yt-dlp, pdftotext, chromium,
tesseract, whisper) only if you use that adapter.
Try it offline
python -m gather.cli --help
gather docs ./notes --scope "your,terms" --store ./corpus
gather corpus verify ./corpusOr run the bundled tours, which need no install and nothing downloaded: python examples/demo.py
(a video parsed, scoped, digested, then a receipt catching tampering) and python examples/pipeline.py (the whole organ: run, store, verify, recall, with the record re-checking
from disk).
Honest scope
The static web adapter reads what the server sends and does not run JavaScript (the browser
adapter does, and says so in its method). The arXiv adapter returns abstracts, not full text. OCR
and audio transcription are best-effort machine readings, labelled as such. Recall matches scope
terms as case-insensitive substrings. A corpus is written by one process at a time. The full design
and threat model are in ARCHITECTURE.md; the version history is in
CHANGELOG.md.