Skip to content

Releases: HarperZ9/gather

v1.6.1

Choose a tag to compare

@HarperZ9 HarperZ9 released this 07 Jul 16:23
542f4e7

What's Changed

Full Changelog: v1.6.0...v1.6.1

gather v1.5.0

Choose a tag to compare

@HarperZ9 HarperZ9 released this 25 Jun 19:49

gather v1.5.0

The accountable research-intake organ, complete.

Gather pulls information in from scattered, awkward sources, and records how each piece was
obtained, so that downstream a quote is never confused with an inference. It is the one isolated
place where network access, third-party tools, and credentials are allowed to live, hidden behind a
single adapter shape, so the rest of your system stays clean. Every item it brings back carries a
provenance receipt; every run emits a witnessed digest you can re-check.

This is the 1.5.0 milestone: organic completion. Built across fifteen reviewed releases, gated by
two independent multi-lens whole-system reviews (at 1.0 and again at 1.5). Zero runtime
dependencies, fair-source, 165 tests.

What you can do with it

  • Reach almost anywhere, behind one shape. Eleven adapters share a one-method Source seam:
    video (yt-dlp), static web pages, RSS/Atom feeds, local files, arXiv papers, PDFs (pdftotext),
    authenticated JSON APIs, JavaScript-rendered pages (a headless browser), scanned images (OCR), and
    audio (transcription). Awkward access is an adapter problem, not your problem.
  • Know how every item was obtained. Each item carries a Provenance receipt: a sha256 of the
    content plus a method that records HOW it was reached (fetched, OCR'd, transcribed, synthesized).
    The method ladder is enforced at construction, so a fetch can never pose as an inference, nor an
    inference as a quote.
  • Never confuse a quote with a synthesis. Derived items go through a Synthesizer seam; the
    default compiles inputs verbatim and a real synthesis only happens when you plug a model in, with
    derived_from recording the inputs it was built from.
  • Keep a corpus you can still trust. A content-addressed store dedups bodies, keeps every
    receipt, and verify re-hashes every stored body (MATCH / MISSING / CORRUPT). A gather run emits
    a re-checkable record, kept in a durable run history.
  • Do it safely. The single network edge enforces an http(s) allowlist and blocks private and
    cloud-metadata hosts on every redirect hop, strips credentials across origins, and caps response
    size. Credentials live in the environment only, never in a receipt, a URL, or a log. Every
    external tool is invoked without a shell and with its arguments fenced.

Install

pip install gather-engine

The distribution is gather-engine; it installs the gather command and import gather. The core
is pure standard library. An adapter calls its external tool (yt-dlp, pdftotext, chromium,
tesseract, whisper) only if you use that adapter.

Try it offline

python -m gather.cli --help
gather docs ./notes --scope "your,terms" --store ./corpus
gather corpus verify ./corpus

Or run the bundled tours, which need no install and nothing downloaded: python examples/demo.py
(a video parsed, scoped, digested, then a receipt catching tampering) and python examples/pipeline.py (the whole organ: run, store, verify, recall, with the record re-checking
from disk).

Honest scope

The static web adapter reads what the server sends and does not run JavaScript (the browser
adapter does, and says so in its method). The arXiv adapter returns abstracts, not full text. OCR
and audio transcription are best-effort machine readings, labelled as such. Recall matches scope
terms as case-insensitive substrings. A corpus is written by one process at a time. The full design
and threat model are in ARCHITECTURE.md; the version history is in
CHANGELOG.md.