memoryfield-tool

A CLI tool for reading, writing and exporting memoryfields — named collections of Markdown pages with YAML frontmatter and optional vector indexes. It is aimed both at human and AI agent users.

For the RFC-style specification of the memoryfield format, see SPEC.md.

Install

uv tool install memoryfield-tool

or, from a local checkout:

uv tool install .

or, to run from source:

uv run memoryfield-tool --help

Requires Python 3.12+. Semantic search needs a local ollama with nomic-embed-text pulled; everything else works without it (search falls back to substring matching).

Commands

create NAME [--location PATH]     Create a new memoryfield (intro index page)
      [--index-location CACHE|in-field|PATH]
connect NAME LOCATION             Connect an existing directory (or s3:// URI)
      [--index-location CACHE|in-field|PATH]
                                  as a memoryfield
disconnect NAME                Remove a memoryfield from the config (data is
                                  left in place)
catalog [--field NAME] [--sort]   List pages with frontmatter metadata
fields                        List connected fields (name, transport,
                                location, and index.md title)
validate [--field NAME]           Check a field against the spec
read [--field NAME] PAGES...      Print pages with line numbers
edit [--field NAME] PAGE          Open a page in $EDITOR and write it back
write [--field NAME] PAGE         Write stdin to a page (auto frontmatter;
                                  background reindex)
delete [--field NAME] PAGE        Delete a page
path [--field NAME] [PAGE]    Print a field's root path, or a page's full path
rename [--field NAME] OLD NEW Rename a page (preserves uuid/created, refreshes
                              updated; background reindex)
new TITLE [--field NAME] [--name PAGE]   Create a page (generated frontmatter,
                                  slugified filename)
index [--field NAME]              Build or update the vector index
search [--field NAME] QUERY       Semantic search (substring fallback)
pull [--field NAME] QUERY         Search and print full content of every match
export [--field NAME]             Write the field as a .memoryfield.zip
serve [--port N] [--host H]       Serve all fields over HTTP (spec data
      [--allow-writes] [--open]     server + HTML renderer)

Commands that act on multiple fields (search, pull, catalog, validate, index, export) skip unreachable fields with an error: memoryfield <name> warning on stderr instead of aborting; use disconnect to drop a stale field.

Agent-friendly page creation

write fills missing frontmatter (uuid, created, updated; title from the filename stem) — no uuidgen, no hand-built YAML. A summary is never inferred; pass --summary to set one. write --force preserves the stored uuid/created/title and refreshes updated (pass --title to override) and rejects a conflicting incoming uuid. An existing page whose body is exactly the bare # <title> skeleton left by new is replaced without --force. validate --fix rewrites pages whose created/updated are unquoted datetimes, normalizing them to quoted strings and refreshing updated. new derives the page filename from the title (--name overrides) and prints the created page + uuid. memoryfield-tool --schema prints the full command reference (commands, flags, defaults, choices) as JSON for agents to introspect at runtime.

Pull

pull QUERY searches (semantic, substring fallback) and prints the full content of every matching page in one call — no separate read. Pages print in rank order separated by a blank line, each with a filename frontmatter key identifying it (<field>/<page> when more than one field is searched, bare <page> otherwise). Pages without frontmatter gain a minimal block; pages with malformed frontmatter get a ### FILE: header instead. Unreachable fields warn on stderr; a matched page that cannot be read (e.g. a stale index) is reported and the command exits 1. pull is read-only and never writes or reindexes.

S3-compatible stores

Fields may live on any S3-compatible object store (AWS S3, Google Cloud Storage's S3-compatible XML API, MinIO, Cloudflare R2) by connecting a s3://bucket/prefix location:

memoryfield-tool connect cadentia s3://cadentia-bucket/cadentia \
  --endpoint-url https://storage.googleapis.com --region auto
memoryfield-tool create cadentia --location s3://cadentia-bucket/cadentia \
  --endpoint-url https://storage.googleapis.com

• AWS uses the default endpoint, so --endpoint-url is optional; credentials come from the standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (and AWS_REGION) environment variables.
• Credentials can instead be stored per-field in the config: set aws_access_key_id, aws_secret_access_key and (optionally) aws_session_token for a field via the connect/create flags --aws-access-key-id, --aws-secret-access-key and --aws-session-token. Configured keys take precedence over the boto3 default credential chain (env vars, ~/.aws/, IAM roles); keys left unset fall back to that chain. aws_access_key_id and aws_secret_access_key must be set together (the session token is optional). Note that the config file is plaintext — when keys are stored there, run chmod 600 ~/.config/memoryfield-tool/config.toml and prefer the env/chain approach in shared or CI environments.
• GCS exposes an S3-compatible API at https://storage.googleapis.com; create HMAC keys in the GCS console and set them in the same AWS_ environment variables, then pass --endpoint-url https://storage.googleapis.com. The region defaults to "auto" automatically for the GCS endpoint (boto3 needs it for signature generation); pass --region to override.
• The endpoint_url and region choices are persisted per-field in the config and can be overridden at any time.
• connect probes the bucket at startup (fail-fast on bad credentials); serve probes every connected bucket before serving.
• Every command (read, write, catalog, validate, index, search, export) and the HTTP routes work identically for local and s3 fields.
• write --append on s3 is a read-modify-write (S3 has no native append), so concurrent appends can lose data; prefer write --force (PUT) for anything contended.
• edit opens any page in $EDITOR (falling back to $VISUAL, then vi) and writes back on save; s3 pages are downloaded to a temp file and uploaded through the same validation as write. The editor must block until you save and quit (e.g. 'code -w', not bare 'code').

Serving over HTTP (serve)

serve runs a Flask app (via waitress) for every connected memoryfield, exposing both the spec's writable data-server endpoints and an awiki-style HTML renderer. Field names from the config namespace the routes:

Route	Method	Description
/	GET	Google-style search — without q, search home + field list; with ?q=, HTML SERP (embedding search with substring fallback)
/{field}/	GET	Rendered index.md, or a catalog listing
/{field}/{page}	GET	Rendered page HTML
/{field}/{page}.md	GET	Raw page bytes (text/markdown)
/{field}.memoryfield.zip	GET	Full field snapshot as a zip
/{field}/search?q=	GET	Field-scoped search JSON (results array)
/search?q=	GET	Global search across all fields (results gain field)
/catalog	GET	All-pages listing (every page, all fields)
/{field}/catalog	GET	Field-scoped all-pages listing
/{field}/{page}.md	PUT	Create (201) or replace (204) a page
/{field}/{page}.md	DELETE	Remove a page and its index entries (204)

Flags:

• --port N — default 6211.
• --host H — default 127.0.0.1.
• --allow-writes — enable PUT/DELETE. No authentication is implemented, so writes are refused unless the host is loopback (127.0.0.1, localhost, ::1); --allow-writes with any other host refuses to start.
• --open — open the landing page in a browser.

Notes:

• The config is snapshotted at startup — connect a new field and restart to serve it. serve never touches last_used (the config stays CLI-owned).
• s3 pages are read live with no caching in v1 (ETag/Last-Modified caching is future work). Buckets are probed at startup, so an unreachable bucket refuses to start serve rather than 500ing per request.
• Pages literally named search or catalog are shadowed for HTML rendering (the route wins); they are still available raw at /{field}/search.md and /{field}/catalog.md. Field names search, static and catalog are likewise reserved (a field named catalog is still reachable at /catalog/ and via its page routes).
• The renderer bundles pico.css at build time (sha256-pinned, never checked into git); in a source checkout it falls back to the pinned CDN URL.
• PUT reindexes the page synchronously; DELETE drops its index rows. As in the CLI, an unavailable embedding model degrades silently to substring search.

Configuration

The config file lives at ~/.config/memoryfield-tool/config.toml on Linux (platformdirs user config dir; override with MEMORYFIELD_TOOL_CONFIG). On first load after an upgrade, a legacy config at ~/.config/memoryfield-tool.toml is copied to the new path (never when MEMORYFIELD_TOOL_CONFIG is set). It maps memoryfield names to locations:

[memoryfields.notes]
transport = "local"
location = "/home/me/memoryfields/notes"

[memoryfields.cadentia]
transport = "s3"
location = "s3://cadentia-bucket/cadentia"
endpoint_url = "https://storage.googleapis.com"
region = "auto"
index_location = "cache"

create and connect add entries; endpoint_url, region, and index_location are emitted only when set (legacy configs stay byte-compatible).

Vector index

--index-location {cache|in-field|PATH} on create and connect chooses where a field's vector index lives, persisted in the config as index_location. Resolution (all names are <base>/<field>/nomic-embed-text-v1.5.sqlite3):

• absent (default): local fields index in-field at <field>/nomic-embed-text-v1.5.sqlite3; s3 fields index to the cache dir.
• cache: ~/.cache/memoryfield-tool/indexes/<field>/nomic-embed-text-v1.5.sqlite3 (machine-local).
• PATH: a directory; the index lands at <PATH>/<field>/nomic-embed-text-v1.5.sqlite3.
• in-field: the field directory itself (local fields only).

Existing s3 fields switch to the model-coded cache path on upgrade and re-embed once on their next index run (after which rebuilds are incremental). Outside indexes are derived data, machine-local, never included in exports, and can be regenerated with index at any time. search returns every page whose cosine distance is at most --max-distance (default 0.45); cosine distance is shown in the output. It exits with status 1 and a stderr message when no pages match. When nothing meets the cut-off, no results are returned (the substring fallback only applies when the index or embedder is unavailable, and is capped at 20 results). After each write, the index is rebuilt in the background (a detached process; its stderr goes to ~/.cache/memoryfield-tool/reindex.log).

Releasing

The version is derived from git tags (see version.py): tagging a clean commit vX.Y.Z and pushing the tag publishes X.Y.Z to PyPI:

git tag vX.Y.Z
git push origin vX.Y.Z

The publish workflow re-runs the linters, typechecker and tests, builds the sdist and wheel, checks them with twine, smoke-tests the wheel in a fresh venv (the reported version must equal the tag), and uploads to PyPI via trusted publishing (OIDC — no API tokens are stored in GitHub).

Anything other than an exact tag on a clean commit builds with a dev or local version suffix, which the workflow rejects — so a dirty or mid-history tree can never be published by accident. PyPI never allows re-uploading an existing version: if a release is bad, fix forward and tag a new version.