Skip to content

fix(memory): canonical marker rebuild in MemoryWriter + one shared lenient parser for all readers#1593

Open
anikinsasha wants to merge 1 commit into
danielmiessler:mainfrom
anikinsasha:fix/memory-marker-canonical-rebuild
Open

fix(memory): canonical marker rebuild in MemoryWriter + one shared lenient parser for all readers#1593
anikinsasha wants to merge 1 commit into
danielmiessler:mainfrom
anikinsasha:fix/memory-marker-canonical-rebuild

Conversation

@anikinsasha

Copy link
Copy Markdown

Problem

The <!-- BEGIN ENTRIES --> / <!-- END ENTRIES --> marker handling in the hot-layer memory files has three compounding defects:

  1. parseFile locates END via first-indexOf over the whole body, so a stray END before BEGIN sends every write into the recovery branch forever.
  2. serializeFile's self-heal appends a missing BEGIN at the end of the body (after any stray END), manufacturing a permanent END-before-BEGIN inversion, and re-emits one fresh END per write. Files grow a stack of duplicate END markers, +1 per curation write, never compacted.
  3. Every read-side parser (LoadMemory hook, Pulse memory module, MemoryHealthCheck, MemoryRestore) uses strict first-indexOf/lazy-regex logic that bails to zero entries on marker disorder. Memory silently stops loading into sessions while the writer keeps curating, and the health check reports the corrupted file as healthy headroom (0/48).

Files scaffolded by pre-7.0.0 templates (which shipped without the marker pair) are corrupted this way in the wild, and copyMissing never re-templates an existing file, so they never recover on their own. On my install this produced 40+ stacked END markers and weeks of sessions running with zero memory loaded while the panel and health check showed nothing wrong.

Fix

  • MemoryWriter parse is line-based and uniformly lenient: markers count only as whole trimmed lines (an entry merely mentioning a marker can no longer truncate the block); entries = every valid-prefix line anywhere after frontmatter; invalid-prefix orphans stay in the body verbatim; CRLF tolerated; a frontmatter match that would swallow the marker block is demoted.
  • Serialize always emits the canonical shape (frontmatter → body → BEGIN → entries → END → trailing newline): one write converges any corrupted file, repeated writes are byte-identical.
  • parseMemoryContent is exported and every reader imports it — reader and writer can never diverge again.
  • MemoryHealthCheck gains a marker-sanity check (CRITICAL on structural corruption instead of false-green) and a pending-silent-loss check surfacing on-disk entries read() excludes; probe IO errors are recorded, never crash the run; a loud guard prevents importing the CLI script as a library.
  • Hardening from the audit passes on this change: O_EXCL tmp write (a symlink planted at the predictable .tmp path can no longer be written through before the rename), retention-ratio shrink guard (a near-total wipe with one token addition no longer bypasses the catastrophic-shrink guard), directory fsync after rename, embedded-newline and marker-substring entries rejected at validation with observability.
  • The inline smoke test no longer touches the LIVE principal memory file. Previously bun MemoryWriter.ts test ran against the real PRINCIPAL_MEMORY.md and its cleanup emptied it — on a populated install that wipes real memory. It now runs against .memtest.md fixtures confined to the OS temp dir (realpath-checked), and covers the corrupted shapes: END-before-BEGIN + END stack, marker-substring entries, CRLF, frontmatter-less, invalid-prefix orphans, idempotency.
  • New MemoryWriter.test.ts (19 cases) is standalone-runnable with bun test — no framework dependency added.

Verification

On the corrupted production files: one curation write converged both to canonical structure with a verbatim-entry set-diff of zero (all entries preserved), session injection went from (no entries yet) to the full entry set, and the health check went from false-green to a correct CRITICAL before the heal and clean green after. 19/19 tests + smoke pass.

…nient parser for all readers

The BEGIN/END ENTRIES marker handling had three compounding defects:

1. parseFile located END via first-indexOf over the whole body, so a stray
   END before BEGIN sent every write into the recovery branch forever.
2. serializeFile's self-heal appended a missing BEGIN at the END of the body
   (after any stray END), manufacturing a permanent END-before-BEGIN
   inversion, and re-emitted one fresh END per write: files grew a stack of
   duplicate END markers, +1 per curation write, never compacted.
3. Every read-side parser (LoadMemory hook, Pulse memory module,
   MemoryHealthCheck, MemoryRestore) used strict first-indexOf/lazy-regex
   logic that bails to zero entries on marker disorder - memory silently
   stopped loading into sessions while the writer kept curating, and the
   health check reported the corrupted file as healthy headroom (0/48).

Files scaffolded by pre-7.0.0 templates (which shipped without the marker
pair) are corrupted this way in the wild; install/copyMissing never
re-templates an existing file, so they never recover on their own.

Fix:
- MemoryWriter parse is now line-based and uniformly lenient: markers count
  only as whole trimmed lines (an entry mentioning a marker can no longer
  truncate the block); entries = every valid-prefix line anywhere after
  frontmatter; invalid-prefix orphans stay in body verbatim; CRLF tolerated.
- serialize always emits the canonical shape (frontmatter, body, BEGIN,
  entries, END, trailing newline): one write converges any corrupted file,
  repeated writes are byte-identical.
- parseMemoryContent is exported and every reader imports it - reader and
  writer can never diverge again.
- MemoryHealthCheck gains CHECK 7.5: marker structural sanity goes CRITICAL
  on corruption instead of false-green.
- validateAndDedup rejects submitted entries containing marker substrings
  (counted as dropped_malformed).
- The inline smoke test no longer touches the LIVE principal memory file
  (it previously wiped it down to its fixture entries on a populated
  install); it now runs against .memtest.md fixtures confined to the OS
  temp dir, and covers the corrupted shapes: END-before-BEGIN + END stack,
  marker-substring entries, CRLF, frontmatter-less, invalid-prefix orphans,
  idempotency.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant