fix(memory): canonical marker rebuild in MemoryWriter + one shared lenient parser for all readers#1593
Open
anikinsasha wants to merge 1 commit into
Conversation
…nient parser for all readers The BEGIN/END ENTRIES marker handling had three compounding defects: 1. parseFile located END via first-indexOf over the whole body, so a stray END before BEGIN sent every write into the recovery branch forever. 2. serializeFile's self-heal appended a missing BEGIN at the END of the body (after any stray END), manufacturing a permanent END-before-BEGIN inversion, and re-emitted one fresh END per write: files grew a stack of duplicate END markers, +1 per curation write, never compacted. 3. Every read-side parser (LoadMemory hook, Pulse memory module, MemoryHealthCheck, MemoryRestore) used strict first-indexOf/lazy-regex logic that bails to zero entries on marker disorder - memory silently stopped loading into sessions while the writer kept curating, and the health check reported the corrupted file as healthy headroom (0/48). Files scaffolded by pre-7.0.0 templates (which shipped without the marker pair) are corrupted this way in the wild; install/copyMissing never re-templates an existing file, so they never recover on their own. Fix: - MemoryWriter parse is now line-based and uniformly lenient: markers count only as whole trimmed lines (an entry mentioning a marker can no longer truncate the block); entries = every valid-prefix line anywhere after frontmatter; invalid-prefix orphans stay in body verbatim; CRLF tolerated. - serialize always emits the canonical shape (frontmatter, body, BEGIN, entries, END, trailing newline): one write converges any corrupted file, repeated writes are byte-identical. - parseMemoryContent is exported and every reader imports it - reader and writer can never diverge again. - MemoryHealthCheck gains CHECK 7.5: marker structural sanity goes CRITICAL on corruption instead of false-green. - validateAndDedup rejects submitted entries containing marker substrings (counted as dropped_malformed). - The inline smoke test no longer touches the LIVE principal memory file (it previously wiped it down to its fixture entries on a populated install); it now runs against .memtest.md fixtures confined to the OS temp dir, and covers the corrupted shapes: END-before-BEGIN + END stack, marker-substring entries, CRLF, frontmatter-less, invalid-prefix orphans, idempotency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
<!-- BEGIN ENTRIES -->/<!-- END ENTRIES -->marker handling in the hot-layer memory files has three compounding defects:parseFilelocates END via first-indexOfover the whole body, so a stray END before BEGIN sends every write into the recovery branch forever.serializeFile's self-heal appends a missing BEGIN at the end of the body (after any stray END), manufacturing a permanent END-before-BEGIN inversion, and re-emits one fresh END per write. Files grow a stack of duplicate END markers, +1 per curation write, never compacted.indexOf/lazy-regex logic that bails to zero entries on marker disorder. Memory silently stops loading into sessions while the writer keeps curating, and the health check reports the corrupted file as healthy headroom (0/48).Files scaffolded by pre-7.0.0 templates (which shipped without the marker pair) are corrupted this way in the wild, and
copyMissingnever re-templates an existing file, so they never recover on their own. On my install this produced 40+ stacked END markers and weeks of sessions running with zero memory loaded while the panel and health check showed nothing wrong.Fix
parseMemoryContentis exported and every reader imports it — reader and writer can never diverge again.read()excludes; probe IO errors are recorded, never crash the run; a loud guard prevents importing the CLI script as a library..tmppath can no longer be written through before the rename), retention-ratio shrink guard (a near-total wipe with one token addition no longer bypasses the catastrophic-shrink guard), directory fsync after rename, embedded-newline and marker-substring entries rejected at validation with observability.bun MemoryWriter.ts testran against the realPRINCIPAL_MEMORY.mdand its cleanup emptied it — on a populated install that wipes real memory. It now runs against.memtest.mdfixtures confined to the OS temp dir (realpath-checked), and covers the corrupted shapes: END-before-BEGIN + END stack, marker-substring entries, CRLF, frontmatter-less, invalid-prefix orphans, idempotency.MemoryWriter.test.ts(19 cases) is standalone-runnable withbun test— no framework dependency added.Verification
On the corrupted production files: one curation write converged both to canonical structure with a verbatim-entry set-diff of zero (all entries preserved), session injection went from
(no entries yet)to the full entry set, and the health check went from false-green to a correct CRITICAL before the heal and clean green after. 19/19 tests + smoke pass.