Skip to content

Draft regulated-memory evaluation preprint#26

Draft
ebeirne wants to merge 4 commits into
masterfrom
agent/regulated-memory-preprint
Draft

Draft regulated-memory evaluation preprint#26
ebeirne wants to merge 4 commits into
masterfrom
agent/regulated-memory-preprint

Conversation

@ebeirne

@ebeirne ebeirne commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • adds an arXiv-compatible preprint source for the regulated-memory evaluation
  • separates live executions from capability-assessed columns
  • documents methodology limits, evaluator conflict of interest, and vendor correction process
  • cites adjacent primary research including LoCoMo, LongMemEval, LongMemEval-V2, WorldDB, TOKI, and A-TMA
  • records a successful Tectonic 0.16.9 build and full four-page visual inspection
  • adds a versioned JSON Schema and a validated machine-readable Lians 0.4.1 live-run archive
  • explicitly refuses to reconstruct missing vendor raw outputs from narrative prose

Verification

  • 12 repository CI jobs pass on prior preprint commits
  • PDF builds successfully with BibTeX
  • four letter-sized pages rendered and inspected
  • no unresolved references or replacement glyphs
  • no Unicode em dashes
  • PDF link borders hidden for clean publication output
  • evidence JSON validates against Draft 2020-12 schema
  • five unique invariant records verified
  • archived run harness blob matches current master

Remaining before arXiv submission

  • incorporate material vendor corrections received during right of reply
  • rerun mem0 OSS and Graphiti OSS and archive their exact machine-readable outputs
  • complete external methodology review
  • confirm final author and affiliation metadata

This remains a draft until those methodology gates are complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants