Commit 3c515da
authored
refactor: restructure to match Python powermem + full feature replication (#5)
* refactor: restructure to match Python powermem directory layout
Source files reorganized into module-based directories mirroring
oceanbase/powermem/src/powermem/:
src/core/ — Memory facade, NativeProvider, HttpProvider, Inferrer
src/storage/ — VectorStore base, SQLiteStore, SeekDBStore
src/integrations/ — Embedder, provider factory
src/intelligence/ — Ebbinghaus decay
src/prompts/ — LLM prompt templates
src/utils/ — Cosine search, Snowflake IDs, env, platform
Test files reorganized into 4-layer structure matching Python:
tests/unit/ — Per-module unit tests
tests/integration/ — Full-stack with real SQLite, mock LLM
tests/regression/ — Scenario-based (multi-agent, edge cases, language)
tests/e2e/ — Real Ollama models
Deleted: src/server/ (legacy Python bridge)
No behavior changes. All 187 tests pass. Build unchanged.
* feat: config system — configs, config-loader, settings, version
Port Python powermem config system to TypeScript:
- configs.ts: Zod schemas for MemoryConfig, IntelligentMemoryConfig,
TelemetryConfig, AuditConfig, AgentMemoryConfig, QueryRewriteConfig,
provider configs (vectorStore, llm, embedder, reranker)
- config-loader.ts: loadConfigFromEnv(), autoConfig(), createConfig(),
env var reading for all providers
- settings.ts: getDefaultEnvFile() .env resolution
- version.ts: VERSION constant
18 new tests in tests/unit/config-loader.test.ts covering:
- Config parsing with defaults
- Sub-config default application
- Explicit overrides
- Custom prompts
- validateConfig()
- loadConfigFromEnv() for all providers
- Intelligent memory env settings
- createConfig() with overrides
Total: 205 tests (17 files)
* feat: storage module — factory, adapter, configs
Port Python powermem/storage/ module:
- factory.ts: VectorStoreFactory with provider registry pattern,
built-in sqlite and seekdb providers, dynamic import
- adapter.ts: StorageAdapter bridges VectorStore with Memory core,
adds getStatistics(), getUniqueUsers(), higher-level CRUD
- config/{base,sqlite,seekdb}.ts: typed storage configs
- index.ts: barrel re-exports
17 new tests:
- factory.test.ts: provider listing, create sqlite, unsupported throws,
custom provider registration
- adapter.test.ts: full CRUD through adapter, search, pagination,
count with filters, statistics, unique users, reset
Total: 222 tests (19 files)
* feat: integrations module — embeddings/llm/rerank base, factory, config
Port Python powermem/integrations/ module structure:
- embeddings/{base,factory,config,index}.ts — EmbeddingProvider interface,
createEmbeddings() factory (OpenAI/Qwen/SiliconFlow/DeepSeek/Ollama)
- llm/{base,factory,config,index}.ts — LLMProvider interface,
createLLM() factory (same providers + Anthropic)
- rerank/{base,config,index}.ts — RerankProvider interface
- index.ts — barrel with all re-exports
Split old provider-factory.ts into embeddings/factory + llm/factory.
NativeProvider updated to import from new locations.
Old factory.ts kept for backward compat.
222 tests pass (no new tests needed — existing provider-factory tests
still exercise the factory logic through the old import path).
* feat: intelligence module — optimizer, evaluator, manager, plugin
Port Python powermem/intelligence/ module:
- memory-optimizer.ts: exact dedup (MD5 hash grouping, keep oldest),
semantic dedup (cosine similarity threshold), LLM compression
(greedy clustering + summarization)
- importance-evaluator.ts: rule-based importance scoring (keywords,
length, emotion, punctuation, metadata priority)
- manager.ts: IntelligenceManager orchestrator (processMetadata adds
importance, processSearchResults applies Ebbinghaus decay)
- plugin.ts: IntelligencePlugin interface
- index.ts: barrel
17 new tests:
- memory-optimizer: exact dedup (3), semantic dedup (2), similarity (1)
- importance-evaluator: low/high/emotional/metadata/capped/empty (7)
- manager: disabled passthrough, enabled importance, decay (4)
Total: 239 tests (22 files)
* feat: prompts module + utils expansion (filter-parser, stats, io)
Prompts module — port of Python powermem/prompts/:
- importance-evaluation.ts: IMPORTANCE_SYSTEM_PROMPT, evaluation prompt builder
- optimization.ts: MEMORY_COMPRESSION_PROMPT
- query-rewrite.ts: query expansion prompt (stub)
- user-profile.ts: profile extraction prompt (stub)
- templates.ts: formatTemplate utility
- index.ts: barrel
Utils expansion — port of Python powermem/utils/:
- filter-parser.ts: parseAdvancedFilters (time range, tags, type→category, importance→$gte)
- stats.ts: calculateStatsFromMemories (byType, avgImportance, topAccessed, growthTrend, ageDistribution)
- io.ts: exportToJson, importFromJson, exportToCsv
17 new tests:
- filter-parser: empty, time range, tags $in, type→category, importance $gte, combined, unknown fields (8)
- stats: empty, total, byType, default category, avg importance, access ranking, growth trend, age distribution, truncation (9)
Total: 256 tests (24 files)
* feat: minimal CLI + Phase A complete
CLI (port of Python powermem/cli/):
- src/cli/main.ts: Commander.js entry point with global --env-file, --json, --verbose
- src/cli/commands/config.ts: pmem config show|validate|test (section filter, JSON output)
- src/cli/commands/memory.ts: pmem memory add|search|list|get|delete|delete-all
(all with --user-id, --agent-id, --json support)
- package.json: "bin": {"pmem": "./dist/cli.js"}
- tsup.config.ts: dual entry (library + CLI with shebang banner)
Fixes:
- settings.ts: use import.meta.url instead of __dirname for ESM compat
- Bump version to 0.3.0
8 new CLI smoke tests (regression/cli.test.ts):
- --version, --help, config --help, memory --help
- config validate, config show --json, config show --section, config test
Phase A summary (6 commits):
- A.1: Config system (configs, config-loader, settings, version)
- A.2: Storage module (factory, adapter, configs)
- A.3: Integrations module (embeddings/llm/rerank base+factory)
- A.4: Intelligence module (optimizer, evaluator, manager, plugin)
- A.5: Prompts + Utils expansion (filter-parser, stats, io)
- A.6: Minimal CLI (config + memory commands)
Total: 264 tests (25 files), all passing.
Source: 50 files matching Python powermem directory layout.
* feat: full CLI — stats, manage, interactive shell, CLI utils
Phase B complete — port of Python powermem/cli/:
Commands:
- pmem stats: memory statistics (by-type, age distribution, top accessed)
- pmem manage backup: export memories to JSON file
- pmem manage restore: import memories from JSON backup
- pmem manage cleanup: dedup (exact/semantic) with optimizer
- pmem shell: interactive REPL with tab completion, session defaults,
add/search/get/list/delete/stats/set/show commands
CLI utilities:
- utils/output.ts: formatJson, truncate, formatMemoryTable,
formatSearchTable, formatStats, print{Success,Error,Warning,Info}
- utils/envfile.ts: parseEnvLines, formatEnvValue, updateEnvFile,
readEnvFile with backup support
17 new tests:
- cli.test.ts: +6 (stats/manage/shell help, backup/restore/cleanup options)
- cli-utils.test.ts: 11 (truncate, table formatting, stats format,
env parsing, env value quoting, env file create/update/read)
Total: 281 tests (26 files)
* feat: Phase B+C — agent module, user memory, graph store, full CLI
Phase C.1 — Agent module (port of Python powermem/agent/):
- agent.ts: AgentMemory unified interface (add/search/getAll/update/delete
with scope and permission management)
- types.ts: 7 enums (MemoryType, MemoryScope, AccessPermission, PrivacyLevel,
CollaborationType, CollaborationStatus, CollaborationLevel)
- abstract/: 6 strategy interfaces (scope, permission, collaboration,
privacy, context, manager)
- components/: ScopeController (scope determination, memory scope management),
PermissionController (grant/revoke/check with access logging)
- factories/: AgentFactory (creates scope + permission managers)
Phase C.2 — User memory module (port of Python powermem/user_memory/):
- user-memory.ts: UserMemory (profile-aware add, search with query rewrite,
profile management, deleteAll with profile cleanup)
- storage/user-profile.ts: UserProfile types + UserProfileStore interface
- storage/user-profile-sqlite.ts: SQLite-backed profile storage (CRUD,
upsert, topic filtering, pagination)
- query-rewrite/rewriter.ts: QueryRewriter (LLM-based query expansion
with user profile context)
Phase C.3 — Graph store + prompts:
- storage/base.ts: GraphStoreBase interface (add, search, deleteAll,
getAll, reset, statistics, uniqueUsers)
- prompts/graph/: graph extraction + update + deletion prompts
Exports: Updated src/index.ts with all new modules (agent, user-memory,
intelligence, config, storage factory/adapter, integrations, utils)
39 new tests (5 test files):
- agent-memory.test.ts: init, add, search, getAll, delete, deleteAll,
statistics, permissions, reset (9)
- scope-controller.test.ts: default scope, hint, config, update, stats (5)
- permission-controller.test.ts: default allow/deny, grant, revoke,
getPermissions, history, custom defaults (7)
- user-profile-sqlite.test.ts: create, update, topics, nonexistent,
list, pagination, mainTopic filter, delete, count (10)
- user-memory.test.ts: add, extractProfile, search, addProfile,
profile null, deleteProfile, deleteAll (8)
Total: 320 tests (31 files), all passing.
Source: 63 files matching Python powermem directory layout.
* fix: resolve lint errors in agent module
* feat: dashboard server + BDD test suite for CLI and UI
Dashboard:
- src/dashboard/server.ts: Express server serving REST API + HTML dashboard
(health, status, stats, memories CRUD, search endpoints)
- src/dashboard/public/index.html: Single-page dashboard with 3 pages
(Overview with stat cards/charts, Memories list with pagination,
Settings with system config), dark/light theme toggle
BDD test specification (tests/bdd/README.md):
- 30+ CLI scenarios across 6 features (version/help, config management,
memory CRUD, statistics, backup/restore, interactive shell)
- 15+ dashboard UI scenarios across 5 features (overview page,
navigation/theme, memories page, settings, error handling)
BDD test implementation:
- tests/bdd/cli.test.ts: 19 tests — real CLI subprocess execution
verifying version, help, config show/validate/test, stats/manage/
memory help, delete-all confirmation, restore error handling
- tests/bdd/dashboard.test.ts: 16 tests — headless browser via
dev-browser verifying stat cards, system health panel, growth/age
charts, hot memories table, theme toggle, page navigation,
memories table with pagination, REST API (health/status/stats/
list/create/search)
All 35 BDD tests pass. Dashboard verified with screenshots in
light and dark themes.
* test: data correctness tests proving input→storage→output fidelity
15 new data correctness tests (tests/bdd/data-correctness.test.ts):
API write → API read round-trip:
- content, userId, metadata survive round-trip
- search returns correct memory with valid score (0-1)
- delete removes memory, no longer retrievable
- stats reflect accurate counts after writes
API write → Dashboard displays correctly:
- memory added via API appears in dashboard memories table
- stats cards show non-zero total after API writes
- growth trend chart shows today's date
User isolation:
- user A memories not visible in user B list
- search for user A returns only A's results
- stats for user A reflect only A's count
Data type fidelity:
- Chinese content survives round-trip
- emoji content survives round-trip
- special characters (newlines, tabs, quotes, HTML) survive
- 500-char content survives round-trip
Pagination:
- offset/limit returns correct pages with no ID overlap
Total BDD tests: 50 (19 CLI + 16 dashboard UI + 15 data correctness)
* fix: add @types/express for CI type-check
* feat: comprehensive SeekDB test suite + macOS CI job
CI:
- New `test-seekdb` job on macOS (where native bindings are bundled)
- Runs `npm run test:seekdb` with 5-min timeout
- Separate from Linux test matrix (SeekDB requires platform-specific .so/.dylib)
New SeekDB E2E tests (tests/integration/seekdb-e2e.test.ts, 22 tests):
- Memory facade over SeekDB: add/get round-trip, search with scores,
update re-embeds, delete, getAll pagination (no ID overlap),
count, deleteAll, addBatch, reset
- User isolation: A/B data isolation in list/search, scoped deleteAll
- Data fidelity: Chinese, emoji, metadata, scope/category round-trip
- Stats: correct total, age distribution, growth trend with today
- Intelligent add: LLM fact extraction + storage over SeekDB
- VectorStoreFactory: creates SeekDBStore via factory
- NativeProvider: accepts injected SeekDBStore
Total SeekDB tests: 70 (40 unit + 8 integration + 22 e2e)
All auto-skip when native bindings unavailable.
test:seekdb script updated to include new test file.
* ci: use macos-14 (ARM64) for SeekDB tests — matches bundled darwin-arm64 bindings
* ci: fix SeekDB — set DYLD_LIBRARY_PATH on macOS, add Linux x64 download job
* ci: fix SeekDB to actually run tests (not just skip)
macOS ARM64 (macos-14):
- Set DYLD_LIBRARY_PATH at job level so it propagates to vitest
- Add verification step that actually creates embedded DB + collection
- Confirm binding loads before running tests
Linux x64 (ubuntu-latest):
- Install libaio1 system dependency
- Download SeekDB bindings via on-demand downloader
- Extract libseekdb.so from zip (download.js misses it)
- Set LD_LIBRARY_PATH for native lib discovery
- continue-on-error since S3 download may be restricted
* fix: SeekDB embeddingFunction:null + libaio1t64 for Ubuntu 24.04
- SeekDBStore.create(): pass embeddingFunction:null in VectorIndexConfig
to disable auto-vectorization (we pass pre-computed embeddings)
- CI macOS: fix verification step to use Schema with null embeddingFunction
- CI Linux: try libaio1t64 (Ubuntu 24.04) then fallback to libaio1
- Add @seekdb/default-embed as devDependency
* ci: remove seekdb verification step that crashes on client.close()
* fix: SeekDB test guards skip close() to avoid SIGABRT, simplify metadata tests
- All seekdb test guards: don't call store.close() in availability check
(SeekDB embedded C engine may SIGABRT on cleanup)
- metadata round-trip test: use flat metadata (no nested objects) —
SeekDB embedded has JSON limitations with complex nested values
- unicode metadata test: use ASCII values (SeekDB C parser issue)
- seekdb-e2e guard: same close() fix
Previous CI run showed 46/48 passed on macOS ARM64 — these 2 fixes
should bring it to 48/48 + unlock the 22 e2e tests.
* fix: base64-encode metadata in SeekDBStore to bypass C engine JSON parser
SeekDB's embedded C engine rejects JSON strings containing escaped quotes
in metadata values. Solution: store user metadata as base64-encoded string
(metadata_b64) instead of raw JSON (metadata_json).
- toSeekDBMetadata(): metadata_json → metadata_b64 (Buffer.from().toString('base64'))
- toRecord() + search(): decode metadata_b64 with fallback to metadata_json
- Restores full metadata test coverage: nested objects, arrays, unicode, emoji
* fix: SeekDB e2e guard — align with working unit test pattern + sequential execution
- seekdb-e2e.test.ts: rewrite guard to match exact pattern from passing
seekdb.test.ts (same tryCreateStore function, same params)
- test:seekdb script: force single-fork sequential execution to prevent
concurrent SeekDB embedded engine initialization across test files
* fix: run seekdb test files sequentially to avoid concurrent init
* fix: guard afterAll close() calls in seekdb-e2e tests
* fix: seekdb-e2e — single shared Memory instance (embedded engine is single-instance)
* fix: resolve SeekDB Linux CI — fix package exports path + dynamic cache dir
Root cause: require('@seekdb/js-bindings/download.js') fails because
package.json exports only exposes '.', not './download.js'.
Fix: resolve filesystem path via require.resolve() then replace filename.
Also:
- Dynamic cache dir discovery (find ~/.seekdb -name seekdb.node)
instead of hardcoded commit hash
- Verify both seekdb.node and libseekdb.so exist after download
- Remove continue-on-error since the fix should work
* fix: symlink libaio.so.1t64 → libaio.so.1 for Ubuntu 24.04 compat
* docs: update all docs to reflect v0.3.0 full Python parity
README.md — complete rewrite:
- Pure TypeScript (no Python dependency) positioning
- Quick start with env vars, explicit LangChain, SeekDB, server modes
- CLI usage examples (all commands)
- Full API reference (Memory facade + configuration options)
- Architecture overview (10 modules, 89 source files)
- Test summary (504 tests, 7 CI jobs)
- Dependencies and peer deps
docs/architecture.md — complete rewrite:
- Module structure matching Python powermem layout
- Key flows (create, intelligent add, search)
- Storage backends (SQLite + SeekDB with details)
- CLI and Dashboard descriptions
- Test architecture (6 layers, 7 CI jobs, 8 testing perspectives)
- Python parity mapping table
CHANGELOG.md — v0.3.0 release notes:
- Directory restructure, all new modules, SeekDB improvements
- Test counts (504 total), CI jobs (7, all green)
tests/bdd/README.md — added data correctness scenarios:
- API round-trip, dashboard display, user isolation, data fidelity, pagination1 parent ad4352c commit 3c515da
130 files changed
Lines changed: 6826 additions & 766 deletions
File tree
- .github/workflows
- docs
- src
- agent
- abstract
- components
- factories
- cli
- commands
- utils
- core
- dashboard
- public
- integrations
- embeddings
- config
- llm
- config
- rerank
- config
- intelligence
- prompts
- graph
- server
- storage
- config
- seekdb
- sqlite
- user-memory
- query-rewrite
- storage
- utils
- tests
- bdd
- e2e
- integration
- regression
- unit
- agent
- core
- intelligence
- storage
- user-memory
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | | - | |
38 | | - | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
39 | 150 | | |
40 | 151 | | |
41 | 152 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
3 | 46 | | |
4 | 47 | | |
5 | 48 | | |
| |||
0 commit comments