feat: Support embedded seekdb (#393)

* feat: Add native hybird search (#197)

* add sparse vector embedding

* hybrid search add sparse vector search

* add checking version logic

* add qwen sparse vector

* adjust weight

* update sparse vector function

* update sparse vector function

* fix bug

* fix bug

* optimise function

* optimise function

* optimise function

* optimise function

* optimise function

* optimise function

* optimise function

* fix bug

* add migrate function

* update alembic function

* update alembic function

* update alembic function

* adjust file struct

* update alembic

* update version

* optimise

* fix bug

* update

* update schema update method

* update schema update method

* update schema update method

* update schema update method

* update schema update method

* update migrate method

* update migrate method

* update env.example

* update env.example

* update migrate sparse vector

* update migrate sparse vector

* adjust threshold score logic

* update remark

* add guides and examples

* add benchmark param

* fix bug

* fulltext parsers support

* adjust enable sparse vector setting

* adjust env.example

* adjust docs

* update version

* fix bug

* optimise check

* adjust file construct

* adjust file construct

* add native search

* add file

* remove log

* remove log

* fix bug

* update pyobvector

* add rerank

* adjust

* add limit

* adjust config

* feat: User Profile Support native language output (#198)

* support native language

* support native language

* add docs

* Reconstruct LLM setting (#200)

* feat(llm): enhance configuration management with pydantic-settings

- Introduced a unified configuration system for LLM providers using pydantic-settings.
- Added provider-specific settings for Anthropic, Azure, DeepSeek, Ollama, OpenAI, Qwen, Vllm, and Zai.
- Improved environment variable handling and validation through Field and AliasChoices.
- Removed legacy initialization methods in favor of a cleaner, more maintainable structure.
- Updated LLMFactory to utilize the new provider registration mechanism.

* chore: Update LLM configuration management and improve environment variable handling

- Refactor LLM configuration imports to use BaseLLMConfig.
- Replace direct attribute access with getattr for safer environment variable retrieval.
- Remove deprecated LLMConfig and streamline related code for better maintainability.

* refactor: unify configuration governance for agent, core, and server modules (#199)

* Reconstruct setting in Rerank,Vector,Graph (#202)

* feat(llm): enhance configuration management with pydantic-settings

- Introduced a unified configuration system for LLM providers using pydantic-settings.
- Added provider-specific settings for Anthropic, Azure, DeepSeek, Ollama, OpenAI, Qwen, Vllm, and Zai.
- Improved environment variable handling and validation through Field and AliasChoices.
- Removed legacy initialization methods in favor of a cleaner, more maintainable structure.
- Updated LLMFactory to utilize the new provider registration mechanism.

* chore: Update LLM configuration management and improve environment variable handling

- Refactor LLM configuration imports to use BaseLLMConfig.
- Replace direct attribute access with getattr for safer environment variable retrieval.
- Remove deprecated LLMConfig and streamline related code for better maintainability.

* feat: Enhance rerank configuration and integration

- Introduced BaseRerankConfig for improved configuration management across rerank providers.
- Updated rerank integration files to utilize the new base configuration structure.
- Added support for additional configuration fields such as api_base_url and top_n.
- Refactored rerank factory to accommodate new configuration handling and provider registration.
- Removed deprecated RerankConfig and streamlined related code for better maintainability.
- Updated API request handling in rerank classes to support custom HTTP clients.

* * refactor(powermem): remove unused storage configuration management module

- Removed `VectorStoreConfig` and `GraphStoreConfig` classes
- Deleted associated validation logic and import statements
- Streamlined codebase by eliminating unused components

* feat(powermem): enhance sparse embedder configuration management

- Introduced BaseSparseEmbedderConfig for unified sparse embedding configuration.
- Updated MemoryConfig to utilize BaseSparseEmbedderConfig.
- Refactored SparseEmbedderFactory to support new configuration handling.
- Improved handling of sparse embedder settings across various components.

* feat(powermem): enhance user profile storage with provider registration

- Added a registry mechanism to UserProfileStoreBase for automatic provider registration.
- Implemented class paths for OceanBase and SQLite user profile storage implementations.
- Updated UserProfileStoreFactory to utilize the new registry for provider class retrieval.
- Refactored imports to trigger auto-registration of user profile storage classes.
- Improved handling of provider names in the factory for better compatibility.

* feat(powermem): synchronize embedding model dimensions across configurations

- Added logic to sync `embedding_model_dims` from the embedder to both `vector_store` and `graph_store` if not already set.
- Updated `config_loader.py` and `configs.py` to ensure consistent embedding dimensions across components.

* feat(powermem): enhance OceanBase configuration and query handling

- Added `enable_native_hybrid` field to `OceanBaseConfig` for native hybrid search support.
- Updated query handling in `OceanBaseVectorStore` to use a safe query format, preventing SQL injection risks.

* oceanbase native language case (#220)

* Oceanbase Native Hybrid Search Cases (#223)

* oceanbase native language case

* Oceanbase Native Hybrid Search Cases

* Optimise searching in Intelligent mode And fix SILICONFLOW_LLM_BASE_URL bug (#224)

* Enhance memory operations with background threading support

- Added a global background thread pool for asynchronous memory updates and deletions in the Memory class.
- Updated the handling of memory updates and deletions to submit tasks to the background executor, improving performance and responsiveness.

* format

* Enhance SiliconFlowConfig API key handling

- Updated `SiliconFlowConfig` to improve API key and base URL handling by adding new validation aliases for better compatibility.

* add enable_native_hybrid in benchmark

* Enhance configuration management for OceanBase in config_loader.py

- Added backward compatibility for OceanBase by constructing connection arguments from vector store configuration.
- Updated unit tests to verify the inclusion of internal settings in the configuration.

* disable env file

* Fix unit test issues caused by setting changes (#228)

* Enhance configuration management for OceanBase in config_loader.py

- Added backward compatibility for OceanBase by constructing connection arguments from vector store configuration.
- Updated unit tests to verify the inclusion of internal settings in the configuration.

* disable env file

* Fixed run failure caused by incorrect folder name

* Fixed run failure caused by incorrect folder name (#229)

* Enhance configuration management for OceanBase in config_loader.py

- Added backward compatibility for OceanBase by constructing connection arguments from vector store configuration.
- Updated unit tests to verify the inclusion of internal settings in the configuration.

* disable env file

* Fixed run failure caused by incorrect folder name

* feat: Enhance rerank configuration and integration

- Introduced new RERANKER_* environment variables for improved configuration management across rerank providers.
- Updated .env.example to include new rerank settings and reorganized sections for clarity.
- Refactored rerank integration files to utilize the new base configuration structure, ensuring consistency in API key and base URL handling.
- Updated error messages in rerank classes to reflect the new environment variable naming convention.

* adjust version

* add rerank setting in OceanBaseConfig

* Enhance OceanBase configuration for embedded SeekDB support

- Updated .env.example to clarify connection settings for OceanBase, allowing an empty host for embedded SeekDB.
- Modified config_loader.py to include 'ob_path' in connection arguments for OceanBase.
- Enhanced CLI prompts in config.py to guide users on using embedded SeekDB.
- Updated base.py and oceanbase.py to support 'ob_path' for embedded SeekDB data directory.
- Refactored oceanbase_graph.py and oceanbase.py to handle connections based on host presence, supporting both remote and embedded modes.
- Introduced utility methods in oceanbase_util.py for safe fetching of results from OceanBase.
- Adjusted user_memory and user_profile modules to accommodate new connection parameters.

* Add ensure_embedded_database_exists utility method and update OceanBase classes

- Introduced `ensure_embedded_database_exists` method in `oceanbase_util.py` to verify and create the target database for embedded SeekDB.
- Updated `oceanbase_graph.py`, `oceanbase.py`, and `user_profile.py` to call this method, ensuring the database exists before establishing connections.
- Enhanced error handling and logging for database creation failures.

* Update Python version requirements to 3.11 across project files

- Modified `pyproject.toml` to require Python 3.11 and updated target versions for tools like Black and Mypy.
- Updated README files in English, Chinese, and Japanese to reflect the new Python version.
- Adjusted GitHub workflows to use Python 3.11 in build, publish, regression, and test configurations.
- Revised documentation and examples to specify Python 3.11 as the minimum requirement.

* Update database provider in .env.example from sqlite to oceanbase

* Update OceanBase index type to HNSW and enhance handling for embedded SeekDB

- Changed the default index type in .env.example from IVF_FLAT to HNSW.
- Added logic in oceanbase.py to automatically switch to HNSW for embedded SeekDB when using IVF-family indexes on small datasets to prevent crashes.
- Improved row mapping to support both SQLAlchemy Row objects and plain dicts, ensuring compatibility and stability during data fetching.

* fix query bug

* Enhance memory update and delete operations for embedded SeekDB support

- Added logic to handle updates and deletes synchronously for embedded SeekDB to prevent crashes due to concurrent connections.
- Updated the memory management code to check if the storage is an embedded store and adjust the execution of update and delete operations accordingly.

* Add limitation note for AsyncMemory with embedded SeekDB

- Documented that AsyncMemory cannot be used with embedded SeekDB due to its single-threaded C++ engine, which leads to crashes from concurrent access.
- Provided code examples illustrating the correct usage of Memory class for embedded SeekDB and the supported use of AsyncMemory with remote OceanBase.

* add pyseekdb dependents

* Refactor update method in OceanBaseVectorStore to improve payload merging and preserve existing fields

- Updated the update method to fetch all existing columns, ensuring that partial payloads do not overwrite existing data.
- Enhanced the merging logic for payloads, allowing new keys to override existing ones while preserving necessary fields.
- Improved handling of sparse embeddings to prevent accidental data loss during updates.
- Streamlined the preparation of update data for better clarity and maintainability.

* Update version to 1.0.3 in project files

- Incremented version number in pyproject.toml, version.py, and audit/telemetry modules to reflect the new release.
- Ensured consistency across all relevant files for the updated version.

* Implement service singletons and error handling in FastAPI application

- Added an async context manager for initializing and cleaning up service singletons at application startup and shutdown.
- Updated service retrieval functions in API routes to fetch services from the app state, with error handling for unavailable services.
- Introduced logging for service initialization and shutdown processes, enhancing observability of the application lifecycle.
- Added a utility function to check for embedded storage in the CLI server configuration, ensuring proper worker settings for embedded databases.

* Update version to 1.1.0 across project files

- Incremented version number in pyproject.toml, version.py, and relevant modules to reflect the new release.
- Ensured consistency in versioning across all affected files for the updated version.

---------

Co-authored-by: jingshun.tq <35712518+Teingi@users.noreply.github.com>
Co-authored-by: Chifang <40140008+Ripcord55@users.noreply.github.com>

Author: 3 people

.env.example

@@ -14,7 +14,7 @@ TIMEZONE=Asia/Shanghai
 # 1. Database Configuration (Required)
 # =============================================================================
 # Choose your database provider: sqlite, oceanbase, postgres
-DATABASE_PROVIDER=sqlite
+DATABASE_PROVIDER=oceanbase
 
 # -----------------------------------------------------------------------------
 # SQLite Configuration (Default - Recommended for development)
@@ -27,15 +27,17 @@ SQLITE_COLLECTION=memories
 # -----------------------------------------------------------------------------
 # OceanBase Configuration
 # -----------------------------------------------------------------------------
-OCEANBASE_HOST=127.0.0.1
+# Connection mode: set OCEANBASE_HOST for remote, leave empty for embedded SeekDB
+OCEANBASE_HOST=
+OCEANBASE_PATH=./seekdb_data
 OCEANBASE_PORT=2881
 OCEANBASE_USER=root@sys
 OCEANBASE_PASSWORD=your_password
 OCEANBASE_DATABASE=powermem
 OCEANBASE_COLLECTION=memories
 
 ## Keep the default settings, as modifications are generally not needed.
-OCEANBASE_INDEX_TYPE=IVF_FLAT
+OCEANBASE_INDEX_TYPE=HNSW
 OCEANBASE_VECTOR_METRIC_TYPE=cosine
 OCEANBASE_TEXT_FIELD=document
 OCEANBASE_VECTOR_FIELD=embedding

.github/workflows/build.yml

@@ -12,10 +12,9 @@ on:
       python_version:
         description: 'Python version to build with'
         required: false
-        default: '3.10'
+        default: '3.11'
         type: choice
         options:
-          - '3.10'
           - '3.11'
           - '3.12'
 
@@ -53,7 +52,7 @@ jobs:
     needs: build-dashboard
     strategy:
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.11", "3.12"]
       fail-fast: false
 
     steps:

.github/workflows/publish.yml

@@ -36,7 +36,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
-          python-version: "3.10"
+          python-version: "3.11"
 
       - name: Inject Frontend and Build Package
         run: |

.github/workflows/regression.yml

@@ -36,7 +36,7 @@ jobs:
     if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' || github.event_name == 'schedule' || (github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.full_name == github.repository)
     strategy:
       matrix:
-        python-version: ["3.10"]
+        python-version: ["3.11"]
       fail-fast: false
 
     steps:
@@ -174,7 +174,7 @@ jobs:
     if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' || github.event_name == 'schedule' || (github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.full_name == github.repository)
     strategy:
       matrix:
-        python-version: ["3.10"]
+        python-version: ["3.11"]
       fail-fast: false
 
     steps:

.github/workflows/test.yml

@@ -26,7 +26,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.11", "3.12"]
       fail-fast: false
 
     steps:

README.md

@@ -27,8 +27,8 @@ One command to add PowerMem memory to OpenClaw: `openclaw plugins install memory
     <a href="https://github.com/oceanbase/powermem/blob/master/LICENSE">
         <img alt="license" src="https://img.shields.io/badge/license-Apache%202.0-green.svg" />
     </a>
-    <a href="https://img.shields.io/badge/python%20-3.10.0%2B-blue.svg">
-        <img alt="pyversions" src="https://img.shields.io/badge/python%20-3.10.0%2B-blue.svg" />
+    <a href="https://img.shields.io/badge/python%20-3.11.0%2B-blue.svg">
+        <img alt="pyversions" src="https://img.shields.io/badge/python%20-3.11.0%2B-blue.svg" />
     </a>
     <a href="https://deepwiki.com/oceanbase/powermem">
         <img alt="Ask DeepWiki" src="https://deepwiki.com/badge.svg" />

README_CN.md

@@ -27,8 +27,8 @@
     <a href="https://github.com/oceanbase/powermem/blob/master/LICENSE">
         <img alt="license" src="https://img.shields.io/badge/license-Apache%202.0-green.svg" />
     </a>
-    <a href="https://img.shields.io/badge/python%20-3.10.0%2B-blue.svg">
-        <img alt="pyversions" src="https://img.shields.io/badge/python%20-3.10.0%2B-blue.svg" />
+    <a href="https://img.shields.io/badge/python%20-3.11.0%2B-blue.svg">
+        <img alt="pyversions" src="https://img.shields.io/badge/python%20-3.11.0%2B-blue.svg" />
     </a>
     <a href="https://deepwiki.com/oceanbase/powermem">
         <img alt="Ask DeepWiki" src="https://deepwiki.com/badge.svg" />

README_JP.md

@@ -27,8 +27,8 @@
     <a href="https://github.com/oceanbase/powermem/blob/master/LICENSE">
         <img alt="license" src="https://img.shields.io/badge/license-Apache%202.0-green.svg" />
     </a>
-    <a href="https://img.shields.io/badge/python%20-3.10.0%2B-blue.svg">
-        <img alt="pyversions" src="https://img.shields.io/badge/python%20-3.10.0%2B-blue.svg" />
+    <a href="https://img.shields.io/badge/python%20-3.11.0%2B-blue.svg">
+        <img alt="pyversions" src="https://img.shields.io/badge/python%20-3.11.0%2B-blue.svg" />
     </a>
     <a href="https://deepwiki.com/oceanbase/powermem">
         <img alt="Ask DeepWiki" src="https://deepwiki.com/badge.svg" />

benchmark/server/main.py

@@ -153,7 +153,7 @@ def create(self, *args: Any, **kwargs: Any) -> Any:
 app = FastAPI(
     title="PowerMem Benchmark REST APIs",
     description="A REST API for managing and searching memories for benchmark testing scenarios.",
-    version="1.0.0",
+    version="1.1.0",
     docs_url="/docs",
     redoc_url="/redoc",
 )

docs/api/0002-async_memory.md

@@ -263,16 +263,36 @@ async def batch_process():
 asyncio.run(batch_process())
 ```
 
+### Limitation: Embedded SeekDB Does Not Support Async
+
+Embedded SeekDB (local file mode with no `host` configured) uses a single-threaded C++ engine that **does not support concurrent multi-threaded access**. `AsyncMemory` internally submits synchronous operations to a `ThreadPoolExecutor`, which causes multiple threads to read and write the same embedded SeekDB instance simultaneously. This leads to C++-level crashes such as `pure virtual method called` or `Segmentation fault`.
+
+**`AsyncMemory` cannot be used with embedded SeekDB.** Use the synchronous `Memory` class instead.
+
+```python
+# ❌ Not supported with embedded SeekDB
+from powermem import AsyncMemory
+async_memory = AsyncMemory(config=embedded_seekdb_config)  # crashes
+
+# ✓ Use the synchronous interface with embedded SeekDB
+from powermem import Memory
+memory = Memory(config=embedded_seekdb_config)
+```
+
+Remote OceanBase (with `host` configured) is not affected by this limitation and fully supports `AsyncMemory`.
+
 ### When to Use AsyncMemory
 
 Use `AsyncMemory` when:
 - Processing many memories concurrently
 - Building async web applications (FastAPI, aiohttp)
 - Implementing batch processing pipelines
 - Need non-blocking memory operations
+- Using **remote OceanBase** (with `host` configured)
 
 Use `Memory` when:
 - Simple synchronous scripts
 - Interactive notebooks
 - Simple use cases without concurrency needs
+- Using **embedded SeekDB** (local file mode, no `host`)