Support for multiple indices

.editorconfig

@@ -0,0 +1,9 @@
+root = true
+[*]
+indent_style = space
+tab_width = 2
+indent_size = 2
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true

README.md

@@ -14,14 +14,112 @@ or using `docker`:
 docker run -it --rm -p 3000:3000 pqnet/minivec
 ```
 
-add new documents to the store with a HTTP POST:
+## Creating Indices
+
+Minivec now supports multiple indices for different document properties. First, create an index:
+
+```bash
+# Create default index for document content
+curl -X POST -H "Content-Type: application/json" -d '{"name":"default","indexedPropertyPath":"$.content","description":"Default content index"}' localhost:3000/api/indices
+
+# Create another index for document titles
+curl -X POST -H "Content-Type: application/json" -d '{"name":"title-index","indexedPropertyPath":"$.title","description":"Index for document titles"}' localhost:3000/api/indices
+```
+
+## Adding Documents
+
+Add new documents to the store with a HTTP POST. Minivec will automatically detect and use all applicable indices for your documents:
+
+```bash
+# Add a document - will automatically be indexed in all applicable indices
+curl -X POST -H "Content-Type: application/json" -d '{
+  "documents": [
+    {
+      "content": "Artificial intelligence is transforming industries across the globe.",
+      "title": "AI Revolution",
+      "metadata": { "tags": ["ai", "technology"] }
+    }
+  ]
+}' localhost:3000/api/documents
+```
+
+You can also specify which indices you want to use (any non-applicable indices will be ignored):
+
+```bash
+# Index document only in the title index (if applicable)
+curl -X POST -H "Content-Type: application/json" -d '{
+  "documents": [
+    {
+      "content": "Artificial intelligence is transforming industries across the globe.",
+      "title": "AI Revolution",
+      "metadata": { "tags": ["ai", "technology"] }
+    }
+  ],
+  "indices": ["title-index"]
+}' localhost:3000/api/documents
+```
+
+The response includes a summary of which indices were used:
+
+```json
+{
+  "success": true,
+  "count": 1,
+  "indices": "default (1/1), title-index (1/1)"
+}
+```
+
+## Document Structure
+
+Documents in Minivec have flexible JSON structures. The only requirement is that the property referenced by the index's `indexedPropertyPath` must exist as a string when you want to index the document.
+
+Examples of valid documents:
+
+```json
+// Simple document with content property
+{ "content": "This is the document content" }
+
+// Document with nested properties
+{
+  "title": "My Document",
+  "details": {
+    "content": "This is the document content",
+    "author": "John Doe"
+  },
+  "metadata": {
+    "tags": ["example", "documentation"]
+  }
+}
+```
+
+You can create indices with appropriate JSON paths like `$.content`, `$.title`, or `$.details.content` to target different properties in your documents.
+
+## Searching Documents
+
+Search using HTTP GET, specifying which index to use:
+
 ```bash
-curl -H "Content-Type: application/json" -d  '{ "documents": [{ "content": "hello world", "metadata":{} }]}' localhost:3000/api/documents
+# Search using the default content index
+curl 'localhost:3000/api/documents?q=artificial%20intelligence'
+
+# Search using the title index
+curl 'localhost:3000/api/documents?q=revolution&index=title-index'
 ```
 
-And search them using HTTP GET:
+## Listing Available Indices
+
+List all available indices:
+
 ```bash
-curl 'localhost:3000/api/documents?q=hello'
+curl 'localhost:3000/api/indices'
+```
+
+## Building Indices for Existing Documents
+
+If you add new indices after inserting documents, you can build embeddings for existing documents:
+
+```bash
+curl -X POST 'localhost:3000/api/indices/title-index/build'
 ```
 
 ## Persistence
@@ -39,6 +137,7 @@ docker run -it --rm -p 3000:3000 -v minivec-models-cache:/models -v minivec-data
 
 ## Configuration
 Use environment variables to configure which models to load. see [nitro.config.ts](nitro.config.ts) for a full list of the usable variables
+
 ### Model choice
 `bge-m3` (for embedding) and `bge-reranker-v2-m3` (for reranking) are automatically downloaded and used by the container.
 It is possible to choose different models by specifying a local file name, an http/https URL or an huggingface repository to download the models automatically.

nitro.config.ts

@@ -6,6 +6,12 @@ export default defineNitroConfig({
   experimental: {
     database: true,
   },
+  database: {
+    default: {
+      connector: "better-sqlite3",
+      options: {},
+    },
+  },
   runtimeConfig: {
     disableWrite: false,
     localModels: {

package.json

@@ -4,17 +4,19 @@
   "description": "",
   "type": "module",
   "scripts": {
+    "test": "tsx --test ./test/**/*.test.ts",
+    "test:integration": "node ./test/api-integration.test.js",
     "dev": "nitro dev",
     "build": "nitro build",
     "build-container": "podman build . -t docker.io/pqnet/minivec:latest"
   },
   "keywords": [],
   "author": "",
   "license": "ISC",
-  "packageManager": "pnpm@10.4.1",
+  "packageManager": "pnpm@10.6.3+sha512.bb45e34d50a9a76e858a95837301bfb6bd6d35aea2c5d52094fa497a467c43f5c440103ce2511e9e0a2f89c3d6071baac3358fc68ac6fb75e2ceb3d2736065e6",
   "devDependencies": {
     "@types/better-sqlite3": "^7.6.12",
-    "nitropack": "^2.10.4",
+    "nitropack": "^2.11.6",
     "tsx": "^4.19.3"
   },
   "pnpm": {
@@ -38,8 +40,8 @@
     ]
   },
   "dependencies": {
-    "better-sqlite3": "^11.8.1",
-    "db0": "^0.2.4",
+    "better-sqlite3": "^11.9.0",
+    "db0": "^0.3.1",
     "node-llama-cpp": "^3.6.0",
     "sqlite-vec": "0.1.7-alpha.2"
   }