This document describes the API exposed by the Otzaria Search Engine through Flutter/Dart bindings. The API is generated from Rust code using flutter_rust_bridge.
Main search engine for full-text search with regex support.
SearchEngine.new(String path)Synchronous constructor that creates a new search engine instance.
Parameters:
path(String): File system path where the search index will be stored
Returns: SearchEngine instance
Future<void> addDocument(
int id,
String title,
String reference,
String topics,
String text,
int segment,
bool isPdf,
String filePath
)Adds a document to the search index.
Parameters:
id(int/u64): Unique document identifiertitle(String): Document titlereference(String): Document reference/citationtopics(String): Faceted topics (hierarchical, e.g., "category/subcategory")text(String): Full document text to be indexedsegment(int/u64): Segment number within the documentisPdf(bool): Whether the document is a PDFfilePath(String): File path to the document
Returns: Future
Future<void> commit()Commits all pending document additions to the index. Must be called after adding documents to make them searchable.
Returns: Future
Future<List<SearchResult>> search({
required List<String> regexTerms,
required List<String> facets,
required int limit,
required int offset,
required int slop,
required int maxExpansions,
required ResultsOrder order,
HighlightConfig? highlight,
})Performs a search query on the index using regex patterns.
Parameters:
regexTerms(List): List of regex patterns to search for- Single term: matching index terms are materialized (capped at
maxExpansions) - Multiple terms: uses RegexPhraseQuery with specified slop
- Single term: matching index terms are materialized (capped at
facets(List): List of topic facets to filter by (empty list = no facet filter)limit(int/u32): Maximum number of results to returnoffset(int/u32): Number of leading results to skip (pagination)slop(int/u32): Maximum distance between terms in phrase queries (for multi-term searches)maxExpansions(int/u32): Regex-expansion ceiling. A single term truncates its term collection at the ceiling (partial results, flagged via the status-bearing variants). A multi-term phrase checks Tantivy's cumulative expansions independently in every segment: when every segment fits, the exactRegexPhraseQueryruns; otherwise the engine falls back to a term-list phrase built from per-position materialized term sets (per-position caps, truncation flagged) — never an errororder(ResultsOrder): Sort order for results (Catalogue or Relevance)highlight(HighlightConfig?, optional): Snippet/highlight configuration; defaults to<font color=red>tags and 800 chars
Returns: Future<List>
Variants:
searchAndCount(...)→SearchPageResult— same arguments, returns the total hit count alongside the page in a single index passsearchStream(..., chunkSize)→Stream<List<SearchResult>>— emits results in chunks as snippets are builtcount(regexTerms, facets, slop, maxExpansions)→int— count only
Higher-level APIs that take a raw query string and build the query in Rust:
// Exact: term/phrase match after nikud-stripping + tokenization. Fastest.
Future<List<SearchResult>> searchExact({query, facets, limit, offset, order})
// Fuzzy: Levenshtein matching per token (maxDistance edits, 0–2).
Future<List<SearchResult>> searchFuzzy({query, facets, limit, offset, maxDistance, order})
// Advanced: Hebrew morphological query builder (prefixes, suffixes,
// full/deficient spelling, typo tolerance, alternative words, custom spacing).
Future<List<SearchResult>> searchAdvanced({
query, facets, limit, offset, distance,
customSpacing, alternativeWords, searchOptions, order,
wordMatchMode, wordMatchCount,
})wordMatchMode (optional, advanced family only) relaxes the all-words
requirement: all (default — current behavior), anyWord, mostWords
(n/2+1), or atLeast together with wordMatchCount (clamped to
[1, n]). Any mode other than all drops the order/distance requirement:
wordDistance scope behaves like sameParagraph, and sameSection
requires the section to carry at least the required number of distinct
query words. Duplicate query words count once toward the threshold.
The negative query always requires all of its words.
Each mode also provides count*, searchAndCount* and search*Stream
variants (countExact, searchAndCountFuzzy, searchAdvancedStream, …).
Queries are normalized like the index (nikud stripped, lowercased); empty
queries return no results. Advanced-mode highlighting wraps every
morphological variant that actually matched, not just the literal words;
fuzzy-mode highlighting likewise wraps every term within the requested edit
distance, not just exact occurrences of the query words.
Future<void> addDocumentsBatch({required List<DocumentInput> docs}) // bulk add, no commit
Future<void> upsertDocument({...}) // delete-by-id + re-insert, no commit
Future<void> upsertDocumentsBatch({required List<DocumentInput> docs})
Future<void> deleteDocumentById({required BigInt id})
Future<void> rollback() // discard writes since last commit
Future<void> optimize() // commit pending + merge all segments
Future<BigInt> getDocumentCount()
Future<int> getSegmentCount()
Future<List<FacetCount>> getFacetCounts({...}) // per-child facet counts for a prefix
Future<Map<String, int>> countByBook({...}) // per-filePath hit counts for a queryFuture<void> clear()Removes all documents from the index.
Returns: Future
Future<void> removeDocumentsByTitle(String title)Removes all documents with the specified title from the index.
Parameters:
title(String): Title of documents to remove
Returns: Future
Future<Map<String, int>> countDocumentsByFilePath()Returns the number of live (committed, non-deleted) documents per distinct filePath across the whole index.
The result is read from the index itself rather than from any external state, so callers can reconstruct indexing progress directly from an index — e.g. after pointing the engine at a directory that already contains an index built elsewhere — and compare it against the current library to decide whether re-indexing is needed.
Returns: Future<Map<String, int>> - Map from filePath to its live document count
Future<List<String>> getIndexedFilePaths()Returns the distinct filePath values present in the index — i.e. which books have at least one live document. Convenience wrapper over countDocumentsByFilePath().
Returns: Future<List> - List of indexed file paths (unordered)
IndexCompatibility checkIndexCompatibility({required String path})Checks whether an existing index is compatible with the current search engine schema.
The engine writes an otzaria_index_meta.json sidecar file next to compatible indexes when they are opened. For older indexes without that sidecar, this function falls back to Tantivy's meta.json and verifies the current required schema shape.
Parameters:
path(String): File system path of the Tantivy index directory
Returns: IndexCompatibility
Common status values:
compatible: Otzaria metadata exists and matches the current schema versionlegacy_compatible: Otzaria metadata is missing, but the full Tantivy schema matches the current enginerebuild_required: The index schema is older or incompatible and should be rebuiltengine_too_old: The index schema is newer than this engine supportsmissing_index: The index directory does not existinvalid_index_path: The given path is not a valid directory path
Result returned from SearchEngine.search()
Fields:
class SearchResult {
String title; // Document title
String reference; // Document reference/citation
String text; // Highlighted snippet or full text
int id; // Document ID (u64)
int segment; // Segment number (u64)
bool isPdf; // Whether document is PDF
String filePath; // Path to document file
}Note: The text field contains a snippet with HTML highlighting when matches are found. Highlights are wrapped in <font color=red>...</font> tags by default (configurable via HighlightConfig). If no snippet is generated, it contains the full document text.
Result returned from the searchAndCount* family.
Fields:
class SearchPageResult {
List<SearchResult> results; // The requested page
int totalCount; // Total hits for the query
}Optional snippet/highlight configuration accepted by search, searchAndCount, searchStream and searchFuzzyTerms.
Fields:
class HighlightConfig {
String highlightPrefix; // default: "<font color=red>"
String highlightPostfix; // default: "</font>"
int maxChars; // snippet length budget, default: 800
}Result returned from checkIndexCompatibility().
Fields:
class IndexCompatibility {
bool compatible; // Whether the current engine can use this index
String status; // Machine-readable status
int? foundSchemaVersion; // Version found in metadata, when known
int requiredSchemaVersion; // Version required by this engine
String engineVersion; // Rust engine package version
String metadataPath; // Expected otzaria_index_meta.json path
String? reason; // Human-readable detail for non-trivial states
}Compatibility is controlled by requiredSchemaVersion, not by the package release number. A patch release can keep the same schema version when no rebuild is required.
Enum specifying the sort order for search results.
Values:
enum ResultsOrder {
Catalogue, // Sort by document ID (ascending)
Relevance // Sort by search relevance score (descending)
}// Initialize search engine
final searchEngine = SearchEngine.new('/path/to/index');
// Add documents
await searchEngine.addDocument(
1,
'Example Book',
'Chapter 1',
'category/subcategory',
'This is the full text content to be indexed',
1,
false,
'/path/to/book.txt'
);
await searchEngine.commit();
// Search with regex
final results = await searchEngine.search(
['text', 'content'], // Search for these terms
['category'], // Filter by topic
10, // Limit to 10 results
2, // Allow 2 words between terms
1000, // Max regex expansions
ResultsOrder.Relevance
);
// Count matching documents
final count = await searchEngine.count(
['text'],
['category'],
0,
1000
);The SearchEngine uses Rust regex syntax. Common patterns:
.- matches any character.*- matches any sequence\w+- matches word characters[אבגד]- character class (matches any of these Hebrew letters)(pattern1|pattern2)- alternation
Topics use hierarchical facet notation with forward slashes:
"category"- top level"category/subcategory"- nested"category/subcategory/item"- deep nesting
Documents match a facet query if their topic starts with any of the specified facets.
Alongside the category path, a document can carry extra facet values supplied at
indexing time (extraFacets on addTextBook/addTextBookBytes/addPdfBook/
addDocument/upsertDocument, or DocumentInput.extraFacets). Three English
roots are reserved as filter dimensions (they cannot collide with Hebrew
category names): /author/<name>, /era/<period>, /base (foundational books).
Filtering semantics (facets parameter, all search/count functions): the facet
list is partitioned by root — each reserved dimension is its own group, and all
other paths (the category tree) form one group. Within a group facets are OR'ed
(prefix match, as before); across groups they are AND'ed. So
["/תנך", "/era/ראשונים"] = "in תנ"ך AND by a rishon", while
["/era/ראשונים", "/era/אחרונים"] = "either era". A call with only category
facets behaves exactly as before.
The exact/advanced/fuzzy search*, searchAndCount*, search*Stream and
search*StreamWithCounts functions accept an optional grouping parameter
(ResultGrouping?, default null = flat results):
sameSection— hits under the same heading in the same book (samesectionId; a PDF page is a section) collapse into one result with a count.identicalText— lines whose Hebrew letters are identical (punctuation, spacing and quotes ignored; seelineHash) collapse across books. Lines shorter than 12 Hebrew letters never merge.
When grouping is active: limit/offset count groups; each returned
SearchResult is the group's best representative (by the requested order) and
carries mergedCount (total group size) plus merged (up to 10 sibling
locations — title/reference/id/segment/isPdf/filePath, no snippet).
SearchPageResult.groupCount / the first SearchStreamUpdate.groupCount report
the total number of groups; totalCount and bookCounts remain raw hit counts.
The search engine stores its index on disk at the specified path. The index persists between application runs and can be reused without rebuilding.
SearchEngine is designed to be used from a single thread. If you need concurrent access, create separate instances or implement your own locking mechanism.
The search engine uses memory-mapped files (MmapDirectory) for efficient index access. The index writer is initialized with a 50MB buffer (50_000_000 bytes).
When implementing this API in other languages:
- Index Format: Use Tantivy-compatible index format or implement a translation layer
- Regex Engine: Ensure regex engine supports similar syntax to Rust's regex crate
- Field Types: Map types appropriately:
u64→ unsigned 64-bit integerString→ UTF-8 stringbool→ boolean
- Snippet Highlighting: Implement HTML snippet generation with configurable highlight tags
- Facet Structure: Implement hierarchical facet matching with
/delimiter - Async/Sync: Constructor is synchronous, all other methods are asynchronous
This documentation is based on the current implementation as of the latest commit. Check the repository for updates and changes to the API.