-
Notifications
You must be signed in to change notification settings - Fork 174
Public APIs
The examples of ForestDB APIs described in this page are in the C language. The basic indexing unit of ForestDB is a document that consists of key, metadata, and body, each of which is an arbitrary length* sequence of uninterpreted bytes. Documents are ordered by key (defaulting to a simple lexicographic sort, like memcmp) and can be iterated in that order.
* The current implementation limits the length of keys to 3840 bytes, but this will be relaxed in the future.
* You can find the old APIs used in the 1.0 beta at the Public APIs (1.0 Beta) page.
Each DB instance has a unique name, that is used as a prefix of path-names for corresponding DB files. A DB instance contains more than one KV store, which is an actual database instance where documents are actually stored.
The following example shows how to create and open an DB instance and open the default KV store from the DB file handle. Note that the default configuration creates a new DB file if it does not exist.
#include <assert.h>
#include "libforestdb/forestdb.h"
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_status status;
fdb_config config;
fdb_kvs_config kvs_config;
config = fdb_get_default_config();
kvs_config = fdb_get_default_kvs_config();
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
assert(status == FDB_RESULT_SUCCESS);
status = fdb_kvs_open(dbfile, &db, NULL, &kvs_config);
assert(status == FDB_RESULT_SUCCESS);
...When NULL is passed as KV store name, then it creates a new KV store instance named "default". Note that there is an alternative API fdb_kvs_open_default for opening the default KV store, so that you can use it as follows:
...
status = fdb_kvs_open_default(dbfile, &db, &kvs_config);
assert(status == FDB_RESULT_SUCCESS);
...Almost all ForestDB API calls return their results as fdb_status type. The return value will be FDB_RESULT_SUCCESS if the call succeeded. Otherwise, the value has an error number defined in fdb_errors.h.
ForestDB provides several configuration options via two types: fdb_config and fdb_kvs_config, for DB file instance and KV store instance, respectively. You can get the default configuration by calling fdb_get_default_config and fdb_get_default_kvs_config APIs, where the details are described in fdb_types.h.
Note that an ForestDB file handle (fdb_file_handle) and KV store handle (fdb_handle) cannot be used simultaneously by two or more threads, which means that each thread should have its own handles. As some languages (e.g., Golang) abstract threading, we recommend them to use the pooling for readers and writers.
ForestDB supports multiple KV stores in a DB instance, and you can create new empty KV stores using the fdb_kvs_open API. The example below shows how to create a new KV store in the given DB instance:
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_status status;
fdb_kvs_config kvs_config;
...
kvs_config = fdb_get_default_kvs_config();
status = fdb_kvs_open(dbfile, &db, "KV store name", &kvs_config);
assert(status == FDB_RESULT_SUCCESS);
...ForestDB’s block cache is an alternative to the OS page cache, whose default size is 128 MB. You can change the size in bytes, or disable the cache by setting the size to zero:
...
fdb_config config;
config = fdb_get_default_config();
config.buffercache_size = 1073741824; // 1 GB
......
fdb_config config;
config = fdb_get_default_config();
config.buffercache_size = 0; // disable
...The block cache is globally shared across all active DB instances, thus buffercache_size will be ignored if the block cache is already initialized by the previously called fdb_open API.
If you want to close a KV store handle, call fdb_kvs_close as follows:
fdb_handle *db;
fdb_status status;
...
status = fdb_kvs_close(db);
assert(status == FDB_RESULT_SUCCESS);fdb_close API is used for closing a DB file handle. All opened KV store handles that belong to the DB file handle are closed together with the fdb_close API call:
fdb_file_handle *dbfile;
fdb_handle *db_default, *db1, *db2;
fdb_kvs_config kvs_config;
fdb_status status;
...
status = fdb_kvs_open_default(dbfile, &db_default, &kvs_config);
status = fdb_kvs_open(dbfile, &db1, "First DB", &kvs_config);
status = fdb_kvs_open(dbfile, &db2, "Second DB", &kvs_config);
...
// This call also closes db_default, db1, and db2 handles.
status = fdb_close(dbfile);
assert(status == FDB_RESULT_SUCCESS);Note that global resources such as the block cache and file management data still reside in memory for fast restart, even after closing the DB file handle. If you want to completely terminate the ForestDB library, call fdb_shutdown as follows:
fdb_status status;
...
status = fdb_shutdown();
assert(status == FDB_RESULT_SUCCESS);As we mentioned, ForestDB performs indexing on a per-document basis. To create an in-memory document, call fdb_doc_create. (This does not add the document to the database; it just allocates memory for the fdb_doc structure and its contents.)
...
fdb_doc *doc;
fdb_status status;
status = fdb_doc_create(&doc, "foo", 3, "00", 2, "bar", 3);
assert(status == FDB_RESULT_SUCCESS);
...Note that any field can be empty:
fdb_doc *doc1, *doc2, *doc3;
fdb_status status;
status = fdb_doc_create(&doc1, "foo", 3, NULL, 0, NULL, 0);
status = fdb_doc_create(&doc2, NULL, 0, NULL, 0, NULL, 0);
status = fdb_doc_create(&doc3, "bar", 3, NULL, 0, "baz", 3);If you want to change the metadata or body of a document in memory, call fdb_doc_update. (Again, this does not affect the database; it just updates the in-memory values stored in the fdb_doc.)
...
fdb_doc *doc;
fdb_status status;
status = fdb_doc_create(&doc, "foo", 3, "00", 2, "bar", 3);
assert(status == FDB_RESULT_SUCCESS);
status = fdb_doc_update(&doc, "11", 2, "baz", 3);
assert(status == FDB_RESULT_SUCCESS);
...This example modifies the metadata and body of the document doc from 00 and bar to 11 and baz, respectively.
To free an fdb_doc structure and the data it stores, just call fdb_doc_free:
fdb_doc *doc;
fdb_status status;
...
status = fdb_doc_free(doc);
assert(status == FDB_RESULT_SUCCESS);The following example shows how to insert a document into a KV store by calling fdb_set:
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_doc *doc;
fdb_status status;
...
status = fdb_doc_create(&doc, "foo", 3, "00", 2, "bar", 3);
status = fdb_set(db, doc);
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
status = fdb_doc_free(doc);
...If a document with the same key already exists, ForestDB overwrites it with the new document.
After a successful fdb_set the fdb_doc can be freed if desired, since its contents have now been copied into the database.
After making one or more changes to the database, you must invoke fdb_commit to commit the updates to disk. Without this, the document would be lost after closing the database handle or upon a system failure. fdb_commit writes any pending in-memory data, updates the tree structures, adds a new header at the end of the file, and calls fsync to flush all dirty data into the corresponding DB file.
If you want to write data to disk asynchronously, ForestDB supports an option for asynchronous commit mode in fdb_config. The following example shows how to write a document and commit the update asynchronously. Note that fdb_commit should be called even though the DB instance is opened with asynchronous option.
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_doc *doc;
fdb_status status;
fdb_config config;
config = fdb_get_default_config();
config.durability_opt = FDB_DRB_ASYNC;
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
...
status = fdb_doc_create(&doc, "foo", 3, "00", 2, "bar", 3);
status = fdb_set(db, doc);
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
status = fdb_doc_free(doc);
...To commit more than one document update at once, call fdb_set for each document and then invoke fdb_commit at the end. Note that updates across multiple KV stores can be batched. :
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_doc *doc1, *doc2, *doc3;
fdb_status status;
...
status = fdb_set(db, doc1);
status = fdb_set(db, doc2);
status = fdb_set(db, doc3);
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
...If you want to add large numbers of documents at once, the wal_flush_before_commit option in fdb_config is useful. This option periodically flushes WAL entries into the on-disk tree before fdb_commit API is called, in order to restrict the overall memory consumption by WAL indexes. fdb_commit only has to be invoked once at the end of the bulk load.
Note that wal_flush_before_commit option is enabled by default.
fdb_file_handle *dbfile;
fdb_status status;
fdb_config config;
config = fdb_get_default_config();
config.wal_flush_before_commit = true;
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
... (do bulk write) ...
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
...Documents can be stored as a compressed form when they are written in the disk. ForestDB provides a compression option in fdb_config, that compresses and decompresses the body of documents using the Snappy library. Snappy does not provide optimal compression, but it's very fast. You can enable the option as follows:
fdb_file_handle *dbfile;
fdb_status status;
fdb_config config;
config = fdb_get_default_config();
config.compress_document_body = true;
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
...ForestDB provides two kinds of retrieval methods: retrieval by key and retrieval by sequence number. If you want to get a document using its key, create a document with empty meta and body fields, and call fdb_get:
fdb_handle *db;
fdb_doc *doc;
fdb_status status;
...
status = fdb_doc_create(&doc, "foo", 3, NULL, 0, NULL, 0);
status = fdb_get(db, doc);
...
status = fdb_doc_free(doc);
...If the target document is found, then the meta and body fields of doc will be filled with those of the corresponding document.
A document can also be retrieved by its sequence number, which is a value assigned to it whenever it's created or updated, taken from a monotonically-increasing sequence counter maintained by the database. It is a 64-bit integer defined as the fdb_seqnum_t type, and available as the seqnum field of an fdb_doc after it's been saved by fdb_set.
To get a document by its sequence number, create an empty document, assign a sequence number to it, and call fdb_get_byseq:
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_doc *doc1, *doc2;
fdb_status status;
fdb_seqnum_t seqnum;
...
status = fdb_doc_create(&doc1, "foo", 3, "00", 2, "bar", 3);
status = fdb_set(db, doc1);
seqnum = doc1->seqnum;
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
status = fdb_doc_free(doc1);
...
status = fdb_doc_create(&doc2, NULL, 0, NULL, 0, NULL, 0);
doc2->seqnum = seqnum;
status = fdb_get_byseq(db, doc2);
...
status = fdb_doc_free(doc2);
...To read a document's metadata only, without its body, ForestDB provides two operations: fdb_get_metaonly and fdb_get_metaonly_byseq. Their usage is same as those of fdb_get and fdb_get_byseq:
fdb_handle *db;
fdb_doc *doc;
fdb_status status;
...
status = fdb_doc_create(&doc, "foo", 3, NULL, 0, NULL, 0);
status = fdb_get_metaonly(db, doc);
...fdb_handle *db;
fdb_doc *doc;
fdb_status status;
fdb_seqnum_t seqnum;
...
status = fdb_doc_create(&doc, NULL, 0, NULL, 0, NULL, 0);
doc->seqnum = seqnum;
status = fdb_get_metaonly_byseq(db, doc);
...In the case of fdb_get_metaonly_byseq, both key and metadata are returned by calling the API.
After the call returns, the length of the body is available in the bodylen field, even though the body field is NULL.
Afterwards, the document's body can be read very efficiently by calling fdb_get_byoffset, as described in the next section.
Documents also can be retrieved by their disk offset, which is literally the byte offset from the start of the database file to the position where the document is stored. This is very efficient because it bypasses the tree entirely, requiring only a simple read operation.
Each document contains an offset field (a 64-bit unsigned integer) which is set by the fdb_set, fdb_get, fdb_get_byseq, fdb_get_metaonly, and fdb_get_metaonly_byseq functions. To get a document using disk offset, create an empty document, assign a disk offset to it, and call fdb_get_byoffset:
#include <stdint.h>
...
fdb_handle *db;
fdb_doc *doc1, *doc2;
fdb_status status;
uint64_t offset;
...
status = fdb_doc_create(&doc1, "foo", 3, "00", 2, "bar", 3);
status = fdb_set(db, doc1);
offset = doc1->offset;
...
status = fdb_doc_create(&doc2, NULL, 0, NULL, 0, NULL, 0);
doc2->offset = offset;
status = fdb_get_byoffset(db, doc2);
...Deleting a document is similar to updating it. The following example shows how to delete a document:
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_doc *doc;
fdb_status status;
...
status = fdb_doc_create(&doc, "foo", 3, NULL, 0, NULL, 0);
status = fdb_del(db, doc);
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
status = fdb_doc_free(doc);
...Note that documents deleted using fdb_del are not physically removed at that time (since the database file is append-only). ForestDB lazily purges those documents upon the next compaction after the period configured in fdb_config. Although the deleted documents cannot be retrieved by fdb_get or fdb_get_byseq, you can still read their metadata by calling fdb_get_metaonly or fdb_get_metaonly_byseq until they are purged from the DB.
You can change the purging interval defined in fdb_config as follows:
fdb_file_handle *dbfile;
fdb_status status;
fdb_config config;
config = fdb_get_default_config();
config.purging_interval = 3600;
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
...The unit of the purging interval is in seconds. If set to zero (which is the default), deleted documents are immediately purged at the next compaction.
Document updates in ForestDB are first added to the write-ahead log (WAL), and later reflected into the HB+trie. WAL entries are automatically flushed when fdb_commit is called, or when the total number of documents in the WAL exceeds a configured threshold in fdb_config. The default value is 4,096 documents, and you can change the threshold as follows:
fdb_file_handle *dbfile;
fdb_status status;
fdb_config config;
config = fdb_get_default_config();
config.wal_threshold = 65536;
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
...If you want to manually flush WAL entries before the threshold is reached, call fdb_commit with the FDB_COMMIT_MANUAL_WAL_FLUSH option, as follows:
fdb_file_handle *dbfile;
fdb_status status;
...
status = fdb_commit(dbfile, FDB_COMMIT_MANUAL_WAL_FLUSH);
...ForestDB provides simplified key-value operations, for those who do not want to use document-related features such as sequence number and metadata. There are three functions for key-value operations: fdb_set_kv, fdb_get_kv, and fdb_del_kv.
To insert a key-value pair, call fdb_set_kv:
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_status status;
...
status = fdb_set_kv(db, "foo", 3, "bar", 3);
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
...By calling fdb_get_kv, you can get a value corresponding to a key:
fdb_handle *db;
fdb_status status;
void *value;
size_t value_len;
...
status = fdb_get_kv(db, "foo", 3, &value, &value_len);
...
free(value);You should free the value returned from fdb_get_kv API by calling free.
Key-value pairs can be deleted using fdb_del_kv API:
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_status status;
...
status = fdb_del_kv(db, "foo", 3);
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
...Since these key-value pairs are internally converted to and stored as documents, the simplified functions are compatible with the original APIs such as fdb_set and fdb_get. The following example gets the body of a document inserted using fdb_set, by calling fdb_get_kv.
fdb_file_handle *dbfile;
fdb_handle *db;
fdb_doc *doc;
fdb_status status;
void *value;
size_t value_len;
...
status = fdb_doc_create(&doc1, "foo", 3, "00", 2, "bar", 3);
status = fdb_set(db, doc1);
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
...
status = fdb_get_kv(db, "foo", 3, &value, &value_len);
...An iterator traverses a KV store instance in order by key or sequence number. It can also traverse a smaller range of keys or sequences. An index operates on a snapshot of the database, so any mutations made after the iterator is created have no effect on the documents returned by the iterator.
fdb_iterator_init and fdb_iterator_sequence_init create an iterator for keys and sequence numbers, respectively:
fdb_handle *db,
fdb_iterator *itr_key, *itr_seq;
fdb_status status;
...
status = fdb_iterator_init(db, &itr_key,
NULL, 0, NULL, 0,
FDB_ITR_NONE);
assert(status == FDB_RESULT_SUCCESS);
...
status = fdb_iterator_sequence_init(db, &itr_seq,
0, 0, FDB_ITR_NONE);
assert(status == FDB_RESULT_SUCCESS);
...You can assign a specific range to the iterator, by passing start and end values of the range. The following example creates an iterator traversing from key aaa to key zzz (inclusive):
...
status = fdb_iterator_init(db, &itr_key,
"aaa", 3, "zzz", 3,
FDB_ITR_NONE);
...If you pass NULL as the start key, the iterator starts at the first (smallest) key. If the end key is NULL, the iterator terminates after the last (largest) key. If the start key does not exist in the DB instance, the iterator begins from the smallest key that is greater than the start key. If the end key does not exist in the DB instance, the iterator ends at the greatest key that is smaller than the end key.
The last parameter to fdb_iterator_init is a set of option flags. The available options are to return only document metadata not bodies (FDB_ITR_METAONLY), and to omit deleted documents from the iteration (FDB_ITR_NO_DELETES).
fdb_iterator_sequence_init follows the same semantics. The example below shows how to create an iterator for the range of sequence number from 10 to 20:
...
status = fdb_iterator_sequence_init(db, &itr_seq,
10, 20, FDB_ITR_NONE);
...Once an iterator is created, you can get the documents from the iterator by calling fdb_iterator_next, regardless of the type of the iterator. fdb_iterator_next returns FDB_RESULT_SUCCESS if the next document for the iterator exists; in that case its doc parameter will point to an fdb_doc filled in with the contents of the next document. (As usual, this document structure should be freed when you're done using it.)
fdb_iterator_next returns FDB_RESULT_ITERATOR_FAIL when it reaches the end of the range. When finished, you should close the iterator by calling fdb_iterator_close.
fdb_handle *db,
fdb_doc *doc;
fdb_iterator *itr_key;
fdb_status status;
...
status = fdb_iterator_init(db, &itr_key,
NULL, 0, NULL, 0,
FDB_ITR_NONE);
while(fdb_iterator_next(itr_key, &doc) == FDB_RESULT_SUCCESS) {
...
fdb_doc_free(doc);
}
status = fdb_iterator_close(itr_key);
...fdb_handle *db,
fdb_doc *doc;
fdb_iterator *itr_seq;
fdb_status status;
...
status = fdb_iterator_sequence_init(db, &itr_seq,
0, 0, FDB_ITR_NONE);
while(fdb_iterator_next(itr_seq, &doc) == FDB_RESULT_SUCCESS) {
...
fdb_doc_free(doc);
}
status = fdb_iterator_close(itr_seq);
...If you want to move an iterator forward to get a document for a specific key, call fdb_iterator_seek:
fdb_iterator *itr_key;
fdb_status status;
fdb_doc doc;
...
status = fdb_iterator_seek(itr_key, "car", 3);
status = fdb_iterator_next(itr_key, &doc);
...Afterwards, the next call to fdb_iterator_next will return the document of the key that you sought to.
If the given key does not exist, fdb_iterator_seek moves to the smallest key greater than the given key.
If you want to move an iterator backward, call fdb_iterator_prev:
status = fdb_iterator_prev(itr_key, &doc);
...Iterator can move in both direction, and can seek in either direction.
You can get some information about a DB file using fdb_get_dbinfo:
fdb_file_handle *dbfile;
fdb_info info;
fdb_status status;
...
status = fdb_get_dbinfo(dbfile, &info);The return type is a fdb_info struct that contains the following information:
-
filename: The filename of the DB instance. -
new_filename: The name of the new file, if the DB instance is being compacted. -
doc_count: The approximate total number of documents stored in the DB instance. -
space_used: The approximate space occupied by live data in the DB instance. -
file_size: The total size of the DB file.
A similar API fdb_get_kvs_info exists for KV store instances. The example below shows how to get information about a KV store:
fdb_handle *db;
fdb_kvs_info kvs_info;
fdb_status status;
...
status = fdb_get_kvs_info(db, &kvs_info);The attributes of the return type fdb_kvs_info are as follows:
-
name: The name of the KV store. -
last_seqnum: The last sequence number of the KV store.
Note that there is a separate API that returns the last sequence number only:
fdb_handle *db;
fdb_seqnum_t seqnum;
fdb_status status;
...
status = fdb_get_seqnum(db, &seqnum);ForestDB can create a snapshot of a specific KV store instance, using a sequence number that corresponds to one of the commits that have been persisted in the same database instance. A snapshot is a read-only immutable DB handle: subsequent changes to the database will have no effect on the snapshot.
The following example shows how to create a snapshot:
fdb_file_handle *dbfile;
fdb_handle *db, *snapshot,
fdb_status status;
fdb_seqnum_t seqnum;
...
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
status = fdb_get_seqnum(db, &seqnum);
status = fdb_snapshot_open(db, &snapshot, seqnum);
...
status = fdb_kvs_close(snapshot);A snapshot is treated as a read-only KV store handle, thus it must be closed with fdb_kvs_close when it is not needed anymore.
You can rollback a KV store instance to a specific point represented by a sequence number that corresponds to a commit. fdb_rollback reverts a KV store instance as follows:
fdb_file_handle *dbfile,
fdb_status status;
fdb_seqnum_t seqnum;
...
status = fdb_commit(dbfile, FDB_COMMIT_NORMAL);
status = fdb_get_seqnum(db, &seqnum);
...
status = fdb_rollback(&db, seqnum);
...Note that all other mutations of the DB file are blocked while the rollback operation is being performed.
ForestDB supports transactional features which makes it possible to atomically update multiple documents. You can begin a transaction by calling fdb_begin_transaction, and commit dirty updates by calling fdb_end_transaction. Atomic updates across multiple KV stores in a DB instance are possible. Here's a simple example that atomically updates doc1 and doc2 to KV stores db1 and db2, respectively:
fdb_file_handle *dbfile;
fdb_handle *db1, *db2;
fdb_doc *doc1, *doc2;
fdb_status status;
...
status = fdb_begin_transaction(dbfile, FDB_ISOLATION_READ_COMMITTED);
status = fdb_set(db1, doc1);
status = fdb_set(db2, doc2);
status = fdb_end_transaction(dbfile, FDB_COMMIT_NORMAL);
...There are two isolation levels supported by the current version of ForestDB:
-
FDB_ISOLATION_READ_COMMITTED: Read committed isolation level. Dirty updates (i.e., uncommitted updates) by other transactions are not visible, although updates committed by other transactions will be visible. -
FDB_ISOLATION_READ_UNCOMMITTED: Read uncommitted isolation level. Dirty updates by other transactions are visible.
If you want to revert all dirty updates after fdb_begin_transaction, invoke fdb_abort_transaction instead of fdb_end_transaction:
fdb_file_handle *dbfile;
fdb_handle *db1, *db2;
fdb_doc *doc1, *doc2;
fdb_status status;
...
status = fdb_begin_transaction(dbfile, FDB_ISOLATION_READ_COMMITTED);
status = fdb_set(db1, doc1);
status = fdb_set(db2, doc2);
status = fdb_abort_transaction(dbfile);
...Since ForestDB is based on an MVCC model, concurrent read and update operations do not block each other. All ForestDB operations are thread-safe, but you should open a separate handle on a DB file for each thread. Both the fdb_file_handle and fdb_handle structure is not thread-safe, and ForestDB does not guarantee thread-safety if multiple threads share one DB or KV store handle. The following example shows how to concurrently access to a DB instance:
- Thread 1
fdb_file_handle *dbfile_thread1;
fdb_handle *db_thread1;
fdb_status status;
fdb_config config;
fdb_kvs_config kvs_config;
...
status = fdb_open(&dbfile_thread1, "/tmp/db_filename", &config);
status = fdb_kvs_open_default(dbfile_thread1, &db_thread1, &kvs_config);
...
status = fdb_kvs_close(db_thread1);
status = fdb_close(dbfile_thread1);- Thread 2
fdb_file_handle *dbfile_thread2;
fdb_handle *db_thread2;
fdb_status status;
fdb_config config;
fdb_kvs_config kvs_config;
...
status = fdb_open(&db_thread2, "/tmp/db_filename", &config);
status = fdb_kvs_open_default(dbfile_thread2, &db_thread2, &kvs_config);
...
status = fdb_kvs_close(db_thread2);
status = fdb_close(dbfile_thread2);The threads access the same DB instance /tmp/db_filename using their own DB file handles (dbfile_thread1 and dbfile_thread2), and they do not need to grab a lock on the shared DB instance when they invoke ForestDB operations.
Note: Multiple handles on the same database file share the same underlying file caches, so the memory overhead per extra handle is low.
As ForestDB uses append-only style disk logging, the overall space occupied by a DB instance grows with more mutations. This fragmented DB instance should be compacted to reclaim the space used by stale data. You can manually compact a DB instance by calling fdb_compact:
fdb_file_handle *dbfile;
fdb_status status;
fdb_config config;
...
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
...
status = fdb_compact(dbfile, "/tmp/db_filename2");
...After compaction, the previous file /tmp/db_filename is removed and new file /tmp/db_filename2 is created. All DB handles referring to the DB instance automatically change their information to point to the new file, thus it is not necessary to close the existing DB handles and open the new file.
Note that you do not need to manually compact the DB instance in auto-compaction mode, but manual compaction is possible by calling fdb_compact with a NULL filename as follows:
fdb_file_handle *dbfile;
fdb_status status;
fdb_config config;
...
config = fdb_get_default_config();
config.compaction_mode = FDB_COMPACTION_AUTO;
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
...
status = fdb_compact(dbfile, NULL);
...DB instances created under auto-compaction mode cannot be opened under manual-compaction mode, and vice versa. To open a DB instance based on different compaction mode, it has to switch its compaction mode using fdb_switch_compaction_mode API:
fdb_file_handle *dbfile;
fdb_status status;
fdb_config config;
...
config = fdb_get_default_config();
config.compaction_mode = FDB_COMPACTION_AUTO;
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
...
status = fdb_switch_compaction_mode(dbfile, FDB_COMPACTION_MANUAL, 0);
status = fdb_close(dbfile);
config.compaction_mode = FDB_COMPACTION_MANUAL;
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
...fdb_switch_compaction_mode also can be used for changing the compaction threshold for the given DB instance whose compaction mode is currently auto-compaction. The following example changes the compaction threshold of the DB instance from 30% to 50%:
fdb_file_handle *dbfile;
fdb_status status;
fdb_config config;
...
config = fdb_get_default_config();
config.compaction_mode = FDB_COMPACTION_AUTO;
config.compaction_threshold = 30;
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
...
status = fdb_switch_compaction_mode(dbfile, FDB_COMPACTION_AUTO, 50);
...The compaction threshold is calculated as stale_data_size / total_file_size. For example, when the compaction threshold and live data size are 30% and 70 MB, respectively, compaction is triggered when the DB file size gets larger than 100 MB, since the stale data size (i.e., 100 MB - 70 MB = 30 MB) exceeds 30% of the file size.
Note that compaction will not be executed if you set the compaction threshold to zero or 100%.
ForestDB supports pluggable custom comparison function. A custom comparison functions is given pointers to two keys, key1 and key2 for example, and must return zero if the keys are equal, a negative value if key1 < key2, or a positive value if key1 > key2.
The following code represents an example that uses a custom comparison function for floating-point numbers represented as the double primitive type:
- Custom comparison function for
doubletype:
int _cmp_double(void *key1_ptr, size_t keylen1,
void *key2_ptr, size_t keylen2)
{
double key1, key2;
key1 = *(double *)key1_ptr;
key2 = *(double *)key2_ptr;
if (key1 < key2) return -1;
else if (key1 > key2) return 1;
else return 0;
}fdb_file_handle *dbfile;
fdb_handle *db;
fdb_doc *doc;
fdb_status status;
fdb_config config;
fdb_kvs_config kvs_config;
double key;
void *value;
size_t value_len;
...
config = fdb_get_default_config();
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
kvs_config = fdb_get_default_kvs_config();
kvs_config.custom_cmp = _cmp_double;
status = fdb_kvs_open_default(dbfile, &db, &kvs_config);
...
status = fdb_doc_create(&doc,
(void *)&key, sizeof(double),
NULL, 0, value, value_len);
status = fdb_set(db, doc);
status = fdb_doc_free(doc);
...Note: This example is not endian-safe, so a database created on a little-endian CPU would not be readable on a big-endian CPU! Any real usage of custom key types should be careful to represent keys with a consistent byte ordering.
The next example shows another code that uses a custom comparison function for non-primitive type keys:
- Custom comparison function that compares only last four bytes in each key:
int _cmp_substring(void *key1, size_t keylen1,
void *key2, size_t keylen2)
{
if (keylen1 < 4 || keylen2 < 4) return 0;
return memcmp((char*)key1 + keylen1 - 4,
(char*)key2 + keylen2 - 4, 4);
}fdb_file_handle *dbfile;
fdb_handle *db;
fdb_status status;
fdb_config config;
fdb_kvs_config kvs_config
...
config = fdb_get_default_config();
status = fdb_open(&dbfile, "/tmp/db_filename", &config);
kvs_config = fdb_get_default_kvs_config();
kvs_config.custom_cmp = _cmp_substring;
status = fdb_kvs_open_default(dbfile, &db, &kvs_config);
...If you open a DB file that contains at least one KV store that is based on custom key order, all custom comparison functions should be passed with corresponding KV store names, through fdb_open_custom_cmp API. Suppose that there are four KV stores: default, KV1, KV2, and KV3, where the default and KV3 use custom comparison functions _cmp_double and _cmp_substring, respectively, the following example represents how to pass the custom comparison functions when the DB file is opened:
fdb_file_handle *dbfile;
fdb_handle *default, *kv1, *kv2, *kv3;
fdb_status status;
fdb_config config;
fdb_kvs_config kvs_config
char *kvs_names[] = {NULL, (char*)"KV3"};
fdb_custom_cmp_variable functions[] = {_cmp_double,
_cmp_substring};
...
config = fdb_get_default_config();
status = fdb_open_custom_cmp(&dbfile, "/tmp/db_filename", &config,
2, kvs_names, functions);
kvs_config = fdb_get_default_kvs_config();
status = fdb_kvs_open_default(dbfile, &default, &kvs_config);
status = fdb_kvs_open(dbfile, &kv1, &kvs_config);
status = fdb_kvs_open(dbfile, &kv2, &kvs_config);
status = fdb_kvs_open(dbfile, &kv3, &kvs_config);
...Note that KV stores based on the default (i.e., lexicographical) key order do not need to be included in the kvs_names and functions parameters. Once the DB file is opened, the passed custom comparison functions are automatically assigned to corresponding KV store handles, so that you do not need to assign the custom_cmp attribute in the fdb_kvs_config type.
If you wish to clean up all the files associated with a ForestDB instance, you can call the fdb_destroy api as follows:
fdb_status status;
fdb_config config;
...
config = fdb_get_default_config();
status = fdb_destroy("/tmp/db_filename", &config);
...The above routine can be useful to clean up all ForestDB files including meta data files for an auto compaction setting. If the destroyed file is currently being compacted by the auto compaction daemon which is still running, then fdb_destroy will return FDB_RESULT_IN_USE_BY_COMPACTOR, and the operation would need to be retried after the file compaction completes. In case of manual compaction a best-effort is done to wipe out all known files associated with the ForestDB path provided.
You can remove a specific KV store instance in a DB instance, without affecting any other KV store instances in the same DB. The example below removes the KV store named "KV1":
fdb_file_handle *dbfile;
fdb_status status;
...
status = fdb_kvs_remove(dbfile, "KV1");
...All handles to the KV store should be closed before calling fdb_kvs_remove. Once you drop a KV store, all documents stored in the KV store will be permanently removed. Note that the DB file size will not be reduced immediately, because the stale documents still reside in the file. If you want to physically eliminate the stale data, call fdb_compact after dropping a KV store instance.