Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove mdbx support #125

Merged
merged 2 commits into from
Mar 8, 2022
Merged

Remove mdbx support #125

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
197e2ad
Change the CI to only check that the LMDB backend compile
Kerollmops Mar 1, 2022
cc590a7
Remove all the references to MDBX
Kerollmops Mar 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 3 additions & 40 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,16 @@ on: [pull_request]

name: Cargo test
jobs:
check_lmdb:
name: Test the heed project with LMDB
check:
name: Test the heed project
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
include:
- os: ubuntu-latest
artifact_name: meilidb-http
asset_name: meilidb-http-linux-amd64
- os: windows-latest
artifact_name: meilidb-http.exe
asset_name: meilidb-http-windows-amd64
- os: macos-latest
artifact_name: meilidb-http-macos-amd64
asset_name: meilidb-http-macos-amd64

steps:
- uses: actions/checkout@v2
Expand All @@ -29,39 +23,8 @@ jobs:
override: true
- name: Run cargo test
run: |
cd heed
cargo clean
cargo test --features 'lmdb serde-json' --no-default-features

check_mdbx:
name: Test the heed project with MDBX
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
include:
- os: ubuntu-latest
artifact_name: meilidb-http
asset_name: meilidb-http-linux-amd64
- os: windows-latest
artifact_name: meilidb-http.exe
asset_name: meilidb-http-windows-amd64
- os: macos-latest
artifact_name: meilidb-http-macos-amd64
asset_name: meilidb-http-macos-amd64

steps:
- uses: actions/checkout@v2
- uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: stable
override: true
- name: Run cargo test
run: |
cd heed
cargo clean
cargo test --features 'mdbx serde-json' --no-default-features
cargo test

fmt:
name: Ensure the heed project is formatted
Expand Down
29 changes: 7 additions & 22 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,15 @@
# heed
A fully typed [LMDB]/[MDBX] wrapper with minimum overhead, uses zerocopy internally.
A fully typed [LMDB](https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database) wrapper with minimum overhead, uses zerocopy internally.

[![Build Status](https://dev.azure.com/renaultcle/heed/_apis/build/status/Kerollmops.heed?branchName=master)](https://dev.azure.com/renaultcle/heed/_build/latest?definitionId=1&branchName=master)
[![Dependency Status](https://deps.rs/repo/github/Kerollmops/heed/status.svg)](https://deps.rs/repo/github/Kerollmops/heed)
[![Heed Doc](https://docs.rs/heed/badge.svg)](https://docs.rs/heed)
[![Crates.io](https://img.shields.io/crates/v/heed.svg)](https://crates.io/crates/heed)
[![License](https://img.shields.io/badge/license-MIT-green)](#LICENSE)
[![Crates.io](https://img.shields.io/crates/v/heed)](https://crates.io/crates/heed)
[![Docs](https://docs.rs/heed/badge.svg)](https://docs.rs/heed)
[![dependency status](https://deps.rs/repo/github/meilisearch/heed/status.svg)](https://deps.rs/repo/github/meilisearch/heed)
[![Build](https://github.com/meilisearch/heed/actions/workflows/test.yml/badge.svg)](https://github.com/meilisearch/heed/actions/workflows/test.yml)

![the opposite of heed](https://thesaurus.plus/img/antonyms/153/heed.png)

This library is able to serialize all kind of types, not just bytes slices, even Serde types are supported.

## Using MDBX instead of LMDB

Heed can also [be used with MDBX][MDBX], this is a compatible backend, you can activate it by using the `mdbx` feature. When MDBX is used the generated databases cannot be read by using the `lmdb` feature, those two kv-stores are not compatible and vice-versa.

Notice that some specific features will only be accessible by using the `mdbx` cargo feature.
Environment creation flags depends on the backend you chose.

To test that heed works with MDBX you can run this command:

```bash
cargo test --features 'mdbx serde-json' --no-default-features
```
This library is able to serialize all kind of types, not just bytes slices, even _Serde_ types are supported.

## Example Usage

Expand Down Expand Up @@ -56,6 +44,3 @@ assert_eq!(ret, Some(5));
```

You want to see more about all the possibilities? Go check out [the examples](heed/examples/).

[LMDB]: https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database
[MDBX]: https://github.com/erthink/libmdbx
12 changes: 2 additions & 10 deletions heed/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ byteorder = { version = "1.3.4", default-features = false }
heed-traits = { version = "0.7.0", path = "../heed-traits" }
heed-types = { version = "0.7.2", path = "../heed-types" }
libc = "0.2.80"
lmdb-rkv-sys = { git = "https://github.com/meilisearch/lmdb-rs", optional = true }
mdbx-sys = { version = "0.7.1", optional = true }
lmdb-rkv-sys = { git = "https://github.com/meilisearch/lmdb-rs" }
once_cell = "1.5.2"
page_size = "0.4.2"
serde = { version = "1.0.118", features = ["derive"], optional = true }
Expand All @@ -33,20 +32,13 @@ url = "2.2.0"
[features]
# The `serde` feature makes some types serializable,
# like the `EnvOpenOptions` struct.
default = ["lmdb", "serde", "serde-bincode", "serde-json"]
default = ["serde", "serde-bincode", "serde-json"]

# The NO_TLS flag is automatically set on Env opening and
# RoTxn implements the Sync trait. This allow the user to reference
# a read-only transaction from multiple threads at the same time.
sync-read-txn = []

# Choose between using the MDBX key-value store or LMDB
# MDBX is a fork of LMDB: https://github.com/erthink/libmdbx
#
# You cannot use both features at the same time, you have to choose!
lmdb = ["lmdb-rkv-sys"]
mdbx = ["mdbx-sys"]

# Enable the serde en/decoders for bincode or serde_json
serde-bincode = ["heed-types/serde", "heed-types/bincode"]
serde-json = ["heed-types/serde", "heed-types/serde_json"]
Expand Down
116 changes: 0 additions & 116 deletions heed/src/db/polymorph.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,122 +113,6 @@ impl PolyDatabase {
PolyDatabase { env_ident, dbi }
}

/// Retrieve the sequence of a database.
///
/// This function allows to retrieve the unique positive integer of this database.
///
/// ```
/// # use std::fs;
/// # use std::path::Path;
/// # use heed::EnvOpenOptions;
/// use heed::Database;
/// use heed::types::*;
///
/// # fn main() -> Result<(), Box<dyn std::error::Error>> {
/// # fs::create_dir_all(Path::new("target").join("zerocopy.mdb"))?;
/// # let env = EnvOpenOptions::new()
/// # .map_size(10 * 1024 * 1024) // 10MB
/// # .max_dbs(3000)
/// # .open(Path::new("target").join("zerocopy.mdb"))?;
/// let db = env.create_poly_database(Some("use-sequence-1"))?;
///
/// // The sequence starts at zero
/// let rtxn = env.read_txn()?;
/// let ret = db.sequence(&rtxn)?;
/// assert_eq!(ret, 0);
/// rtxn.abort()?;
///
/// let mut wtxn = env.write_txn()?;
/// # db.clear(&mut wtxn)?;
/// let incr = db.increase_sequence(&mut wtxn, 32)?;
/// assert_eq!(incr, Some(0));
/// let incr = db.increase_sequence(&mut wtxn, 28)?;
/// assert_eq!(incr, Some(32));
/// wtxn.commit()?;
///
/// let rtxn = env.read_txn()?;
/// let ret = db.sequence(&rtxn)?;
/// assert_eq!(ret, 60);
///
/// # Ok(()) }
/// ```
#[cfg(all(feature = "mdbx", not(feature = "lmdb")))]
pub fn sequence<T>(&self, txn: &RoTxn<T>) -> Result<u64> {
assert_eq!(self.env_ident, txn.env.env_mut_ptr() as usize);

let mut value = mem::MaybeUninit::uninit();

let result = unsafe {
mdb_result(ffi::mdbx_dbi_sequence(
txn.txn,
self.dbi,
value.as_mut_ptr(),
0, // increment must be 0 for read-only transactions
))
};

match result {
Ok(()) => unsafe { Ok(value.assume_init()) },
Err(e) => Err(e.into()),
}
}

/// Increment the sequence of a database.
///
/// This function allows to create a linear sequence of a unique positive integer
/// for this database. Sequence changes become visible outside the current write
/// transaction after it is committed, and discarded on abort.
///
/// Returns `Some` with the previous value and `None` if increasing the value
/// resulted in an overflow an therefore cannot be executed.
///
/// ```
/// # use std::fs;
/// # use std::path::Path;
/// # use heed::EnvOpenOptions;
/// use heed::Database;
/// use heed::types::*;
///
/// # fn main() -> Result<(), Box<dyn std::error::Error>> {
/// # fs::create_dir_all(Path::new("target").join("zerocopy.mdb"))?;
/// # let env = EnvOpenOptions::new()
/// # .map_size(10 * 1024 * 1024) // 10MB
/// # .max_dbs(3000)
/// # .open(Path::new("target").join("zerocopy.mdb"))?;
/// let db = env.create_poly_database(Some("use-sequence-2"))?;
///
/// let mut wtxn = env.write_txn()?;
/// let incr = db.increase_sequence(&mut wtxn, 32)?;
/// assert_eq!(incr, Some(0));
/// let incr = db.increase_sequence(&mut wtxn, 28)?;
/// assert_eq!(incr, Some(32));
/// wtxn.commit()?;
///
/// let rtxn = env.read_txn()?;
/// let ret = db.sequence(&rtxn)?;
/// assert_eq!(ret, 60);
///
/// # Ok(()) }
/// ```
#[cfg(all(feature = "mdbx", not(feature = "lmdb")))]
pub fn increase_sequence<T>(&self, txn: &mut RwTxn<T>, increment: u64) -> Result<Option<u64>> {
assert_eq!(self.env_ident, txn.txn.env.env_mut_ptr() as usize);

use crate::mdb::error::Error;

let mut value = mem::MaybeUninit::uninit();

let result = unsafe {
mdb_result(ffi::mdbx_dbi_sequence(txn.txn.txn, self.dbi, value.as_mut_ptr(), increment))
};

match result {
Ok(()) => unsafe { Ok(Some(value.assume_init())) },
Err(Error::Other(c)) if c == i32::max_value() => Ok(None), // MDBX_RESULT_TRUE
Err(e) => Err(e.into()),
}
}

/// Retrieves the value associated with a key.
///
/// If the key does not exist, then `None` is returned.
Expand Down
23 changes: 0 additions & 23 deletions heed/src/db/uniform.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,29 +124,6 @@ impl<KC, DC> Database<KC, DC> {
Database { dyndb: PolyDatabase::new(env_ident, dbi), marker: std::marker::PhantomData }
}

/// Retrieve the sequence of a database.
///
/// This function allows to retrieve the unique positive integer of this database.
/// You can see an example usage on the `PolyDatabase::sequence` method documentation.
#[cfg(all(feature = "mdbx", not(feature = "lmdb")))]
pub fn sequence<T>(&self, txn: &RoTxn<T>) -> Result<u64> {
self.dyndb.sequence(txn)
}

/// Increment the sequence of a database.
///
/// This function allows to create a linear sequence of a unique positive integer
/// for this database. Sequence changes become visible outside the current write
/// transaction after it is committed, and discarded on abort.
/// You can see an example usage on the `PolyDatabase::increase_sequence` method documentation.
///
/// Returns `Some` with the previous value and `None` if increasing the value
/// resulted in an overflow an therefore cannot be executed.
#[cfg(all(feature = "mdbx", not(feature = "lmdb")))]
pub fn increase_sequence<T>(&self, txn: &mut RwTxn<T>, increment: u64) -> Result<Option<u64>> {
self.dyndb.increase_sequence(txn, increment)
}

/// Retrieves the value associated with a key.
///
/// If the key does not exist, then `None` is returned.
Expand Down
8 changes: 0 additions & 8 deletions heed/src/env.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,20 +464,12 @@ impl Env {
Ok(())
}

#[cfg(all(feature = "lmdb", not(feature = "mdbx")))]
pub fn force_sync(&self) -> Result<()> {
unsafe { mdb_result(ffi::mdb_env_sync(self.0.env, 1))? }

Ok(())
}

#[cfg(all(feature = "mdbx", not(feature = "lmdb")))]
pub fn force_sync(&self) -> Result<()> {
unsafe { mdb_result(ffi::mdb_env_sync(self.0.env))? }

Ok(())
}

/// Returns the canonicalized path where this env lives.
pub fn path(&self) -> &Path {
&self.0.path
Expand Down
Loading