Name

lua-resty-rocksdb - Lua library to build high performance but easy-to-use driver for RocksDB

Table of Contents

• Name
• Status
• Description
• Synopsis
• Methods
  • rocksdb.open_db
  • rocksdb.open_with_ttl
  • rocksdb.open_for_read_only
  • writer.put
  • rocksdb.delete
  • reader.get
  • reader.multi_get
  • iterator.new
  • iterator.seek_to_first
  • iterator.seek_to_last
  • iterator.seek
  • iterator.valid
  • iterator.get_error
  • iterator.next
  • iterator.prev
  • iterator.key
  • iterator.value
  • iterator.destroy
• Prerequisites
• Author
• Copyright and License

Status

This library is production ready

Synopsis

Demonstrate the usage of the resty.rocksdb.writer and resty.rocksdb.reader module.

http {
    # you may need to change the path depending on where the libs are installed.
    lua_package_path "/path/to/lua-resty-rocksdb/lib/?.lua;;";

    server {
        listen 8080;

        location = /t {
            rewrite_by_lua_block {
                local rocksdb = require("resty.rocksdb.rocksdb")
                local options = require("resty.rocksdb.options")
                local writer = require("resty.rocksdb.writer")
                local reader = require("resty.rocksdb.reader")
                
                local opt = options.rocksdb_options_create()
                # create the DB if it's not already present
                options.rocksdb_options_set_create_if_missing(opt, true)
                
                local db, err_code, err_msg = rocksdb.open_db(opt, "/tmp/rocksdb_simple_example_backup")
                if err_code ~= nil then
                    ngx.status = 500
                    ngx.say("failed to open db: ", err_code, err_msg)
                    ngx.exit(ngx.HTTP_OK)
                end
                
                local write_opt = options.rocksdb_writeoptions_create()
                
                # put key-value
                local _, err_code, err_msg = writer.put(db, write_opt, "foo", "bar")
                if err_code ~= nil then
                    ngx.status = 500
                    ngx.say("failed to write db: ", err_code, err_msg)
                    ngx.exit(ngx.HTTP_OK)
                end
                
                local read_opt = options.rocksdb_readoptions_create()
                
                # get value
                local value, err_code, err_msg = reader.get(db, read_opt, "foo")
                if err_code ~= nil then
                    ngx.status = 500
                    ngx.say("failed to read db: ", err_code, err_msg)
                    ngx.exit(ngx.HTTP_OK)
                end
                
                ngx.say("the value is:", value))
                
                # close db
                rocksdb.rocksdb_close()
                
                ngx.eof()
                ngx.exit(ngx.HTTP_OK)
            }
        }
    }
}

Back to TOC

Description

This library uses FFI to encapsulate the RocksDB storage engine, provides a persistent key value store.

It provides the commonly used operation, like put、delete、get、multiget and iterate methods which can modify/query the database. More methods are still in the follow-up development plan.

Methods

rocksdb.open_db

syntax: db, err_code, err_msg = rocksdb.open_db(opts, db_path)

Creates a rocksdb instance. In case of failures, returns db is nil a string describing the err_msg.

The opts argument is generated by options.rocksdb_options_create, used to set the configuration options of db instance.

The db_path argument specifies the path of the database to store data.

Notice: calling this method multiple times will return the same instance, it is safe. Another thing to note is that method conflicts with rocksdb.open_with_ttl、rocksdb.open_for_read_only, only can call one of them in same ngx work.

Back to TOC

rocksdb.open_with_ttl

syntax: db, err_code, err_msg = rocksdb.open_with_ttl(opts, db_path, ttl)

Creates a rocksdb instance with Time to Live(TTL).

Expired TTL values are deleted in compaction only(Timestamp+ttl<time_now).

Get/Iterator may return expired entries(compaction not run on them yet).

The ttl is a number type accepted in seconds, be careful when passing ttl with a small positive value because the whole database may be deleted in a small amount of time. Return values and other arguments are the same as the rocksdb.open_db.

By calling ROCKSDB_LIBRARY_API rocksdb_open_with_ttl to implement, For more details, please refer to Time-to-Live

Back to TOC

rocksdb.open_for_read_only

syntax: db, err_code, err_msg = rocksdb.open_for_read_only(opts, db_path, error_if_wal_file_exist)

Opens the database in ReadOnly mode, the database guarantees that the application may not modify anything in the database. In this mode only support Get Operation, provides higher read performance.

The error_if_wal_file_exist argument is bool type, if the value is true it will raise error when wal file exists. Return values and other arguments are the same as the rocksdb.open_db.

Notice: rocksdb only allows a single Primary(write/read) instance, but many concurrent ReadOnly instances are allowed, so you can open many instances in different ngx work.

By calling ROCKSDB_LIBRARY_API rocksdb_open_for_read_only to implement, For more details, please refer to Read-only-and-Secondary-instances

Back to TOC

rocksdb.close_db

syntax: nil, err_code, err_msg = rocksdb.close_db()

Close the current rocksdb instance, it will release all resources. Returns err_code is not nil when rocksdb instance has closed or not yet created.

Back to TOC

writer.put

syntax: nil, err_code, err_msg = writer.put(db, write_opts, key, val)

Sets a key with a value. In case of error, returns err_code is not nil and a string describing the err_msg.

The write_opts argument is generated by options.rocksdb_writeoptions_create.

Back to TOC

rocksdb.delete

syntax: nil, err_code, err_msg = rocksdb.delete(write_opts, key, val)

Removes the database data for a key, In case of error, returns err_code is not nil and a string describing the err_msg.

The write_opts argument is generated by options.rocksdb_writeoptions_create.

Back to TOC

reader.get

syntax: value, err_code, err_msg = reader.get(db, read_opts, key)

Fetches a value with the key. Returns value is nil if the key does not exist in the rocksdb.

The read_opts argument is generated by options.rocksdb_readoptions_create.

Back to TOC

reader.multi_get

syntax: values, err_code, err_msg = reader.multi_get(db, read_opts, keys, partial_success?)

Fetches multi values with the keys list.

The keys argument is a table contains the keys to fetch.

The optional partial_success argument is bool type. A false argument means that if an exception is triggered when a key in the keys gets the value(which is the default), returns values is nil and a string describing the err_msg. The difference from the above, if partial_success is true, the nil value insert to values.

The read_opts argument is generated by options.rocksdb_readoptions_create.

Back to TOC

iterator.new

syntax: iter = iterator.new(db, read_opts)

Creates a new iterator object, a consistent-point-in-time view of the database is created when the iterator is created.

Caller should delete the iterator when it is no longer needed.

The returned iterator should be deleted before this db is deleted.

The read_opts argument is generated by options.rocksdb_readoptions_create.

For more details, please refer to Iterator

Back to TOC

iterator.seek_to_first

syntax: iterator.seek_to_first(iter)

Seek to the first key in total sorted order keys, and then the application can start scanning at a time from that point.

Here is an example:

local rocksdb = require("resty.rocksdb.rocksdb")
local options = require("resty.rocksdb.options")
local iterator = require("resty.rocksdb.iterator")

local read_opt = options.rocksdb_readoptions_create()

local iter = iterator.new(db, read_opt)
local result = {}
iter:seek_to_first()

while iter:valid() do
    local key = iter:key()
    local value = iter:value()
    ngx.say("scaned key is ", key, " value is ", value)
    iter:next()
end

local err = iter:get_error()
if err ~= nil then
    ngx.say("failed to iterator db: ", err)
end

iter:destroy()

Back to TOC

iterator.seek_to_end

syntax: iterator.seek_to_end(iter)

Seek to the end key in total sorted order keys, function is similar to iterator.seek_to_first. Usually used with method iterator.prev, that will scan the sorted key in reverse.

Here is an example:

local rocksdb = require("resty.rocksdb.rocksdb")
local options = require("resty.rocksdb.options")
local iterator = require("resty.rocksdb.iterator")

local read_opt = options.rocksdb_readoptions_create()

local iter = iterator.new(db, read_opt)
local result = {}
iter:seek_to_end()

while iter:valid() do
    local key = iter:key()
    local value = iter:value()
    ngx.say("scaned key is ", key, " value is ", value)
    iter:prev()
end

local err = iter:get_error()
if err ~= nil then
    ngx.say("failed to iterator db: ", err)
end

iter:destroy()

Back to TOC

iterator.seek

syntax: iterator.seek(iter, key)

Seek to a specified key, and then the application can start scanning at a time from that point.

Back to TOC

iterator.valid

syntax: ok = iterator.valid(iter)

Check the iterator valid, if returns ok is true, is guaranteed iterator.key and iterator.value can get right value. On the other hand, if ok is false, there are two possibilities:

• (1) We reached the end of the data. In this case, iterator.get_error is nil;
• (2) there is an error. In this case iterator.get_error is not nil. It is always a good practice to check iterator.get_error if the iterator is invalidated.

Back to TOC

iterator.get_error

syntax: ok = iterator.get_error(iter)

Try to get iterator error, usually used with method iterator.valid.

Back to TOC

iterator.next

syntax: iterator.next(iter)

Seek to the next key point of current iterator.

Back to TOC

iterator.prev

syntax: iterator.prev(iter)

Seek to the previous key point of current iterator.

Back to TOC

iterator.key

syntax: key = iterator.key(iter)

Returns the key name of iterator's current key point.

Back to TOC

iterator.value

syntax: value = iterator.value(iter)

Returns the value data of iterator's current key point.

Back to TOC

iterator.destroy

syntax: value = iterator.destroy(iter)

Delete iterator and release iterator resources.

Back to TOC

Prerequisites

• LuaJIT 2.1+
• OpenResty 1.13.6.3+
• RocksDB 6.29.5+

Back to TOC

Author

Wuyipu (吴义谱) [email protected].

Luoguangxu (罗光旭) [email protected].

Lianzhilei (连志雷) [email protected].

Back to TOC

Copyright and License

The MIT License (MIT)

Copyright (c) 2022, by Yipu Wu.

Copyright (c) 2022, by Guangxu Luo.

Copyright (c) 2022, Zhilei Lian.

Back to TOC