RediS3

Poor man's HA and distributed key-value storage GO library running on top of AWS S3.

In some projects/PoC you may require some kind of persistence, perhaps accessible from different nodes/processes and in a key-value format.
RediS3 is a simple key-value library that leverages AWS S3 to provide HA and distributed persistence.

This library is in early stages and is missing some key features, but you can see what is this about, therefore PR and suggestions are very welcome.

Features

• HA and distributed (kindly provided by AWS S3)
• Key locking. Soft and Hard consistency
• Store GO objects (GO built in and struct objects)
• Key expiration
• List keys
• Read-only client
• Configurable Exponential Back-Off for AWS calls
• Client stats/metrics

Requirements

RediS3 leverages AWS S3 service to persist data. This means that the node running RediS3 requires proper access to AWS S3 service.

There are several ways to provide AWS credentials and proper access level to S3 buckets:

• For testing purposes, run Moto AWS mock locally. Example running Moto in a Docker container:

docker pull picadoh/motocker
docker run --rm --name s3 -d -e MOTO_SERVICE=s3 -p 5001:5000 -i picadoh/motocker
export AWS_ACCESS_KEY_ID=DUMMYAWSACCESSKEY
export AWS_SECRET_ACCESS_KEY=DUMMYAWSSECRETACCESSKEY

• For testing purposes, use Roly to use AWS S3 with STS tokens in your local machine.
• If you are running RediS3 on an EC2 instance, attach an Instance Profile with proper permissions to the EC2 instance.
• (Not recommended) use environment variables to provide proper access credentials.

Installation

Install:

go get -u github.com/danfaizer/redis3

Import:

import "github.com/danfaizer/redis3"

Quickstart

// Create RediS3 client
client, err := redis3.NewClient(
  &redis3.Options{
    Bucket:             "redis3-database",
    AutoCreateBucket:   true,
    Region:             "eu-west-1",
    Timeout:            1,
    EnforceConsistency: true,
  })
if err != nil {
  panic(err)
}

type person struct {
    uuid string
    name string
    age  int
}

p := person{
  uuid: "123e4567-e89b-12d3-a456-426655440000",
  name: "Daniel",
  age:  35,
}

var err error

err = client.Set(p.id, p, 0)
if err != nil {
  panic(err)
}

var b person

_, err = client.Get("123e4567-e89b-12d3-a456-426655440000", &b)
if err != nil {
  panic(err)
}

fmt.Printf("%+v", b)

{uuid:123e4567-e89b-12d3-a456-426655440000 name:Daniel age:35}