MongoTSDB : Mongo Time Series DB

MongoTSDB is a small python library which help you to save datas in a time-series way and make time-series queries. It's powered by MongoDB aggregation framework, so it requires MongoDB version 2.1.

Metric format

• A timestamp.
• A value.
• A name.

Insertion

Insert a sample metric

from mongotsdb import TSDB
tsdb = TSDB('database')

data = {'date': 1348813543, 'value': 42, 'name': 'connections'}
tsdb.insert()

Queries

For all the queries below, we have theses data inserted

for i in range(20):
    tsdb.insert({'date': i, 'value': i*10, 'name': 'sample'})

Queries format:

• request format: 'operator(metric_name)'
• start date, a timestamp
• stop date, a timestamp
• step, an int

Get the sum of all metric

tsdb.request({'request': 'sum(sample)', 'start': 0, 'stop': 20, 'step': 20})

This request will return

[{u'_id': {u'date': 0}, u'value': 1900}]

Not very interesting, try to get the sum of metrics by time range equal to 5

tsdb.request({'request': 'sum(sample)', 'start': 0, 'stop': 20, 'step': 5})

Output

[{u'_id': {u'date': 15}, u'value': 850},
 {u'_id': {u'date': 5}, u'value': 350},
 {u'_id': {u'date': 10}, u'value': 600},
 {u'_id': {u'date': 0}, u'value': 100}]

Huzzah!

You have access to several operators:

• sum
• avg
• min
• max

Example with avg

tsdb.request({'request': 'sum(sample)', 'start': 0, 'stop': 20, 'step': 5})

Output

[{u'_id': {u'date': 15}, u'value': 170.0},
 {u'_id': {u'date': 5}, u'value': 70.0},
 {u'_id': {u'date': 10}, u'value': 120.0},
 {u'_id': {u'date': 0}, u'value': 20.0}]

Tags

When inserting metrics or make request on them, you can use tags. It allows to make multi-dimensionnal requests.

Inserting

You can add tags to metrics, for example

from mongotsdb import TSDB
tsdb = TSDB('database')

data = {'date': 1348813543, 'value': 42, 'name': 'connections'}
tags = {'server': 'server1', 'application': 'youtube'}
data['tags'] = tags
tsdb.insert(data)

Requests

In each request, you can set tags and request will process only metrics which match tags.

For each following queries, these metrics are in DB

datas = [(1, 10, 'host1', '1'), (3, 20, 'host1', '3'),
    (13, 20, 'host2', '1'), (17, 20, 'host2', '4'),
    (19, 30, 'host1', '2'), (5, 40, 'host2', '2'),
    (23, 50, 'host1', '3')]

for data in datas:
    metric = {'date': data[0], 'value': data[1],
        'name': 'metric_tags'}
    metric_tags = {'host': data[2], 'category': data[3]}
    tsdb.insert(metric, **metric_tags)

You can now make request by setting tags value

tsdb.request({'stop': 30, 'start': 0, 'step': 10, 'request':
    'avg(sample)', 'tags': {'host': 'host1'}})

Output

[{u'_id': {u'date': 20, u'tags': {u'host': u'host1'}}, u'value': 50.0},
 {u'_id': {u'date': 10, u'tags': {u'host': u'host1'}}, u'value': 30.0},
 {u'_id': {u'date': 0, u'tags': {u'host': u'host1'}}, u'value': 15.0}]

You can also set multiple tags values

tsdb.request({'stop': 30, 'start': 0, 'step': 10, 'request':
    'avg(sample)', 'tags': {'host': 'host1', 'category': '3'}})

Output

[{u'_id': {u'date': 20, u'tags': {u'category': u'3', u'host': u'host1'}},
  u'value': 50.0},
 {u'_id': {u'date': 0, u'tags': {u'category': u'3', u'host': u'host1'}},
  u'value': 20.0}]

The real awesome feature is that you can choose to group by tags

tsdb.request({'stop': 30, 'start': 0, 'step': 10, 'request':
    'avg(sample)', 'tags': {'host': '*'}})

Output

[{u'_id': {u'date': 20, u'tags': {u'host': u'host1'}}, u'value': 50.0},
 {u'_id': {u'date': 0, u'tags': {u'host': u'host2'}}, u'value': 40.0},
 {u'_id': {u'date': 10, u'tags': {u'host': u'host1'}}, u'value': 30.0},
 {u'_id': {u'date': 10, u'tags': {u'host': u'host2'}}, u'value': 20.0},
 {u'_id': {u'date': 0, u'tags': {u'host': u'host1'}}, u'value': 15.0}]

As you can see, for the date 10, we have two results, one for host1 and another one for host2.

You can also group by multiple tags

tsdb.request({'stop': 30, 'start': 0, 'step': 10, 'request':
    'avg(sample)', 'tags': {'host': '*', 'category': '*'}})

Output

[{u'_id': {u'date': 20, u'tags': {u'category': u'3', u'host': u'host1'}},
  u'value': 50.0},
 {u'_id': {u'date': 0, u'tags': {u'category': u'2', u'host': u'host2'}},
  u'value': 40.0},
 {u'_id': {u'date': 10, u'tags': {u'category': u'4', u'host': u'host2'}},
  u'value': 20.0},
 {u'_id': {u'date': 10, u'tags': {u'category': u'1', u'host': u'host2'}},
  u'value': 20.0},
 {u'_id': {u'date': 0, u'tags': {u'category': u'3', u'host': u'host1'}},
  u'value': 20.0},
 {u'_id': {u'date': 10, u'tags': {u'category': u'2', u'host': u'host1'}},
  u'value': 30.0},
 {u'_id': {u'date': 0, u'tags': {u'category': u'1', u'host': u'host1'}},
  u'value': 10.0}]

You can even combine wildcard tag value with custom tag value.

Run tests

In order to run tests, you can:

• Run tests files by hand.
• Install nosetests (pip install nose) and run it from project root.

Contribute

If you'd like to contribute, simply fork the repository, commit your changes to the develop branch (or branch off of it), and send a pull request. Make sure you add yourself to AUTHORS.

Copyright and license

Copyright 2011-2012 FELD Boris

Licensed under the GPL License, Version 3; you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:

http://www.gnu.org/licenses/gpl-3.0.txt