DB: pick a single DB

[tac0turtle]

Summary

Pick one db and optimize for our needs.

Proposal

Remove usage of tm-db and move the DB interface internally to lazyledger core. If we pick badger as our DB we can migrate the current implementation (https://github.com/tendermint/tm-db/tree/master/badgerdb) to our repo.

After this is done we should try to use the features of a specific DB.

This work includes removing suport for multiple dbs within e2e tests.

Tendermint is also looking at doing something similar to this. We can collaborate with them and possibly upstream the work.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[liamsi]

related:

• Supported database backends tendermint/tm-db#38
• Tendermint shows time dependent performance degredation tendermint/tendermint#1835 (comment)

[liamsi]

If we pick badger as our DB we can migrate the current implementation (https://github.com/tendermint/tm-db/tree/master/badgerdb) to our repo.

So you are basically suggesting removing the tm-db abstraction, right? We should probably also use the latest badger release then: https://github.com/dgraph-io/badger/releases instead of the version in tm-db - I think we can update to github.com/dgraph-io/badger/v3 without any issues.

[tzdybal]

Why I like badger:

• actively developed
• widely used
• based on recent papers
• SSD optimized
• pure go
• Jepsen tests
• highly configurable/tunable
• performance

Personal note: I saw with my own eyes (while working at Dgraph) the emphasis on quality, durability and performance during development.

[tac0turtle]

If we pick badger as our DB we can migrate the current implementation (https://github.com/tendermint/tm-db/tree/master/badgerdb) to our repo.

So you are basically suggesting removing the tm-db abstraction, right? We should probably also use the latest badger release then: https://github.com/dgraph-io/badger/releases instead of the version in tm-db - I think we can update to github.com/dgraph-io/badger/v3 without any issues.

Yes, remove tm-db abstraction and that repo as a dependency. Absorb the db interface and badger implementation. Seems like the cleanest approach to me.

Agree with you @tzdybal

[Wondertan]

It is important to mention that after implementing a single BadgerV3 DB, we will still end up having two Badger instances in mem. One for LL's data and another for IPFS data, which unluckily uses V2 for now. There is a plan for an update, but there were no movements regarding that recently and it is not clear where this will be resolved. Furthermore, even after IPFS updates to the same version, we will still need to figure out the way two use only one instance of Badger, as it's intended to be used.

@tzdybal, would love to hear the comment from you regarding the use of two Badgers, as you worked closely with it.

[liamsi]

Good point! I think It's worth opening a specific issue (or a discussion) for this. Later down the road we ideally would only use one data store for all the things. For now, having two badger instances isn't worse than before (it's only more apparent bc we use badger for both ipfs and tendermint's store now).