Document Mina daemon's databases

[georgeee]

Produce a document describing what are databases that Daemon uses.

Result of this task is expected to be a table of databases and descriptions.

Table's rows, I imagine, would be:

• Location of database
• Description
• Ocaml's type for key
• Ocaml's type for value

Relates to #13971

[georgeee]

This document is needed to analyze usage of RocksDB in block production/processing.

Major focus should be on frontier and ledger-related DBs.

[georgeee]

AFAIU all DBs we have are key-value storages.

And we probably have more than one key-value space within some of these DBs, let's have one "key-value space" per row in a resuting table.

To start with the task I suggest one launching a Mina node that connects to mainnet. And then checking what you have in .mina-config.
Ideally we'd be able to inspect every DB with manual tool and check that there are no keys unaccounted, but it might be too tedious, so maybe just invetigating usages in codebase would be alright.

What I see on a mainnet's node:

• genesis/<...> <- three genesis ledger folders
• frontier <- transition frontier, where the recent blocks are kept
• wallets <- for locally stored secret keys, empty if no key configured
• mina_net2/<..> <- some databases used by networking implementation
  • mina_net2/block_db <- lmdb database for blocks, to be used in future by the new catchup algorithm
• root <- some data on the "root" transition, including uncommitted snarked ledgers
• trust <- database to store information about peers, now it's effectively containing peers we know and whether a peer is banned until some datetime

[Geometer1729]

root

The root/snarked_ledger location is defined here:

mina/src/lib/transition_frontier/persistent_root/persistent_root.ml

Line 40 in 4495af5

let snarked_ledger root = Filename.concat root "snarked_ledger"

This path is given to Ledger.Db where Ledger is Mina_ledger.Ledger.
So the snarked_ledger Db type gets defined here:

mina/src/lib/mina_ledger/ledger.ml

Lines 108 to 120 in 4495af5

	module Db :
	Merkle_ledger.Intf.Ledger.DATABASE
	with module Location = Location_at_depth
	with module Addr = Location_at_depth.Addr
	with type root_hash := Ledger_hash.t
	and type hash := Ledger_hash.t
	and type key := Public_key.Compressed.t
	and type token_id := Token_id.t
	and type token_id_set := Token_id.Set.t
	and type account := Account.t
	and type account_id_set := Account_id.Set.t
	and type account_id := Account_id.t =
	Database.Make (Inputs)

which calls this functor:

mina/src/lib/merkle_ledger/database.ml

Line 1 in 4495af5

module Make (Inputs : Intf.Inputs.DATABASE) = struct

On this module:

mina/src/lib/mina_ledger/ledger.ml

Lines 89 to 106 in 4495af5

	module Inputs = struct
	module Key = Public_key.Compressed
	module Token_id = Token_id
	module Account_id = Account_id
	module Balance = struct
	include Currency.Balance
	let to_int = to_nanomina_int
	end
	module Account = Account.Stable.Latest
	module Hash = Hash.Stable.Latest
	module Kvdb = Kvdb
	module Location = Location_at_depth
	module Location_binable = Location_binable
	module Storage_locations = Storage_locations
	end

The Kvbd.t type it takes seems to be a generic database type with Bigstring.t for both key and value.
But from calls to Kvdb like this:

mina/src/lib/merkle_ledger/database.ml

Lines 101 to 102 in 4495af5

	let get_raw { kvdb; depth; _ } location =
	Kvdb.get kvdb ~key:(Location.serialize ~ledger_depth:depth location)

It looks like the keys represent serialized Location.ts which I think is this enum:

mina/src/lib/merkle_ledger/location.ml

Lines 54 to 55 in 4495af5

	type t = Generic of Bigstring.t | Account of Addr.t | Hash of Addr.t
	[@@deriving hash, sexp, compare]

and the value appears to be the Toeken_id.Set.t type from the inputs.
I'm not sure where Token_id comes from, there are a few opens at the top of the file.

The root/root location is defined here and seems to store a hash of the genesis state.

[Geometer1729]

trust

src/lib/trust_system/peer_trust.ml

Has the key as Peer_id.t which I believe refers to this:

mina/src/lib/trust_system/peer_trust.ml

Lines 252 to 256 in 10a0bf9

	module Peer_id = struct
	type t = int [@@deriving sexp, yojson]
	let ip t = Unix.Inet_addr.of_string (sprintf "127.0.0.%d" t)
	end

and the value Record.t which I beleive reffers to this

mina/src/lib/trust_system/record.ml

Lines 5 to 12 in 10a0bf9

	module V1 = struct
	type t =
	{ trust : float
	; trust_last_updated : Core.Time.Stable.V1.t
	; banned_until_opt : Core.Time.Stable.V1.t option
	}
	let to_latest = Fn.id

[Geometer1729]

frontier

It looks like this module:

mina/src/lib/transition_frontier/persistent_frontier/database.ml

Line 212 in 4495af5

module Rocks = Rocksdb.Serializable.GADT.Make (Schema)

calls this functor:

mina/src/lib/rocksdb/serializable.ml

Line 53 in 4495af5

module Make (Key : Intf.Key.S) : Intf.Database.S with type 'a g := 'a Key.t =

With this argument:

mina/src/lib/transition_frontier/persistent_frontier/database.ml

Line 29 in 4495af5

module Schema = struct

I think that results in the key type being this private Enum:

mina/src/lib/transition_frontier/persistent_frontier/database.ml

Lines 49 to 56 in 4495af5

	type _ t =
	| Db_version : int t
	| Transition : State_hash.Stable.V1.t -> Mina_block.Stable.V2.t t
	| Arcs : State_hash.Stable.V1.t -> State_hash.Stable.V1.t list t
	| Root : Root_data.Minimal.Stable.V2.t t
	| Best_tip : State_hash.Stable.V1.t t
	| Protocol_states_for_root_scan_state
	: Mina_state.Protocol_state.Value.Stable.V2.t list t

Here Root_data.Minimal.Stable.V2 refers to this type:

mina/src/lib/transition_frontier/frontier_base/root_data.ml

Line 114 in 4495af5

type t = { hash : State_hash.Stable.V1.t; common : Common.Stable.V2.t }

and I think this defines a type family like thing for what the value types is depending on the key:

mina/src/lib/transition_frontier/persistent_frontier/database.ml

Lines 74 to 86 in 4495af5

	let binable_data_type (type a) : a t -> a Bin_prot.Type_class.t = function
	| Db_version ->
	[%bin_type_class: int]
	| Transition _ ->
	[%bin_type_class: Mina_block.Stable.Latest.t]
	| Arcs _ ->
	[%bin_type_class: State_hash.Stable.Latest.t list]
	| Root ->
	[%bin_type_class: Root_data.Minimal.Stable.Latest.t]
	| Best_tip ->
	[%bin_type_class: State_hash.Stable.Latest.t]
	| Protocol_states_for_root_scan_state ->
	[%bin_type_class: Mina_state.Protocol_state.Value.Stable.Latest.t list]