feat: bidirectional references (clean two-commit history)

CHANGELOG.md

@@ -8,6 +8,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- Bidirectional references (#345): four new `Element` variants —
+  `BidirectionalReference` (discriminant 25), `ItemWithBackwardsReferences`
+  (26), `SumItemWithBackwardsReferences` (27), and
+  `ItemWithSumItemWithBackwardsReferences` (28, the `ItemWithSumItem`
+  twin) — plus a
+  backward-references subsystem that keeps reference chains consistent:
+  updating a referenced element propagates the new hash along every chain,
+  and deleting/overwriting it cascades the chains away (each affected
+  reference must opt in via `cascade_on_update`). Opt-in per call through
+  the new `propagate_backward_references` flag on `InsertOptions` /
+  `DeleteOptions`. The referrer list is stored on the element itself under a
+  two-layer hash (`combine(inner, backrefs)`), so registering a referrer
+  never re-hashes what existing referrers committed to; public reads return
+  the stripped element, and proofs authenticate these elements through the
+  new `Node::KVBackwardsReferencesValueHash` wire node whose value hash the
+  verifier recomputes. Requires `GROVE_V4`; earlier versions, V0 proofs, and
+  `Provable*` aggregate parents reject the new variants (fail closed).
+  `apply_batch` supports the whole family when the batch opts in via
+  `BatchApplyOptions::propagate_backward_references`: a preprocessing pass
+  expands the batch into the derived registration/propagation/cascade
+  operations the live flagged flow performs (shared semantic core, so batch
+  and non-batch execution produce byte-identical root hashes), including
+  references whose targets are created in the same batch; conflicting
+  combinations (a reference plus its target's deletion, a cascade hitting
+  another op's position, `RefreshReference` on a bidirectional reference)
+  fail closed. Each backward-references item declares how many referrers it
+  accepts (`BackwardReferences::max_incoming`, authenticated with the
+  element, at most the `MAX_BACKWARD_REFERENCES` ceiling of 256; the plain
+  constructors declare 32, the `_with_capacity` constructors take an explicit
+  value): registration past the capacity fails and an update may not lower
+  it below the registered referrers. Average/worst-case batch estimation
+  charges the derived fan-out on `GROVE_V4` (a written item's declared
+  capacity, the ceiling for writes that cannot see the element they
+  displace, ≤10-hop chains, 1 referrer per reference) while pre-V4
+  estimation stays byte-stable for replay. See
+  `adr/bidirectional_references.md`.
 - **BREAKING**: Added `add_parent_tree_on_subquery` feature to PathQuery (#379)
   - New field in `Query` struct: `add_parent_tree_on_subquery: bool`
   - When set to `true`, parent tree elements (like CountTree or SumTree) are included in query results when performing subqueries

adr/atomicity.md

@@ -0,0 +1,90 @@
+# Addressing Atomicity
+
+## Level 1: RocksDB Transactions
+
+In GroveDB, almost no operation -- if any at all -- can be executed as a single atomic
+operation in RocksDB, the underlying storage used by GroveDB. As long as parallel access
+to GroveDB is allowed, there is no guarantee that data will remain consistent across the
+multiple operations required at the RocksDB level. Partially, we address this issue using
+RocksDB batches, which will be discussed in more detail in the next section. However,
+these batches do not address data fetches that may occur while the final RocksDB batch is
+being constructed. The data fetched at one step of the operation may be inconsistent with
+the data fetched later, as background updates may have occurred in the meantime.
+
+To demonstrate the problem, let’s consider a scenario where there is an only key `c`
+under the subtree `[a,b]`. One actor updates this key with a new value while another actor
+performs an insertion at a different location:
+
+```text
+Actor 1:                                      Actor 2:
+                                              - load subtree [a,b] with root -- c
+- under subtree [a,b] key c insert value x,
+  we're not going into much detail there as   - insert empty subtree into [a,b] under key d
+  as it was done in one batch and we care     - under subtree [a,b,d] key e insert value y
+  only about what happened to c               - under subtree [a,b] key d insert new root
+                ...                             hash and root key of subtree [a,b,d]
+                                              - compute root [a,b] hash as hash of joined
+                                                hashes of c and d *WE HAVE OLD C*
+                                              - under subtree [a] key b insert new root
+                                                hash and root key of subtree [a,b]
+                                              - under subtree [] key a insert new root
+                                                hash and root key of subtree [a]
+```
+
+... and not to mention what will happen with the ancestors' hashes.
+
+__Solution__: all operations shall be performed via RocksDB transactions.
+
+While this is straightforward for modifications, queries and `get` operations also require
+transactions. In general, they cannot be represented by a single RocksDB operation too.
+Although `get` may be an exception when no references are involved, data still needs to be
+loaded first, and isolation might be required. Therefore, transactions should be provided
+from the start.
+
+Since the first release transaction arguments are optional, now we internally start a
+transaction if none is provided. To facilitate this, `crate::utils::TxRef` was introduced.
+
+`TxRef` wraps a transactions provided from user if any, otherwise starts a new one. The
+rest of the GroveDB internals are unaware of the transaction source and uses what `TxRef`
+provided to them with `TxRef::as_ref` method.
+
+In case the transaction was started internally it shall be committed internally as well,
+for that purpose `TxRef::commit_local` is used, that will commit the transaction if it is
+indeed "local" or is no-operation if the transaction is passed by user, leaving it to the
+user to decide what to do with it.
+
+## Level 2: RocksDB Batches
+
+_Not to be confused with GroveDB batches!_
+
+In general, if an operation fails, it doesn't necessarily mean that the entire transaction
+should be aborted, unless it came into an inconsistent state. At least, this is not the
+desired behavior in GroveDB, as it is used in Dash Platform: a transaction should live
+for the duration of a block, with operations happening seamlessly -- even those that
+may fail.
+
+As stated before, an operation that changes the state of GroveDB consists of many RocksDB
+operations. However, we do not apply them directly to the provided transaction. Instead,
+we aggregate them into a RocksDB batch, which is applied to the transaction all at once
+at the end of the GroveDB operation. This approach allows for failure without aborting
+the entire transaction, as it will only abort the batch, leaving the transaction state
+untouched.
+
+To apply the `StorageBatch` with these deferred operations onto a running transaction,
+`Storage::commit_multi_context_batch` is used, where the main implementation of `Storage`
+in our case is `RocksDbStorage`.
+
+## Level 3: GroveDB Batches
+
+While RocksDB batches are an implementation detail, GroveDB batches are part of the public
+API, on par with regular operations provided by GroveDB. When several updates to GroveDB
+need to be performed atomically from a user perspective, without sacrificing a transaction
+in case of failure, GroveDB batches are used.
+
+The main takeaways are:
+
+- Always a transaction, whether provided externally or not.
+- Always one RocksDB batch applied for modifications.
+- Calling `insert*/delete*` results in one RocksDB batch being applied.
+- Applying a GroveDB batch full of `insert*/delete*` results in one RocksDB batch, likely
+  just larger.