Incremental decommit (#1344)

This PR adds "incremental decommits" to the Hydra Head protocol, which
allows users to take funds out of an open Head.

- New API endpoint `/decommit` which accepts a "Decommit transaction",
that spends some UTxO and whatever outputs it produces will be made
available on the L1. This can be also done through a new `Decommit`
client input and new server outputs `DecommitRequested`,
`DecommitApproved` and `DecommitFinalized`, as well as `DecommitInvalid`
to inform about status of the decommit.

- Decommits are first approved in a snapshot on L2 via a new network
message `ReqDec`, before a new `decrementTx` can be posted and observed
on-chain.

- Only one decommit can be processed at a given time.

- Update documentation and added how-to about how to use this.

- Acknowledged specification changes by "clearing" of
$\textcolor{red}{\\red}$ areas covered by this implementation in the
specification.

- End-to-end test covering the main scenario of decommitting funds.

- Added mutation tests for Decrement, Close and Contest to cover all
on-chain-verification changes.

- Enhanced `TxTrace` tests to test decrements with various snapshots and
their interaction with close/contest and fanout of a head.

---

* [x] CHANGELOG updated
* [x] Documentation updated
* [x] Haddocks updated
* [x] New TODOs explained hereafter


![image](https://github.com/user-attachments/assets/eed47f06-d519-42cb-a897-98397066fdd9)

  - Two FIXMEs covered by #1524
  - TODO in HeadLogic coverd by #1502
- TODO in tx-cost how we could improve the benchmark output (not
crucial)
- TODO in head logic about rollbacks .. actually something we need to
consider with #199 too

CHANGELOG.md

@@ -18,6 +18,11 @@ changes.
   - Removed the check that prevented committing UTxOs from an internal `hydra-node` wallet.
   - `SpendingNodeUtxoForbidden` error was removed.
 
+- Add capability to move UTxO out of an open Head to the Cardano main chain:
+  - Submitting a decommit transaction to `POST /decommit` or as `Decommit` command through websocket, requests removal of this transactions' outputs from the head.
+  - When successful, `DecommitApproved` and `DecommitFinalized` indicate that all outputs are made available on the layer one.
+  - Invalid transactions are explained through a `DecommitInvalid` server output.
+
 - Change `--start-chain-from` to always use the newer point when also a head state is known.
 
 - Moved several pages from "core concepts" into the user manual and developer docs to futher improve user journey.

docs/docs/dev/protocol.md

@@ -0,0 +1,43 @@
+# Protocol
+
+Additional implementation-specific documentation for the Hydra Head protocol and extensions like incremental decommits.
+
+### Incremental Decommits
+
+```mermaid
+sequenceDiagram
+    Alice->>HeadLogic: Decommit (decTx)
+    HeadLogic->>HeadLogic: canApply decTx
+
+    par broadcast
+        HeadLogic->>HeadLogic: ReqDec decTx
+    and
+        HeadLogic->>Node B: ReqDec decTx
+    end
+
+    HeadLogic -->> Alice: DecommitRequested
+
+    par Alice isLeader
+        HeadLogic->>HeadLogic: ReqSn decTx
+    and
+        HeadLogic->>Node B: ReqSn decTx
+    end
+
+    HeadLogic->>HeadLogic: canApply decTx, decUTxO =  outputs(decTx)
+    HeadLogic->>HeadLogic: sig = sign snapshot incl. decUTxO
+
+    par broadcast
+        HeadLogic->>HeadLogic: AckSn sig
+    and
+        HeadLogic->>Node B: AckSn sig
+    end
+
+    Node B->>HeadLogic: AckSn sig
+
+    HeadLogic -->> Alice: SnapshotConfirmed
+    HeadLogic -->> Alice: DecommitApproved
+
+    HeadLogic ->> Chain: DecrementTx snapshot sig
+    Chain ->> HeadLogic: OnDecrementTx
+    HeadLogic -->> Alice: DecommitFinalized
+```

docs/docs/how-to/event-sinks-and-sources.md

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 4
+sidebar_position: 5
 ---
 
 # Extend the node with event source and sinks

docs/docs/how-to/incremental-decommit.md

@@ -0,0 +1,115 @@
+---
+sidebar_position: 3
+---
+
+# Decommit funds
+
+To take out some `UTxO` present in an open head and send it back to the layer one, you need to do a so-called `decommit`.
+
+This how-to assumes that we are in a similar situation as in the [Getting Started](../getting-started) or [Testnet tutorial](../tutorial). Depending on who owns something in your head, you might need to update the instructions of this tutorial. In our example we decommit funds owned by Alice from their address:
+
+```shell
+export WALLET_SK=credentials/alice-funds.sk
+export WALLET_ADDR=addr_test1vp5cxztpc6hep9ds7fjgmle3l225tk8ske3rmwr9adu0m6qchmx5z
+```
+
+First, we need to find out which `UTxO` we can spend using our address:
+
+```shell
+curl localhost:4001/snapshot/utxo \
+  | jq "with_entries(select(.value.address == \"${WALLET_ADDR}\"))" \
+  > utxo.json
+```
+
+<details>
+<summary> Example output</summary>
+
+```json
+{
+  "f6b004be1cf95dbd3d0abc3daceac40ef6401e502972a919e5e52564b9f5740b#0": {
+    "address": "addr_test1vp5cxztpc6hep9ds7fjgmle3l225tk8ske3rmwr9adu0m6qchmx5z",
+    "datum": null,
+    "datumhash": null,
+    "inlineDatum": null,
+    "referenceScript": null,
+    "value": {
+      "lovelace": 50000000
+    }
+  },
+  "f6b004be1cf95dbd3d0abc3daceac40ef6401e502972a919e5e52564b9f5740b#1": {
+    "address": "addr_test1vp5cxztpc6hep9ds7fjgmle3l225tk8ske3rmwr9adu0m6qchmx5z",
+    "datum": null,
+    "datumhash": null,
+    "inlineDatum": null,
+    "referenceScript": null,
+    "value": {
+      "lovelace": 50000000
+    }
+  }
+}
+```
+
+</details>
+
+Now, the `decommit` command requires us to build a transaction that proves we can spend what we want to decommit. The outputs of this transaction will be the outputs that are also going to be made available on the main chain.
+
+For example, to spend the first UTxO queried above in a transaction sending the same value to Alice's key (so she can spend it on the layer one later):
+
+```shell
+LOVELACE=$(jq -r 'to_entries[0].value.value.lovelace' < utxo.json)
+cardano-cli transaction build-raw \
+  --tx-in $(jq -r 'to_entries[0].key' < utxo.json) \
+  --tx-out ${WALLET_ADDR}+${LOVELACE} \
+  --fee 0 \
+  --out-file decommit.json
+```
+
+You can inspect the transaction with
+
+```shell
+cardano-cli transaction view --tx-file decommit.json
+```
+
+As the transaction spends from Alices funds in the Hydra head, we also need to
+sign it with her key:
+
+```shell
+cardano-cli transaction sign \
+  --tx-file decommit.json \
+  --signing-key-file ${WALLET_SK} \
+  --out-file alice-decommit-tx-signed.json
+```
+
+With the signed decommit transaction, now we can submit it to the `/decommit` endpoint:
+
+```shell
+curl -X POST 127.0.0.1:4001/decommit \
+  --data @alice-decommit-tx-signed.json
+```
+
+<details>
+<summary>Alternative using websocket</summary>
+
+We can also submit a `Decommit` client input using a websocket:
+```shell
+cat alice-decommit-tx-signed.json \
+  | jq -c '{tag: "Decommit", decommitTx: .}' \
+  | websocat "ws://127.0.0.1:4001?history=no"
+```
+
+</details>
+
+If you haven't already, open a websocket session using `websocat ws://0.0.0.0:4001` now.
+
+In the message history you will see a `DecommitRequested` message which
+indicates a decommit is requested.
+
+After some time, a `DecommitFinalized` can be observed which concludes the decommit process and after which the funds are available on the layer one.
+
+To confirm, you can query the funds of the wallet on the layer one from a `cardano-node`:
+
+```shell
+cardano-cli query utxo \
+  --address ${WALLET_ADDR} \
+  --output-json | jq
+```

docs/docs/how-to/operating-hydra.md

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 3
+sidebar_position: 4
 ---
 
 # Operate a Hydra node

docs/sidebars.js

@@ -63,6 +63,7 @@ module.exports = {
       id: "dev/specification",
       label: "Specification",
     },
+    "dev/protocol",
     {
       type: "category",
       link: { type: "doc", id: "dev/architecture/index" },

fourmolu.yaml

@@ -11,6 +11,7 @@ fixities:
   - infixr 0 $
   - infixr 1 &
   - infixl 3 <|>
+  - infixr 2 ||
   - infixr 3 &&
   - infixl 1 <&>
   - infixl 4 <$>

hydra-cardano-api/src/Cardano/Api/UTxO.hs

@@ -85,6 +85,14 @@ render (k, TxOut _ (txOutValueToValue -> v) _ _) =
 min :: UTxO -> UTxO
 min = UTxO . uncurry Map.singleton . Map.findMin . toMap
 
+-- | Remove the right hand side from the left hand side.
+difference :: UTxO' out -> UTxO' out -> UTxO' out
+difference a b = UTxO $ Map.difference (toMap a) (toMap b)
+
+-- | Infix version of 'difference'.
+(\\) :: UTxO' out -> UTxO' out -> UTxO' out
+a \\ b = difference a b
+
 -- * Type Conversions
 
 -- | Transforms a UTxO containing tx outs from any era into Babbage era.

hydra-chain-observer/src/Hydra/ChainObserver.hs

@@ -92,6 +92,7 @@ data ChainObserverLog
   | HeadInitTx {headId :: HeadId}
   | HeadCommitTx {headId :: HeadId}
   | HeadCollectComTx {headId :: HeadId}
+  | HeadDecrementTx {headId :: HeadId}
   | HeadCloseTx {headId :: HeadId}
   | HeadFanoutTx {headId :: HeadId}
   | HeadAbortTx {headId :: HeadId}
@@ -205,6 +206,7 @@ chainSyncClient tracer networkId startingPoint observerHandler =
     OnInitTx{headId} -> HeadInitTx{headId}
     OnCommitTx{headId} -> HeadCommitTx{headId}
     OnCollectComTx{headId} -> HeadCollectComTx{headId}
+    OnDecrementTx{headId} -> HeadDecrementTx{headId}
     OnCloseTx{headId} -> HeadCloseTx{headId}
     OnFanoutTx{headId} -> HeadFanoutTx{headId}
     OnAbortTx{headId} -> HeadAbortTx{headId}

hydra-chain-observer/test/Hydra/ChainObserverSpec.hs

@@ -25,6 +25,7 @@ spec =
                     Just (Init{}) -> transition === Transition.Init
                     Just (Commit{}) -> transition === Transition.Commit
                     Just (CollectCom{}) -> transition === Transition.Collect
+                    Just (Decrement{}) -> transition === Transition.Decrement
                     Just (Abort{}) -> transition === Transition.Abort
                     Just (Close{}) -> transition === Transition.Close
                     Just (Contest{}) -> transition === Transition.Contest

hydra-cluster/src/CardanoClient.hs

@@ -88,14 +88,17 @@ waitForPayments networkId socket amount addr =
   selectPayments (UTxO utxo) =
     Map.filter ((== amount) . selectLovelace . txOutValue) utxo
 
+-- | Wait for transaction outputs with matching lovelace value and addresses of
+-- the whole given UTxO
 waitForUTxO ::
-  NetworkId ->
-  SocketPath ->
+  RunningNode ->
   UTxO ->
   IO ()
-waitForUTxO networkId nodeSocket utxo =
+waitForUTxO node utxo =
   forM_ (snd <$> UTxO.pairs utxo) forEachUTxO
  where
+  RunningNode{networkId, nodeSocket} = node
+
   forEachUTxO :: TxOut CtxUTxO -> IO ()
   forEachUTxO = \case
     TxOut (ShelleyAddressInEra addr@ShelleyAddress{}) value _ _ -> do