Hydra node software,scripts & protocol updates

[ch1bo]

What & Why

How we deal with changes and upgrades to various parts of the Hydra system. This is only about changes we issue, for Cardano network changes, see: #195

We should distinguish breaking and non-breaking changes. Definition of breaking change could be: An already existing Hydra Head becomes unavailable after an update.

Given this definition we can identify at least these steps on a dimensional plan:

• Dirt: All changes are breaking changes.
• Cobblestone: Not all changes (e.g. to the API) are breaking changes and deprecation cycles are employed.
• Highway: Major versions are backward compatible and can migrate Heads by re-opening.

To address this, we want to

• Prevent accidental changes to interfaces
• Detect & inform about incompatibilities
• Define what changes are breaking in what component and how they will impact UX

Scenarios (examples)

• A security relevant bug in the Hydra scripts has been fixed
• A new desired endpoint has been added to the client API
• Changing what we log in the hydra node
• Incremental de-/commits (protocol extension) has been implemented
• Changed/added performant network layer between Hydra nodes has been introduced
• Off-chain protocol has been changed
• The encoding of network messages exchanged between Hydra nodes changed
• Format of the head-state on disk changed

Versioned entities

• Hydra node executable / docker image

  • Client API
  • Network API
  • Cardano API (which era, which Tx format)
  • Hydra on-chain scripts (i.e. hydra-plutus)

• Hydra TUI (example client)

• Hydraw and other applications

How (TBD, incomplete)

• Ensured that the hydra-plutus scripts do not change accidentally (store script binaries and have golden tests on script hashes)
• Add a major version to the network API (to at least notify version misalignment) Use versioned handshake when connecting hydra-nodes #1010
• Ensure that release notes contain instructions on how to deal with upgrades of hydra-node, especially the ones which require manual intervention (e.g. closing/opening Heads)
• Detect JSON schema breakage of logs reliably
• Detect era mismatch on received blocks from Cardano Conway support #1177
• A change in Hydra scripts is always a major change of hydra-node
• Changes to the APIs may be breaking changes, but only after a deprecation cycle of one minor version

[ch1bo]

It seems like all changes to plutus scripts need to be treated as "breaking changes", i.e. a Hydra Head would need to be closed and re-opened. Why? When implementing the V1 script, we would not know the V2 script hash already and hence we could not make it "forward compatible".

Furthermore, any change to the scripts and the upstream components like the plutus-tx compiler will result in changes to the script hashes. So we ought to prevent accidental change!

[ch1bo]

We discussed and detailed the description of this feature a bit today. The on-chain scripts really feel a bit like "shipping hardware" as they won't be upgradable (without introducing authority) and we better get them right / good enough when shipping a 1.0.0.

Besides that, it might be enough to contain those breaking changes and not accidentally change them.

Open question just coming to mind when writing this: What situations where we add significant (but independent of on-chain scripts) improvements to the hydra-node, and some people want/need to use them with V1 and V2 of the scripts? Do we need to maintain two branches of hydra-node?

[ch1bo]

Running a hydra-node which expects Babbage era on the chain layer against an Alonzo testnet produces this error:

hydra-node: HardForkEncoderDisabledEra (SingleEraInfo {singleEraName = "Babbage"})

[ch1bo]

When implementing the V1 script, we would not know the V2 script hash already and hence we could not make it "forward compatible".

An interesting thread in that direction can be found here: https://input-output-rnd.slack.com/archives/C21UF2WVC/p1656494873725559 (ask @ch1bo for details)

The gist:

i.e. authorizing a governance output to enumerate the target (V2) script hashes in the datum and allow re-spending script outputs there while maintaining an identical datums (a.k.a. self-transition in a utxo state machine).

[ch1bo]

We have created a golden test suite which persists scripts as binary data in the repository and detects any changes to them: #772

[ch1bo]

We had a look at this again and this is / was always not much actionable. It turns out, however, that we navigated all mentioned scenarios in the past and implemented most of the regression and version checks anyhow. Only remaining would be #1010, which supersedes this item on our roadmap.