Bolt 1: Specify that extensions to existing messages must use TLV (#754)

The spec already prepared a hook to add additional information to existing
messages (additional bytes at the end of a message must be ignored).

Since we're using TLV in many places, it makes sense to use that optional
additional space at the end of each message to allow an optional tlv stream.

This requires making a few previously optional fields mandatory:

- channel_reestablish commitment points: it makes sense to always include those
  regardless of whether `option_dataloss_protect` or `option_static_remotekey` are set.
- option_upfront_shutdown_script: if you're not using one, just set the length to 0.
  That field is moved to a TLV record because luckily, the resulting bytes are the same.
  This provides more flexibility to later remove the requirement of making this field mandatory.

No need to change the `channel_update`'s `htlc_maximum_msat` because
the `message_flags` encode its presence/absence.
It can still be either included or omitted without causing issues to the extension stream.

01-messaging.md

@@ -22,6 +22,7 @@ All data fields are unsigned big-endian unless otherwise specified.
     * [The `ping` and `pong` Messages](#the-ping-and-pong-messages)
   * [Appendix A: BigSize Test Vectors](#appendix-a-bigsize-test-vectors)
   * [Appendix B: Type-Length-Value Test Vectors](#appendix-b-type-length-value-test-vectors)
+  * [Appendix C: Message Extension](#appendix-c-message-extension)
   * [Acknowledgments](#acknowledgments)
   * [References](#references)
   * [Authors](#authors)
@@ -37,20 +38,12 @@ After decryption, all Lightning messages are of the form:
 1. `type`: a 2-byte big-endian field indicating the type of message
 2. `payload`: a variable-length payload that comprises the remainder of
    the message and that conforms to a format matching the `type`
+3. `extension`: an optional [TLV stream](#type-length-value-format)
 
 The `type` field indicates how to interpret the `payload` field.
 The format for each individual type is defined by a specification in this repository.
 The type follows the _it's ok to be odd_ rule, so nodes MAY send _odd_-numbered types without ascertaining that the recipient understands it.
 
-A sending node:
-  - MUST NOT send an evenly-typed message not listed here without prior negotiation.
-
-A receiving node:
-  - upon receiving a message of _odd_, unknown type:
-    - MUST ignore the received message.
-  - upon receiving a message of _even_, unknown type:
-    - MUST fail the channels.
-
 The messages are grouped logically into five groups, ordered by the most significant bit that is set:
 
   - Setup & Control (types `0`-`31`): messages related to connection setup, control, supported features, and error reporting (described below)
@@ -61,10 +54,9 @@ The messages are grouped logically into five groups, ordered by the most signifi
 
 The size of the message is required by the transport layer to fit into a 2-byte unsigned int; therefore, the maximum possible size is 65535 bytes.
 
-A node:
-  - MUST ignore any additional data within a message beyond the length that it expects for that type.
-  - upon receiving a known message with insufficient length for the contents:
-    - MUST fail the channels.
+A sending node:
+  - MUST NOT send an evenly-typed message not listed here without prior negotiation.
+  - MUST NOT send evenly-typed TLV records in the `extension` without prior negotiation.
   - that negotiates an option in this specification:
     - MUST include all the fields annotated with that option.
   - When defining custom messages:
@@ -75,6 +67,21 @@ A node:
     - SHOULD pick an even `type` identifiers when regular nodes should reject
       the message and close the connection.
 
+A receiving node:
+  - upon receiving a message of _odd_, unknown type:
+    - MUST ignore the received message.
+  - upon receiving a message of _even_, unknown type:
+    - MUST close the connection.
+    - MAY fail the channels.
+  - upon receiving a known message with insufficient length for the contents:
+    - MUST close the connection.
+    - MAY fail the channels.
+  - upon receiving a message with an `extension`:
+    - MAY ignore the `extension`.
+    - Otherwise, if the `extension` is invalid:
+      - MUST close the connection.
+      - MAY fail the channels.
+
 ### Rationale
 
 By default `SHA2` and Bitcoin public keys are both encoded as
@@ -85,8 +92,10 @@ Length is limited to 65535 bytes by the cryptographic wrapping, and
 messages in the protocol are never more than that length anyway.
 
 The _it's ok to be odd_ rule allows for future optional extensions
-without negotiation or special coding in clients. The "ignore
-additional data" rule similarly allows for future expansion.
+without negotiation or special coding in clients. The _extension_ field
+similarly allows for future expansion by letting senders include additional
+TLV data. Note that an _extension_ field can only be added when the message
+`payload` doesn't already fill the 65535 bytes maximum length.
 
 Implementations may prefer to have message data aligned on an 8-byte
 boundary (the largest natural alignment requirement of any type here);
@@ -896,6 +905,27 @@ failure:
 1. Invalid stream: 0xffffffffffffffffff 00 00 00
 2. Reason: valid TLV records but invalid ordering
 
+## Appendix C: Message Extension
+
+This section contains examples of valid and invalid extensions on the `init`
+message. The base `init` message (without extensions) for these examples is
+`0x001000000000` (all features turned off).
+
+The following `init` messages are valid:
+
+- `0x001000000000`: no extension provided
+- `0x00100000000001012a030104`: the extension contains two _odd_ TLV records (with types `0x01` and `0x03`)
+
+The following `init` messages are invalid:
+
+- `0x00100000000001`: the extension is present but truncated
+- `0x00100000000002012a`: the extension contains unknown _even_ TLV records (assuming that TLV type `0x02` is unknown)
+- `0x001000000000010101010102`: the extension TLV stream is invalid (duplicate TLV record type `0x01`)
+
+Note that when messages are signed, the _extension_ is part of the signed bytes.
+Nodes should store the _extension_ bytes even if they don't understand them to
+be able to correctly verify signatures.
+
 ## Acknowledgments
 
 [ TODO: (roasbeef); fin ]

02-peer-protocol.md

@@ -123,8 +123,13 @@ the funding transaction and both versions of the commitment transaction.
    * [`point`:`htlc_basepoint`]
    * [`point`:`first_per_commitment_point`]
    * [`byte`:`channel_flags`]
-   * [`u16`:`shutdown_len`] (`option_upfront_shutdown_script`)
-   * [`shutdown_len*byte`:`shutdown_scriptpubkey`] (`option_upfront_shutdown_script`)
+   * [`open_channel_tlvs`:`tlvs`]
+
+1. tlvs: `open_channel_tlvs`
+2. types:
+    1. type: 0 (`upfront_shutdown_script`)
+    2. data:
+        * [`...*byte`:`shutdown_scriptpubkey`]
 
 The `chain_hash` value denotes the exact blockchain that the opened channel will
 reside within. This is usually the genesis hash of the respective blockchain.
@@ -207,9 +212,11 @@ The sending node:
   - MUST set `channel_reserve_satoshis` greater than or equal to `dust_limit_satoshis`.
   - MUST set undefined bits in `channel_flags` to 0.
   - if both nodes advertised the `option_upfront_shutdown_script` feature:
-    - MUST include either a valid `shutdown_scriptpubkey` as required by `shutdown` `scriptpubkey`, or a zero-length `shutdown_scriptpubkey`.
+    - MUST include `upfront_shutdown_script` with either a valid `shutdown_scriptpubkey` as required by `shutdown` `scriptpubkey`, or a zero-length `shutdown_scriptpubkey` (ie. `0x0000`).
   - otherwise:
-    - MAY include a`shutdown_scriptpubkey`.
+    - MAY include `upfront_shutdown_script`.
+  - if it includes `open_channel_tlvs`:
+    - MUST include `upfront_shutdown_script`.
 
 The sending node SHOULD:
   - set `to_self_delay` sufficient to ensure the sender can irreversibly spend a commitment transaction output, in case of misbehavior by the receiver.
@@ -291,8 +298,13 @@ funding transaction and both versions of the commitment transaction.
    * [`point`:`delayed_payment_basepoint`]
    * [`point`:`htlc_basepoint`]
    * [`point`:`first_per_commitment_point`]
-   * [`u16`:`shutdown_len`] (`option_upfront_shutdown_script`)
-   * [`shutdown_len*byte`:`shutdown_scriptpubkey`] (`option_upfront_shutdown_script`)
+   * [`accept_channel_tlvs`:`tlvs`]
+
+1. tlvs: `accept_channel_tlvs`
+2. types:
+    1. type: 0 (`upfront_shutdown_script`)
+    2. data:
+        * [`...*byte`:`shutdown_scriptpubkey`]
 
 #### Requirements
 
@@ -1127,16 +1139,15 @@ messages are), they are independent of requirements here.
    * [`channel_id`:`channel_id`]
    * [`u64`:`next_commitment_number`]
    * [`u64`:`next_revocation_number`]
-   * [`32*byte`:`your_last_per_commitment_secret`] (option_data_loss_protect,option_static_remotekey)
-   * [`point`:`my_current_per_commitment_point`] (option_data_loss_protect,option_static_remotekey)
+   * [`32*byte`:`your_last_per_commitment_secret`]
+   * [`point`:`my_current_per_commitment_point`]
 
 `next_commitment_number`: A commitment number is a 48-bit
 incrementing counter for each commitment transaction; counters
 are independent for each peer in the channel and start at 0.
 They're only explicitly relayed to the other node in the case of
 re-establishment, otherwise they are implicit.
 
-
 ### Requirements
 
 A funding node:
@@ -1178,16 +1189,14 @@ The sending node:
   next `revoke_and_ack` message it expects to receive.
   - if `option_static_remotekey` applies to the commitment transaction:
     - MUST set `my_current_per_commitment_point` to a valid point.
-  - otherwise, if it supports `option_data_loss_protect`:
+  - otherwise:
     - MUST set `my_current_per_commitment_point` to its commitment point for
-      the last signed commitment it received from its channel peer (i.e. the commitment_point 
+      the last signed commitment it received from its channel peer (i.e. the commitment_point
       corresponding to the commitment transaction the sender would use to unilaterally close).
-  - if `option_static_remotekey` applies to the commitment transaction, or the sending node supports `option_data_loss_protect`:
-    - if `next_revocation_number` equals 0:
-      - MUST set `your_last_per_commitment_secret` to all zeroes
-    - otherwise:
-      - MUST set `your_last_per_commitment_secret` to the last `per_commitment_secret`
-    it received
+  - if `next_revocation_number` equals 0:
+    - MUST set `your_last_per_commitment_secret` to all zeroes
+  - otherwise:
+    - MUST set `your_last_per_commitment_secret` to the last `per_commitment_secret` it received
 
 A node:
   - if `next_commitment_number` is 1 in both the `channel_reestablish` it
@@ -1228,10 +1237,9 @@ A node:
       - MUST NOT broadcast its commitment transaction.
       - SHOULD fail the channel.
     - otherwise:
-	  - if `your_last_per_commitment_secret` does not match the expected values:
+      - if `your_last_per_commitment_secret` does not match the expected values:
         - SHOULD fail the channel.
-  - otherwise, if it supports `option_data_loss_protect`, AND the `option_data_loss_protect`
-  fields are present:
+  - otherwise, if it supports `option_data_loss_protect`:
     - if `next_revocation_number` is greater than expected above, AND
     `your_last_per_commitment_secret` is correct for that
     `next_revocation_number` minus 1: