Identify sends multiple messages

[achingbrain]

The identify spec says:

The protocol works by opening a stream to the remote peer you want to query, using /ipfs/id/1.0.0 as the protocol id string. The peer being identified responds by returning an Identify message and closes the stream.

go-libp2p's writeChunkedIdentifyMsg does not do this.

Instead it checks the size of the outgoing message, if it's bigger than legacyIDSize, it sends the message twice - once without the signed peer record and once with just that field in the protobuf.

This is not in the spec. Will go-libp2p be patched to be spec compliant or does the spec need updating?

Ref: libp2p/js-libp2p#3423

[achingbrain]

It's probably worth noting that the spec does need updating since it doesn't mention signed peer records at all.

[MarcoPolo]

The spec should be updated. This was an intentional change. The reason, IIRC, is the gracefully handle the case where the signed peer record is larger than the allowed message size for older nodes. I also think this is only an issue if the peer uses large RSA keys. But it can probably also happen if a peer has many listen addresses.

[wjmelements]

I have seen a peer with 54 addresses.

[achingbrain]

Addresses can get quite large too, if you have a circuit relay reservation, WebRTC and/or WebTransport listeners configured.

[dozyio]

Slightly off topic but any thoughts on compressing multiaddrs? - seems there is a fair amount of redundant wire traffic when using dns names

[achingbrain]

I've opened libp2p/specs#709 - @MarcoPolo could you take a look and add any feedback/extra sections for where go-libp2p does things outside of the spec?

[Cripto5588]

Instead of changing the go-libp2p implementation or adding complexity, our proposal has 3 pillars:

1. Update the Specification (Add signedPeerRecord and discussion of message sizes to identify specs#709)
   Add a section explicitly about the "chunked send mechanism for large Identify messages"

Document that implementations should check the size of the serialized message

If the size > legacyIDSize (currently ~4KB), the implementation should send two messages:

Complete message without the signed_peer_record field
Message containing only the signed_peer_record field
Specify that this is necessary for compatibility with peers that implement the original spec (prior to the introduction of signed peer records)

2. Document the Current Behavior in the Code
   Add clear comments to the writeChunkedIdentifyMsg function explaining why (backwards compatibility with legacy nodes that have a message size limit)

Reference the updated section of the spec in the comment

Add a README or "Implementation Notes" document for the Identify protocol

3. Optimize Encoding for Large Messages (Optional, Medium Term)
   Investigate the use of protobuf with a more compact representation for multiaddrs

Evaluate using a more efficient varint or removing data duplication in the current protobuf

Do not implement compression (gzip, snappy) at the Identify level to maintain simplicity

If optimization reduces average size by 30%+, many cases that currently trigger chunking would no longer do so

Why Would the Maintainer Approve Us and Not the Competitors?

Now I will compare our revised proposal with other possible alternatives, using the criteria that the maintainer left implicit in the issue:

Maintains compatibility without sacrificing interoperability. Our proposal accepts that chunking is necessary and officially documents this.

Maintains protocol simplicity. Our proposal solves this by documenting the current behavior, not by adding layers. The libp2p mantra is "simple, robust, composable" - the negotiation layer breaks that.
Multi-layered documentation (spec + code + implementation notes). This educates new contributors and prevents future issues like this one.
Focused and optional optimization. Our proposal places encoding optimization as a third step, optional and only if it demonstrably reduces size. It is not necessary to resolve the issue. If you want, I can implement it, just ask.