Rust Docs + refactor: protocols crates

[plebhash]

We recently published our protocols crates to crates.io.
We need to make sure the APIs provided by those crates are properly documented under the Rust Best Practices.

This is also the perfect opportunity to map and address technical debt on those crates. As we go through the code and try to understand it, we take the chance to write down a list of ways the code could be improved. Things such as:

• unintuitive, redundant, broken or suboptimal APIs
• unsafe error handling (unwrap, panic!, expect)
• typos, bad variable names
• // todo comments, todo! macros, edge cases with unimplemented! or panic!

---

Task list

Rust Docs

• v1 (already documented but worth a look before proceding)
  • INPUT: Q&A Discussion:: Questions about `sv1_api` crate #897
  • OUTPUT: not planned
• v2/binary-sv2/binary-sv2
  • INPUT: Q&A Discussion: todo
  • INPUT: Rust Docs Issue Need Rust docs on protocols::v2::binary_sv2::binary_sv2 #1008
  • OUTPUT: Rust Docs PR: todo
  • OUTPUT: Technical Debt issue: Refactor: binary-sv2::binary-sv2 crate #1131
  • OUTPUT: Refactoring PRs: todo
• v2/binary-sv2/no-serde-sv2/codec
  • INPUT: Q&A Discussion: todo
  • INPUT: Rust Docs Issue: Need Rust docs on protocols::v2::binary_sv2::no-serde-sv2::codec #1009
  • OUTPUT: Rust Docs PRs:
    • doc: Document Seq064K #1046 (small doc additions)
  • OUTPUT: Technical Debt issue: Refactor: binary-sv2::no-serde-sv2::codec crate #1132
  • OUTPUT: Refactoring PRs: todo
• v2/binary-sv2/no-serde-sv2/derive_codec
  • INPUT: Q&A Discussion: todo
  • INPUT: Rust Docs Issue: Need Rust docs on protocols::v2::binary_sv2::no-serde-sv2::derive_codec #1010
  • OUTPUT: Rust Docs PR: todo
  • OUTPUT: Technical Debt issue: Need Rust docs on protocols::v2::binary_sv2::no-serde-sv2::derive_codec #1010
  • OUTPUT: Refactoring PRs: todo
• v2/binary-sv2/serde-sv2
  • INPUT: Q&A Discussion: not planned
  • INPUT: Rust Docs Issue: not planned Need Rust docs on protocols::v2::binary_sv2::serde_sv2 #1011
  • OUTPUT: Rust Docs PR: not planned
  • OUTPUT: Technical Debt issue: not planned Refactor: binary-sv2::serde-sv2 crate #1134
  • OUTPUT: Refactoring PRs: not planned
• v2/const-sv2
  • INPUT: Q&A Discussion: Questions about `const_sv2` crate #1159
  • INPUT: Rust Docs Issue: Need Rust docs on protocols::v2::const-sv2 #1016
  • OUTPUT: Rust Docs PR: Rust docs for protocols::v2::const-sv2 #1160
  • OUTPUT: Technical Debt issue: Refactor: const-sv2 crate #1135
  • OUTPUT: Refactoring PRs: todo
• v2/framing-sv2
  • INPUT: Q&A Discussion: Questions about `framing_sv2` crate #858
  • INPUT: Rust Docs Issue: no tracker issue was created (we should avoid skipping for other crates)
  • OUTPUT: Rust Docs PR: Rust Docs framing_sv2 #848
  • OUTPUT: Technical Debt issue: refactor: framing_sv2 crate #903
  • OUTPUT: Refactoring PRs:
    • Framing refactor: framing.rs api #1033
    • Framing refactor: test Sv2Frame and HandShakeFrame #1049
    • Framing crate refactor: cleanup header.rs and remove Frame trait #976
    • Framing crate refactor: cleanup framing.rs  #982
• v2/codec-sv2
  • INPUT: Q&A Discussion: Questions about `codec_sv2` crate #873 (includes refactoring output, which should be eventually converted into issue)
  • INPUT: Rust Docs issue: Need Rust docs on protocols::v2::codec-sv2 #1012
  • OUTPUT: Rust Docs PR: Rust docs for protocols::v2::codec-sv2 #1040
  • OUTPUT: Technical Debt issue: Refactor: codec-sv2 crate #1129
  • OUTPUT: Refactoring PRs: todo
• v2/noise-sv2
  • INPUT: Discussion: Questions about `noise_sv2` crate #916
  • INPUT: Rust Docs Issue: Need Rust docs on protocols::v2::noise-sv2 #1013
  • OUTPUT: Rust Docs PR: Rust docs noise_sv2 #1111
  • OUTPUT: Technical Debt issue: Refactor: noise-sv2 crate #1130
  • OUTPUT: Refactoring PRs: todo
• v2/subprotocols (already documented but worth a look before proceeding)
  • PR improving docs: Add missing doc comments for subprotocols messages #963
  • partially identified refactoring output: remove JobDistributionProtocol from common_messages_sv2  #1065
• v2/roles-logic-sv2
  • INPUT: Q&A Discussion: todo
  • INPUT: Rust Docs Issue: Need Rust docs on protocols::v2::roles-logic-sv2 #1014
  • OUTPUT: Rust Docs PR: todo
  • OUTPUT: Technical Debt issue: todo
  • OUTPUT: Refactoring PRs: todo
• v2/sv2-ffi
  • INPUT: Q&A Discussion: Questions about `sv2_ffi` crate #895
  • INPUT: Need Rust docs on protocols::v2::sv2-ffi #1015
  • OUTPUT: Rust Docs PR: todo
  • OUTPUT: Technical Debt issue: Refactor: sv2-ffi crate #1136
  • OUTPUT: Refactoring PRs: todo

Refactor Implementation

• refactor all the repetitive code in roles-logic-sv2 parsers file into macros #67
• TODO

[rrybarczyk]

We can leverage the work done in #854 in this effort when we start listing the technical debt for each crate.

[jbesraa]

What do you think about merging all the v2/binary-sv2/** into a single v2/binary-sv2 for the following reason #1134 (comment) ?

[Fi3]

I don't think that this is a good idea as we wouldn't be able to opt out serde anymore

[jbesraa]

Oh sorry maybe my comment was not clear. I was just suggesting to merge them in terms of the effort to document and refactor them, not merge the crates into a single one.

[plebhash]

What do you think about merging all the v2/binary-sv2/** into a single v2/binary-sv2 for the following reason #1134 (comment) ?

As discussed in last week's dev call (sep 10th 2024), binary-sv2/serde-sv2 is a candidate for deprecation. This crate is the only reason we maintain the with_serde feature flag, which is known to be causing some trouble on downstream projects. Therefore it is highly desirable that we eliminate the with_serde feature flag by getting rid of this crate.

So spending time documenting this crate would probably be wasted effort.

The only thing preventing us from deprecating it right now is the fact the MG uses it as a dependency.

[plebhash]

we are starting to write many ideas to our Technical Debt issues

we should allow ourselves room to brainstorm and propose ideas, this is the time to get creative

but when we start doing the actual refactoring PRs, it's of strategic importance that we cultivate a disciplined mindset around Goodhart's Law so we can make the right engineering tradeoffs and compromises