HIP-1037 - Protocol Buffer Specification

HIP/hip-1037.md

@@ -0,0 +1,117 @@
+---
+hip: 1037
+title: Protocol Buffer API Specification
+author: Joseph Sinclair <@jsync-swirlds>
+requested-by: Richard Bair <@rbair>
+type: Standards Track
+category: Service
+needs-council-approval: Yes
+status: Review
+created: 2024-08-09
+discussions-to: https://github.com/hashgraph/hedera-improvement-proposal/discussions/1037
+updated: 2024-11-06
+---
+
+## Abstract
+The current protocol buffer API definitions are technically correct, but lack
+usable or useful specification. We should add, in comments compatible with
+protoc-gen-doc, specification text that clearly states both requirement and
+expectation for every message, file, and field. This documentation should be
+automatically transformed to markdown documentation that is published to the
+Hedera API site. In the ideal case this specification will be sufficient for
+any qualified team to implement a completely independent consensus node that
+is fully interoperable with the current consensus node software.
+
+## Motivation
+The existing protocol buffer API documentation is brief, entirely descriptive,
+and often inaccurate. This produces two negative consequences. First, it is not
+possible to produce a compatible independent implementation of the Hedera
+consensus node software without extensive reference to the existing source
+code. Second, developers are often unclear how best to interact with the
+Hedera API, and independent SDK developers, in particular, may struggle to
+correctly implement these API calls.
+
+As part of expanding the Hedera ecosystem we MUST reduce the difficulty and
+hurdles for independent developers when building systems that interact with
+the Hedera network. Clear and accurate API specifications are one element
+required to accomplish this goal.
+
+## Rationale
+This HIP improves the developer experience and enhances decentralization by
+making it easier to design distributed applications or alternative
+implementations of core network systems.
+
+The use of a markdown format that is automatically generated from the
+functional protocol buffer definition files encourages more frequent updates
+that are more closely tied to the functional updates to the API documents, and
+reduces the inevitable "drift" between functional API and API specification.
+
+## User stories
+- As a dApp developer I want to have clear and accurate specification for all
+  Hedera APIs so that I can confidently design my application to call those
+  APIs.
+- As an independent implementor I want to have clear and accurate specification
+  for all Hedera APIs so that I can design an independent, correct, and
+  compatible implementation of Hedera network nodes, including a consensus node.
+- As an Hedera software contributor I want to have clear and accurate
+  specification for all Hedera APIs so that I can confidently implement changes
+  to Hedera-led software.
+- As a dApp developer that receives bytes or is expected to parse, produce, or
+  process bytes representing a HAPI protobuf message I want to have a clear
+  specification of the structure and implications of the corresponding proto
+  messages to understand how to correctly parse them.
+
+## Specification
+The core of this HIP is
+[the guidelines](../assets/hip-1037/Specification-Format-Style-Guidelines.md)
+recommended for this, and all future Hedera protocol buffer specifications.
+This HIP, however, also implements sweeping changes to specification text for
+the existing API documents. These changes affect over 20,000 lines of text
+across almost 150 files in order to add full specification text that complies
+with the guidelines expressed in this improvement proposal. The full reference
+implementation, therefore, is only linked here for brevity and clarity.<br/>
+The implementation of this HIP is detailed in github as a
+pull request linked [later in this document](#reference-implementation).
+
+## Backwards Compatibility
+This HIP does not change any functional characteristic, and documents the
+state of the API as-is.  This does not introduce any change in compatibility.
+
+## Security Implications
+This HIP updates and clarifies specification and expectation for the Hedera
+API. This does not materially impact the security of the system.
+
+## How to Teach This
+The Hedera API is a critical component of the design and interaction for the
+Hedera ecosystem. Improvements to the formality and completeness of the API
+specification help everyone involved to develop distributed applications
+more quickly and with greater confidence.
+
+## Reference Implementation
+We have a
+[pull request](https://github.com/hashgraph/hedera-protobufs/pull/388)
+that implements these guidelines across the full hedera-protobufs repository.
+
+## Rejected Ideas
+1. Use of a highly formal "language" for specification
+   * This was deemed excessive.  Protocol Buffers are already a highly formal
+     language, and adding specification as markdown comments, using the 
+     light weight RFC/HIP language is sufficient while remaining approachable
+     for the largest practical number of community members.
+1. Addition of specialized "tag" elements to specification comments.
+   * This would have required building a complete custom documentation
+     processor and would have been non-standard.  Rather than do so we chose
+     to adhere closely to existing standard approaches for protocol buffers.
+
+
+## Open Issues
+None.
+
+## References
+- Protocol Buffer Document Generator
+  [protoc-gen-doc](https://github.com/pseudomuto/protoc-gen-doc/?tab=readme-ov-file#protoc-gen-doc).
+
+## Copyright/license
+This document is licensed under the Apache License, Version 2.0 --
+see [LICENSE](../LICENSE) or (https://www.apache.org/licenses/LICENSE-2.0)
+