feat: implement /perf/1.0.0 protocol and unified-testing perf test app - #275
Merged
Conversation
Implement the canonical libp2p perf protocol (specs/perf) as a library module and add the perf test daemon for the libp2p/unified-testing framework (issue #130). - LibP2P.Protocol.Perf: /perf/1.0.0 responder and initiator. Client sends an 8-byte big-endian download size, streams zeros, half-closes; server drains to EOF, then sends the requested bytes back. - interop/perf: contract binary reading the perf env vars, coordinating via Redis SET/GET on {TEST_KEY}_listener_multiaddr (the perf contract uses a plain string, unlike the transport contract's RPUSH/BLPOP), and reporting upload/download/latency statistics as YAML on stdout with IQR outlier filtering. - Cross-tests run against nim-libp2p (pinned to the commit perf/images.yaml registers for nim-v1.15): the unified-testing go perf image is a placeholder that opens no perf streams, and its rust image deviates from the spec (two u64 headers), so nim is the partner that actually exercises the canonical wire format. - CI: perf-interop job runs the hs<->hs self-test and both hs<->nim directions. - Shared cleanups: word64BE/readWord64BE join Core.Binary, and closeQuietly moves to MultistreamSelect.Negotiation. Verified locally: 1199 unit tests pass; hs<->hs self-test and both hs<->nim cross directions succeed (upload hs->nim 0.49-0.80 Gbps, download nim->hs limited by the byte-at-a-time StreamIO read path, latency ~0.6 ms).
This was referenced Aug 25, 2026
adust09
added a commit
that referenced
this pull request
Aug 26, 2026
Switch perf's drainUntilEof and discardExactly from streamReadByte to streamReadChunk, completing the bulk-reader migration issue #276 lists. Deferred from the previous commit only because the perf app lived on PR #275's branch; now that #275 is merged the switch happens here. discardExactly caps each chunk request at the bytes still owed, so it still never consumes past the requested download size.
adust09
added a commit
that referenced
this pull request
Aug 27, 2026
…hput (#277) * feat: add streamReadChunk to StreamIO for bulk-read throughput Every consumer of StreamIO read one byte at a time, which bounded the perf test app's download direction at under 5 Mbps (issue #276) while uploads ran at 0.49-0.80 Gbps. Add a chunk-level read to StreamIO: streamReadChunk :: Int -> IO ByteString returning between 1 and n bytes (whatever is buffered or arrives next), with EOF surfacing as an IOException exactly like streamReadByte. The max-length argument (not in the issue's sketch) is what lets readExactBounded use it safely: with no push-back mechanism, an unbounded chunk read would consume bytes past a message boundary. Implemented in the yamux adapter and Noise session wrapper (hand back the buffered chunk / decrypted frame), the TCP socket (recv n), the in-memory test pairs, and a mkByteStreamIO helper that derives a one-byte-per-call chunk read for byte-queue test mocks. readExactBounded now reads chunks, which moves the whole receive path off byte-at-a-time reads: Noise frame reads from the raw socket, the yamux read callback, and every length-delimited protocol reader. The relay's forwardWithLimit also forwards at chunk granularity, still never consuming a byte beyond the circuit's limit. Two DCUtR upgrade tests asserted that no direct connection exists 500ms after the circuit dial; the faster relayed path now lets the automatic DCUtR upgrade pool a direct connection inside that window (on loopback the handler-side dial is an ordinary client dial and succeeds). Those tests now run with the automatic upgrade disabled via zero-length timeout windows, making their pool preconditions deterministic. * feat: drain perf payloads at chunk granularity Switch perf's drainUntilEof and discardExactly from streamReadByte to streamReadChunk, completing the bulk-reader migration issue #276 lists. Deferred from the previous commit only because the perf app lived on PR #275's branch; now that #275 is merged the switch happens here. discardExactly caps each chunk request at the bytes still owed, so it still never consumes past the requested download size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #130 (implementation part; the upstream
perf/images.yamlPR tolibp2p/unified-testingis a follow-up once this merges, since the entry must pin a merge commit).What
LibP2P.Protocol.Perf— canonical/perf/1.0.0from libp2p/specs perf: client sends an 8-byte big-endian download size, streams its upload, half-closes; server drains to EOF, then (never concurrently with the upload) sends the requested bytes back. Registered on the Switch like Ping, exported via theLibP2Pfacade.interop/perf— unified-testing perf contract daemon (docs/write-a-perf-test-app.md): uppercase env vars, Redis coordination via SET / polled GET on{TEST_KEY}_listener_multiaddr(the perf contract's reference apps use a plain string, unlike the transport contract's RPUSH/BLPOP), YAML results on stdout with interpolated quartiles and 1.5×IQR outlier filtering (mirrors the reference implementations' stats math).interop/perf/Dockerfile— same path convention nim/python/lua use inperf/images.yaml(dockerfile: interop/perf/Dockerfile, context = repo root).perf-interopjob: hs↔hs self-test + both hs↔nim cross directions.word64BE/readWord64BEadded toCore.Binary;closeQuietlypromoted toMultistreamSelect.Negotiation(Ping/NAT/Identify/Relay still carry local copies; migrating them is out of scope here).Why nim as the cross-implementation partner
Inspecting
libp2p/unified-testingshowed the go perf image (images/go/v0.45) is a placeholder — it dials but opens no perf streams (// Placeholder: simulate transfer), and the rust image deviates from the spec (sends two u64 headers instead of one). nim-libp2p mounts its real library perf protocol and is the canonical-wire reference, so cross-tests pin nim at the exact commitperf/images.yamlregisters fornim-v1.15(single source of truth:NIM_LIBP2P_COMMITininterop/Makefile).Verification
StreamIOread path; honest measurement, follow-up issue to add a bulk-read op before the upstream submission (1 GiB defaults would be impractical otherwise)Follow-ups
StreamIOto fix download throughput (blocking for upstream submission with default 1 GiB sizes)perf/images.yamlPR tolibp2p/unified-testingpinned at this PR's merge commit