Skip to content

streams: add a bulk-read operation to StreamIO to unblock perf download throughput #276

Description

@adust09

Problem

StreamIO only exposes streamReadByte, so every consumer reads one byte at a time (with an IORef-buffered chunk underneath in the yamux adapter, plus a BS.tail allocation per byte). For message protocols this is fine, but the perf test app (#130, PR #275) drains multi-megabyte payloads through this path:

  • upload hs→nim measured 0.49–0.80 Gbps (write path is chunked, fast)
  • download nim→hs rounds to 0.00 Gbps (< 5 Mbps) — entirely bounded by the per-byte read path

The unified-testing perf harness defaults to UPLOAD_BYTES/DOWNLOAD_BYTES = 1 GiB, which is impractical at this read speed, so this blocks the upstream perf/images.yaml submission.

Proposal

Add a chunk-level read to StreamIO (e.g. streamReadChunk :: IO ByteString, returning whatever is buffered, EOF via exception as today) and implement it in:

  • the yamux adapter (yamuxStreamToStreamIO — hand back the buffered chunk instead of a byte)
  • the Noise session wrapper (noiseSessionToStreamIO — a decrypted frame is already a chunk)
  • the in-memory test pairs (mkMemoryStreamPair, mkEofStreamPair)

then switch readExactBounded, perf's drainUntilEof/discardExactly, and other bulk readers onto it. streamReadByte can remain as a default derived from the chunk read for the message protocols.

Acceptance

  • perf self-test download throughput within the same order of magnitude as upload
  • both hs↔nim perf cross directions still pass

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions