fix: report transferred bytes on the remote-deliver path

[kangclzjc]

Fixes #6

Every other tier records blocks and bytes as a pair through _KVCRCore._record_transfer (core.py:748-762). The peer path records its counters individually, and the destination only ever recorded blocks — TRANSFER_BYTES_METRIC was emitted from exactly one place (remote_fw_dram.py:1412-1416), hardcoded to the source_write scope, reachable only from _SourceWriteOp (:318).

A completed peer transfer therefore reported:

kvcr_transfer_bytes:('source_write',)    = 51380224
kvcr_transfer_blocks:('source_write',)   = 7
kvcr_transfer_blocks:('remote_deliver',) = 7

with no byte counter on the destination, making the verification step in docs/quick-start.md:260-262 — "their block and byte counts must agree" — impossible to carry out.

Why the destination cannot just reuse the source path

The transfer is a source-side push. _SourceWriteOp holds a transfer_id and reads NIXL telemetry via progress.poll_transfer(). _TargetPullOp has no transfer_id; it learns about completion from the write_done control message (:143). It has no NIXL telemetry to report, so its only byte source is its own descriptors.

Change

Record the delivered byte count next to the existing block count. dst_descriptors is positionally aligned with ordered_keys — both are populated together at the single construction site (remote_fw_dram.py:477-489), and the wire path validates len(keys) == len(dst_descriptors) (:848) — so summing the spans for the completed keys covers exactly the keys the block counter reports.

This covers both destination scopes, remote_deliver and remote_fetch (:105).

End-to-end verification

Built the quick-start image with this change and ran a real cross-rank transfer between two vLLM data-parallel ranks:

--- metrics line 188 ---   (source, connected_remotes=1)
    vllm:kvcr_transfer_bytes:('source_write',)=51380224
    vllm:kvcr_transfer_blocks:('source_write',)=7
--- metrics line 190 ---   (destination, connected_remotes=0)
    vllm:kvcr_transfer_blocks:('remote_deliver',)=7
    vllm:kvcr_transfer_bytes:('remote_deliver',)=51380224

The two sides report identical block and byte counts. The differing connected_remotes values confirm these are two distinct KVCR instances rather than one instance reported twice.

51,380,224 also checks out independently: it is exactly 7 x (2 x 28 layers x 8 KV heads x 128 head_dim x 2 bytes x 64 tokens), the KV size of 7 blocks for Qwen3-0.6B.

On partial completion

The source reports the bytes NIXL moved; the destination reports the spans it committed. Both derive from the same slice of the same descriptor list — the source submits dst_descriptors[:completed_count] (:256) and propagates completed_count in the write_done notification (:261), which the destination uses to pick ordered_keys[:completed_count] (:149) — so they should agree on a partial completion too.

That case is not exercised end to end here, only the fully completed one shown above. The unit test's fake NIXL agent returns a constant totalBytes=32 regardless of how much was transferred, so it cannot confirm the partial case either.

Test

Extended test_remote_framework_dram_transfers_available_prefix, which already asserted the source byte counter and the destination block counter, to cover the new counter across all three parametrizations.

Verification

ruff check src tests passes. pytest tests/unit matches main — 152 passed, same 15 pre-existing environmental failures (SO_PEERPIDFD needs Linux 6.5+; CI runners are unaffected).

🤖 Generated with Claude Code

[oandreeva-nv]

Thank you for the PR, @aranadive could you please take a look?