Every other tier records blocks and bytes as a pair through _KVCRCore._record_transfer (core.py:748-762), which emits TRANSFER_BLOCKS_METRIC and TRANSFER_BYTES_METRIC for the same scope. local_disk.py:429 and local_dram.py:487 both go through it.
The peer path in remote_fw_dram.py records its counters individually instead, and the destination only ever records blocks (remote_fw_dram.py:159-164). TRANSFER_BYTES_METRIC is emitted from exactly one place in that file (:1412-1416), hardcoded to the source_write scope.
Impact
A completed peer transfer between two data-parallel ranks reported:
kvcr_transfer_bytes:(\x27source_write\x27,) = 51380224
kvcr_transfer_blocks:(\x27source_write\x27,) = 7
kvcr_transfer_blocks:(\x27remote_deliver\x27,) = 7
There is no byte counter on the destination at all. That makes this verification step in docs/quick-start.md (lines 260-262) impossible to carry out as written:
A successful transfer reports transfer with result success on the source and remote_deliver with result success on the destination; their block and byte counts must agree.
The outcome labels themselves are fine — kvcr_duration_seconds:(\x27transfer\x27,\x27success\x27) appears on the source and (\x27remote_deliver\x27,\x27success\x27) on the destination, exactly as documented. Only the byte half of the cross-check is unavailable, so a user can verify block counts and nothing else.
Suggested fix
Record the delivered byte count next to the existing block count. _TargetPullOp.dst_descriptors is positionally aligned with ordered_keys — both are populated together at the single construction site (remote_fw_dram.py:477-489) — so summing descriptor.size for the completed keys covers exactly the keys the block counter reports.
Note the two sides count different things by design: the source reports the bytes NIXL moved for the transfer, the destination reports the spans it committed. They line up on a fully completed transfer, which is the case the quick-start tells users to verify. If that distinction is intended to stay, the alternative fix is to correct the doc instead.
Found while verifying a real cross-rank transfer from docs/quick-start.md.
Every other tier records blocks and bytes as a pair through
_KVCRCore._record_transfer(core.py:748-762), which emitsTRANSFER_BLOCKS_METRICandTRANSFER_BYTES_METRICfor the same scope.local_disk.py:429andlocal_dram.py:487both go through it.The peer path in
remote_fw_dram.pyrecords its counters individually instead, and the destination only ever records blocks (remote_fw_dram.py:159-164).TRANSFER_BYTES_METRICis emitted from exactly one place in that file (:1412-1416), hardcoded to thesource_writescope.Impact
A completed peer transfer between two data-parallel ranks reported:
There is no byte counter on the destination at all. That makes this verification step in
docs/quick-start.md(lines 260-262) impossible to carry out as written:The outcome labels themselves are fine —
kvcr_duration_seconds:(\x27transfer\x27,\x27success\x27)appears on the source and(\x27remote_deliver\x27,\x27success\x27)on the destination, exactly as documented. Only the byte half of the cross-check is unavailable, so a user can verify block counts and nothing else.Suggested fix
Record the delivered byte count next to the existing block count.
_TargetPullOp.dst_descriptorsis positionally aligned withordered_keys— both are populated together at the single construction site (remote_fw_dram.py:477-489) — so summingdescriptor.sizefor the completed keys covers exactly the keys the block counter reports.Note the two sides count different things by design: the source reports the bytes NIXL moved for the transfer, the destination reports the spans it committed. They line up on a fully completed transfer, which is the case the quick-start tells users to verify. If that distinction is intended to stay, the alternative fix is to correct the doc instead.
Found while verifying a real cross-rank transfer from
docs/quick-start.md.