Add manual-scope v0 to tensormap runtime#482
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces manual dependency tracking to the tensormap_and_ringbuffer runtime, allowing for a hybrid model where same-scope dependencies are explicitly defined while cross-scope relations continue to use TensorMap discovery. The implementation includes significant updates to the orchestrator and scheduler to handle deferred task publication and manual edge recording, as well as a repository-wide restructuring of examples and tests to follow a new {arch}/{runtime} directory convention. Review feedback correctly identified critical issues with bitmask widths and bitwise shifts that could lead to overflows when tracking more than 16 tensor arguments. Suggestions were also made to improve memory safety during buffer reallocation and to simplify logic using standard library functions.
- Capture the hybrid scoped model for tensormap_and_ringbuffer - Define same-scope explicit edges versus cross-scope TensorMap behavior - Record ownership, scope, nesting, and testing constraints before implementation
- Force outer-scope reads in manual scope through TensorMap boundary seeding - Remove the invalid inner-created outer-alias case and keep Tensor layout unchanged - Add explicit scope, tooling, and narrow-change requirements for the implementation PR
- add PTO2ScopeMode::MANUAL, manual submit APIs, and deferred\n scope_end replay for tensormap_and_ringbuffer\n- add paged_attention_partial_manual plus paged_attention*_partial_manual\n ST coverage for nested outer-normal and inner-manual scopes\n- repoint AGENTS.md/CLAUDE.md toward the .agents layout and add a\n placeholder so the directory is tracked
- add a benchmark_rounds runtime selector for the partial-manual scenes\n- keep the current tensormap runtime on the direct selector path\n- record fresh 2026-04-06 hardware comparison data in the design doc
- replace the interim comparison table with the newest fresh device-2 reruns\n- keep the benchmark workflow section aligned with the partial-manual selector\n- record the remaining AUTO-path and partial-manual performance gaps
- cache manual-local tensor classification to avoid repeated scope scans - fuse manual publish and scope-end release into one scheduler pass - limit manual scope sync to active rings and keep submit-path work deferred - chunk paged_attention_partial_manual scopes and add carry deps between updates
Reduce the manual-scope chunk size in the heavy paged-attention\nscene and drop the extra cross-update dependency chain. The\nprevious chunking shape can deadlock the dep pool under benchmark\nload, while the smaller chunk keeps the benchmark path stable and\nlowers the manual-scope overhead.
Manual submit must stay a cheap metadata-recording path and defer\nTensorMap lookup/insert plus dep-pool fanin wiring to manual\nscope_end.\n\nThis reverts the duplicated submit-time work from 0fd6fbc and\nrestores the separate publish/on_scope_end order so manual scopes\ndo not attach fanout edges after releasing their scope reference.
- update the manual dependency design doc to make submit-time boundary discovery and TensorMap-free manual scope_end explicit - cache and retain external producers during manual submit, then merge them with explicit manual edges at publish time - keep the heavy manual paged-attention benchmark on device 3 moving in the right direction without changing example code
- dedupe explicit manual edges when they are recorded and keep an exact incoming edge count per consumer - append local explicit producers directly at scope_end and skip lock/task_state checks for unpublished same-scope producers - keep overflow validation and dep-pool publish ordering unchanged so the optimization stays within existing scheduler invariants
- rewrite the non-unroll partial-manual example to use one manual scope per q tile - move the hub allocation into the manual scope and serialize update tasks explicitly - drop the chunked manual-scope pattern that inflated partial-manual orchestration cost
- mark external inputs and outputs as manual-dep boundaries in the non-unroll partial-manual paged attention example - skip repeated overlap tracking for query, kv-cache, and final output views that are already ordered by the explicit manual dependency chain - keep the manual-scope methodology while improving orch time on real device for both paged-attention cases
- replace the stale benchmark section with fresh 2026-04-08 device-2 results - document how the benchmark wrapper selects new, unmodified, and partial-manual variants - record that partial-manual improved on non-unroll but still misses the aicpu_build_graph target
- replace stale benchmark data in the design doc with fresh device-3 measurements for the four paged-attention runtime lanes - document how benchmark_rounds.sh selects the partial-manual scenes - record the non-unroll boundary-hint A/B results and the safety limits of using manual_dep=true as an example-level boundary annotation
- explain submit-time versus scope-end work in the manual-scope path - document how in-scope and cross-scope tensors are classified and handled - bind the kept example optimizations to measured orch gains and remove stale scope-end frontier wording
- add a short rationale section tying each manual-scope rule to the incorrect or too-expensive alternative it avoids - keep the design doc aligned with the current implemented split between TensorMap boundary discovery and scope-end explicit-edge replay
- delete the copied tensormap_and_ringbuffer_unmodified runtime and ST scenes\n- keep branch docs and benchmark helpers limited to supported runtimes\n- enforce examples/{arch}/{runtime}/{name} in tracked command docs\n- rewrite example and ST path references to use explicit arch prefixes
- add a small hardware test helper that respects PTO_TEST_DEVICE_ID\n- fall back to the lowest NPU with no running processes from npu-smi\n- avoid blocking manual-scope hardware tests behind busy device 0 on shared machines
- widen manual-local masks to 64 bits so manual submit handles all\n MAX_TENSOR_ARGS entries without truncation\n- keep realloc failures from dropping live metadata buffers by setting a\n fatal out-of-memory runtime error before returning\n- simplify the partial-manual paged-attention valid_len calculation with\n std::min
- realize external producer fanout during manual submit while keeping the publish barrier at scope_end - shrink manual scope_end to replay only same-scope explicit edges - keep manual-scope validation and boundary semantics unchanged
- remove dead manual replay metadata from the manual-scope path - skip tensormap sync when a manual submit stays fully in-scope - keep only the publish barrier and dep-pool watermark fixup at scope_end On device 4 with 5 rounds, paged_attention_partial_manual improved from 35.27 ms / 35.12 ms orch to 31.60 ms / 31.44 ms orch for Case1, and from 19.80 ms / 19.38 ms orch to 17.93 ms / 17.49 ms orch for Case2.
- rewrite the design note to match the current manual submit and publish-only\n scope_end implementation\n- record the fresh four-way paged-attention comparison and benchmark\n entrypoints, including the detached worktree flow for the old runtime\n- remove the dead manual_dep_pool_reserve state from the orchestrator
- replace stale replay-heavy scope_end description with the current\n submit-time wiring model\n- document how AUTO, MANUAL, and benchmark selectors map onto the\n paged-attention scenes\n- record the fresh device-6 four-runtime comparison and the observed\n gains for partial-manual and zero-overhead AUTO
Merge manual scope-end validation and dep-pool watermark repair into a\nsingle pass.\n\nThis keeps the manual publish path behavior unchanged while trimming one\nserial walk over the scope task list.
- repair the rebased tensormap submit prologue and task-id write-back\n- restore partial-manual hub kernel sources under the example trees\n- repoint the partial-manual configs so hardware benchmarks build again
- update the rebased partial-manual unroll orchestration to match the\n current qk/pv kernel ABI by passing block_table as a tensor plus\n bt_offset as a scalar\n- rerun the device validation for both unroll cases after the fix\n- refresh the design doc with the rebase root cause and the new 4-way\n benchmark results on device 4 with PTO-ISA d96c8784
30b706a to
e5fa1bc
Compare
- record the approved runtime-first optimization scope for manual mode\n- keep TensorMap semantics unchanged in this pass\n- define the common-case scope_end bypass and O(1) manual edge lookup\n- capture risks and validation before touching runtime code
- break the approved runtime-first optimization into staged tasks\n- keep risk controls explicit for stale lookup state and repair fallback\n- require device validation and fresh four-way benchmarks before PR update
- add failing perf-invariant coverage for manual-scope repair signaling - align existing manual-scope hardware UTs to the current PTO-ISA pin - verify the new file fails before runtime implementation while the existing boundary and guard UTs still pass on a healthy device
- add explicit manual-scope state for the later dep-pool repair bypass - reset the repair flag conservatively at scope boundaries - keep this step behavior-neutral so the new perf-invariant tests still fail only on the missing publish signal
- add manual submit helpers that accept explicit producer ids and use them in the paged-attention partial-manual hot paths - keep explicit manual edges in payload metadata during submit, then link only the cached explicit tail range at manual scope_end - preserve add_dependency semantics for manual scopes while cutting non-unroll partial-manual Case1 from about 32.76 ms to 32.01 ms on device 5
- rewrite the design note to match the cached explicit manual fanin range implementation - replace stale benchmark tables with fresh device-5 four-way comparisons - document which optimizations removed add_dependency cost and reduced scope_end overhead
- Precompute exact manual creator-retention, lookup, and insert masks\n- Skip Step 3 and Step 4 scans when manual submit has no TensorMap work\n- Drop dead local/boundary bookkeeping and only visit the remaining args\n\nHardware check: PTO_TEST_DEVICE_ID=5 pytest tests/ut/test_manual_scope_boundary.py tests/ut/test_manual_scope_guards.py -v\nPerf check: paged_attention_partial_manual on device 5 shows lower device-log medians for Case1 and neutral-to-slightly-better Case2 vs HEAD in the same session
- Populate Tensor::start_offset when external tensors and views are created\n- Keep fresh create-info outputs at zero and stop recomputing offsets in payload init\n- Add a focused C++ unit test that fails when external/view tensors carry stale offsets\n\nVerification:\n- ctest --test-dir tests/ut/cpp/build -R test_a2a3_tmr_tensor_offsets --output-on-failure\n- PTO_TEST_DEVICE_ID=5 python3 -m pytest tests/ut/test_manual_scope_boundary.py tests/ut/test_manual_scope_guards.py -v\n- python3 examples/scripts/run_example.py --build -k tests/st/a2a3/tensormap_and_ringbuffer/paged_attention/kernels -g tests/st/a2a3/tensormap_and_ringbuffer/paged_attention/golden.py -p a2a3 -d 5 --clone-protocol https -c d96c8784\n- Device-log medians for paged_attention_partial_manual improved from 31808.42us to 29144.02us (Case1) and from 31058.64us to 29514.62us (Case2)
- Restore PR-only edits outside examples, src, tests, and tools/benchmark_rounds.sh. - Preserve the original benchmark coverage while adding partial-manual benchmark support.
- add partial-manual scene variants for alternating_matmul_add, benchmark_bgemm, and batch_paged_attention by reusing the baseline kernels/golden data with dedicated manual-scope orchestrations - extend tools/benchmark_rounds.sh so the partial-manual runtime benchmarks the full built-in scene set and no longer relies on example filtering via -e - validate the new scenes with local syntax checks, simulation for alternating_matmul_add/benchmark_bgemm, and hardware runs for all three scenes
Summary
manual_scope v0model totensormap_and_ringbufferusing the same submit APIs as AUTO modeArg.add_dep(task_id)instead of a new manual submit API familyscope_end()replay barrierdocs/manual-scope-v0-design.mdDesign Reference
The main design/reference document for this PR is:
docs/manual-scope-v0-design.mdConstraints Of This Version
This PR intentionally stays narrow.
PTO2_SCOPE()remains AUTO by default;PTO2_SCOPE(PTO2ScopeMode::MANUAL)enables manual scopepto2_rt_submit_aic_task(...)pto2_rt_submit_aiv_task(...)pto2_rt_submit_task(...)Arg.add_dep(task_id)Arg.add_dep(...)is rejected outside manual scopeadd_dependency(...)scope_end()dependency replayCore Runtime Semantics
Inside manual scope, the runtime still publishes tasks immediately at submit time, like AUTO mode.
The additional v0 behavior is:
ArgThis keeps manual scope as a lightweight extension on top of normal PTO2 submit, not a separate graph-construction pipeline.
Where TensorMap Lookup Still Happens
Manual scope does not remove TensorMap entirely.
TensorMap lookup/insert still happens for:
INOUTandOUTPUT_EXISTINGTensorMap lookup can be skipped only for producer-flow tensors that are:
Arg.add_dep(...)orderingThat split is the key design point of v0.
Why
alloc_tensors(...)Mattersalloc_tensors(...)stays output-only, but v0 treats allocation as a task and returns its task id together with the output tensors.That matters because manual consumers can then depend on the allocation explicitly:
auto alloc = alloc_tensors(ci0, ci1); Arg args; args.add_dep(alloc.task_id());This keeps the API light:
Important Limitation
V0 cannot express every same-scope chain using only
Arg.add_dep(...).The main gap is zero-output updater tasks.
They do not expose a returned task id handle, so repeated modifier chains like:
update(i) -> update(i+1)still need TensorMap publication / lookup on
INOUT/OUTPUT_EXISTINGtensors.That is why this PR keeps TensorMap on the modifier path even inside manual scope.
Testing
Verified on this branch tip:
ctest --test-dir tests/ut/cpp/build -R 'test_a2a3_pto2_manual_scope_(api|runtime)' --output-on-failurepython -m pytest tests/st/a2a3/tensormap_and_ringbuffer/test_manual_scope_validation.py --platform a2a3sim --device 0 -qThe design note also records the broader validation / benchmark context used during development:
docs/manual-scope-v0-design.md