Skip to content

Add manual-scope v0 to tensormap runtime#482

Closed
uv-xiao wants to merge 48 commits into
hw-native-sys:mainfrom
uv-xiao:manual-dep-for-tensormap
Closed

Add manual-scope v0 to tensormap runtime#482
uv-xiao wants to merge 48 commits into
hw-native-sys:mainfrom
uv-xiao:manual-dep-for-tensormap

Conversation

@uv-xiao

@uv-xiao uv-xiao commented Apr 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a narrower manual_scope v0 model to tensormap_and_ringbuffer using the same submit APIs as AUTO mode
  • introduce explicit same-scope dependencies through Arg.add_dep(task_id) instead of a new manual submit API family
  • keep task publication at submit time; do not add delayed wiring, delayed linking, or a manual scope_end() replay barrier
  • add manual-scope validation coverage, paged-attention manual-scope example variants, and the design note in docs/manual-scope-v0-design.md

Design Reference

The main design/reference document for this PR is:

  • docs/manual-scope-v0-design.md

Constraints Of This Version

This PR intentionally stays narrow.

  • PTO2_SCOPE() remains AUTO by default; PTO2_SCOPE(PTO2ScopeMode::MANUAL) enables manual scope
  • submit APIs stay unchanged:
    • pto2_rt_submit_aic_task(...)
    • pto2_rt_submit_aiv_task(...)
    • pto2_rt_submit_task(...)
  • explicit ordering is expressed only through Arg.add_dep(task_id)
  • Arg.add_dep(...) is rejected outside manual scope
  • nested manual scopes are rejected in v0
  • no post-submit add_dependency(...)
  • no delayed wiring / delayed linking / delayed publish path
  • no manual-specific scope_end() dependency replay

Core Runtime Semantics

Inside manual scope, the runtime still publishes tasks immediately at submit time, like AUTO mode.

The additional v0 behavior is:

  • read explicit deps from Arg
  • validate that those deps belong to the current manual scope
  • materialize them as ordinary fanins before publish
  • classify tensor args by scope metadata to decide whether TensorMap lookup is still required

This keeps manual scope as a lightweight extension on top of normal PTO2 submit, not a separate graph-construction pipeline.

Where TensorMap Lookup Still Happens

Manual scope does not remove TensorMap entirely.

TensorMap lookup/insert still happens for:

  • boundary tensors: external tensors, AUTO-scope tensors, outer-scope tensors, outer-manual-scope tensors
  • modifier tensors: INOUT and OUTPUT_EXISTING

TensorMap lookup can be skipped only for producer-flow tensors that are:

  • produced in the current manual scope, and
  • already covered by explicit Arg.add_dep(...) ordering

That split is the key design point of v0.

Why alloc_tensors(...) Matters

alloc_tensors(...) stays output-only, but v0 treats allocation as a task and returns its task id together with the output tensors.

That matters because manual consumers can then depend on the allocation explicitly:

auto alloc = alloc_tensors(ci0, ci1);
Arg args;
args.add_dep(alloc.task_id());

This keeps the API light:

  • no new alloc-specific manual API
  • no delayed dependency wiring
  • allocation still participates in manual ordering when needed

Important Limitation

V0 cannot express every same-scope chain using only Arg.add_dep(...).

The main gap is zero-output updater tasks.
They do not expose a returned task id handle, so repeated modifier chains like:

  • update(i) -> update(i+1)

still need TensorMap publication / lookup on INOUT / OUTPUT_EXISTING tensors.

That is why this PR keeps TensorMap on the modifier path even inside manual scope.

Testing

Verified on this branch tip:

  • ctest --test-dir tests/ut/cpp/build -R 'test_a2a3_pto2_manual_scope_(api|runtime)' --output-on-failure
  • python -m pytest tests/st/a2a3/tensormap_and_ringbuffer/test_manual_scope_validation.py --platform a2a3sim --device 0 -q

The design note also records the broader validation / benchmark context used during development:

  • docs/manual-scope-v0-design.md

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces manual dependency tracking to the tensormap_and_ringbuffer runtime, allowing for a hybrid model where same-scope dependencies are explicitly defined while cross-scope relations continue to use TensorMap discovery. The implementation includes significant updates to the orchestrator and scheduler to handle deferred task publication and manual edge recording, as well as a repository-wide restructuring of examples and tests to follow a new {arch}/{runtime} directory convention. Review feedback correctly identified critical issues with bitmask widths and bitwise shifts that could lead to overflows when tracking more than 16 tensor arguments. Suggestions were also made to improve memory safety during buffer reallocation and to simplify logic using standard library functions.

Comment thread src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto_orchestrator.h Outdated
Comment thread src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto_orchestrator.cpp Outdated
Comment thread src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto_orchestrator.cpp Outdated
Comment thread src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto_orchestrator.cpp Outdated
uv-xiao and others added 28 commits April 9, 2026 23:22
- Capture the hybrid scoped model for tensormap_and_ringbuffer
- Define same-scope explicit edges versus cross-scope TensorMap behavior
- Record ownership, scope, nesting, and testing constraints before implementation
- Force outer-scope reads in manual scope through TensorMap boundary seeding
- Remove the invalid inner-created outer-alias case and keep Tensor layout unchanged
- Add explicit scope, tooling, and narrow-change requirements for the implementation PR
- add PTO2ScopeMode::MANUAL, manual submit APIs, and deferred\n  scope_end replay for tensormap_and_ringbuffer\n- add paged_attention_partial_manual plus paged_attention*_partial_manual\n  ST coverage for nested outer-normal and inner-manual scopes\n- repoint AGENTS.md/CLAUDE.md toward the .agents layout and add a\n  placeholder so the directory is tracked
- add a benchmark_rounds runtime selector for the partial-manual scenes\n- keep the current tensormap runtime on the direct selector path\n- record fresh 2026-04-06 hardware comparison data in the design doc
- replace the interim comparison table with the newest fresh device-2 reruns\n- keep the benchmark workflow section aligned with the partial-manual selector\n- record the remaining AUTO-path and partial-manual performance gaps
- cache manual-local tensor classification to avoid repeated scope scans
- fuse manual publish and scope-end release into one scheduler pass
- limit manual scope sync to active rings and keep submit-path work deferred
- chunk paged_attention_partial_manual scopes and add carry deps between updates
Reduce the manual-scope chunk size in the heavy paged-attention\nscene and drop the extra cross-update dependency chain. The\nprevious chunking shape can deadlock the dep pool under benchmark\nload, while the smaller chunk keeps the benchmark path stable and\nlowers the manual-scope overhead.
Manual submit must stay a cheap metadata-recording path and defer\nTensorMap lookup/insert plus dep-pool fanin wiring to manual\nscope_end.\n\nThis reverts the duplicated submit-time work from 0fd6fbc and\nrestores the separate publish/on_scope_end order so manual scopes\ndo not attach fanout edges after releasing their scope reference.
- update the manual dependency design doc to make submit-time boundary discovery and TensorMap-free manual scope_end explicit
- cache and retain external producers during manual submit, then merge them with explicit manual edges at publish time
- keep the heavy manual paged-attention benchmark on device 3 moving in the right direction without changing example code
- dedupe explicit manual edges when they are recorded and keep an exact incoming edge count per consumer
- append local explicit producers directly at scope_end and skip lock/task_state checks for unpublished same-scope producers
- keep overflow validation and dep-pool publish ordering unchanged so the optimization stays within existing scheduler invariants
- rewrite the non-unroll partial-manual example to use one manual scope per q tile
- move the hub allocation into the manual scope and serialize update tasks explicitly
- drop the chunked manual-scope pattern that inflated partial-manual orchestration cost
- mark external inputs and outputs as manual-dep boundaries in the
  non-unroll partial-manual paged attention example
- skip repeated overlap tracking for query, kv-cache, and final output
  views that are already ordered by the explicit manual dependency chain
- keep the manual-scope methodology while improving orch time on real
  device for both paged-attention cases
- replace the stale benchmark section with fresh 2026-04-08 device-2 results
- document how the benchmark wrapper selects new, unmodified, and partial-manual variants
- record that partial-manual improved on non-unroll but still misses the aicpu_build_graph target
- replace stale benchmark data in the design doc with fresh device-3
  measurements for the four paged-attention runtime lanes
- document how benchmark_rounds.sh selects the partial-manual scenes
- record the non-unroll boundary-hint A/B results and the safety limits of
  using manual_dep=true as an example-level boundary annotation
- explain submit-time versus scope-end work in the manual-scope path
- document how in-scope and cross-scope tensors are classified and handled
- bind the kept example optimizations to measured orch gains and remove
  stale scope-end frontier wording
- add a short rationale section tying each manual-scope rule to the
  incorrect or too-expensive alternative it avoids
- keep the design doc aligned with the current implemented split between
  TensorMap boundary discovery and scope-end explicit-edge replay
- delete the copied tensormap_and_ringbuffer_unmodified runtime and ST scenes\n- keep branch docs and benchmark helpers limited to supported runtimes\n- enforce examples/{arch}/{runtime}/{name} in tracked command docs\n- rewrite example and ST path references to use explicit arch prefixes
- add a small hardware test helper that respects PTO_TEST_DEVICE_ID\n- fall back to the lowest NPU with no running processes from npu-smi\n- avoid blocking manual-scope hardware tests behind busy device 0 on shared machines
- widen manual-local masks to 64 bits so manual submit handles all\n  MAX_TENSOR_ARGS entries without truncation\n- keep realloc failures from dropping live metadata buffers by setting a\n  fatal out-of-memory runtime error before returning\n- simplify the partial-manual paged-attention valid_len calculation with\n  std::min
- realize external producer fanout during manual submit while keeping the publish barrier at scope_end
- shrink manual scope_end to replay only same-scope explicit edges
- keep manual-scope validation and boundary semantics unchanged
uv-xiao added 7 commits April 10, 2026 00:20
- remove dead manual replay metadata from the manual-scope path
- skip tensormap sync when a manual submit stays fully in-scope
- keep only the publish barrier and dep-pool watermark fixup at scope_end

On device 4 with 5 rounds, paged_attention_partial_manual improved
from 35.27 ms / 35.12 ms orch to 31.60 ms / 31.44 ms orch for
Case1, and from 19.80 ms / 19.38 ms orch to 17.93 ms / 17.49 ms
orch for Case2.
- rewrite the design note to match the current manual submit and publish-only\n  scope_end implementation\n- record the fresh four-way paged-attention comparison and benchmark\n  entrypoints, including the detached worktree flow for the old runtime\n- remove the dead manual_dep_pool_reserve state from the orchestrator
- replace stale replay-heavy scope_end description with the current\n  submit-time wiring model\n- document how AUTO, MANUAL, and benchmark selectors map onto the\n  paged-attention scenes\n- record the fresh device-6 four-runtime comparison and the observed\n  gains for partial-manual and zero-overhead AUTO
Merge manual scope-end validation and dep-pool watermark repair into a\nsingle pass.\n\nThis keeps the manual publish path behavior unchanged while trimming one\nserial walk over the scope task list.
- repair the rebased tensormap submit prologue and task-id write-back\n- restore partial-manual hub kernel sources under the example trees\n- repoint the partial-manual configs so hardware benchmarks build again
- update the rebased partial-manual unroll orchestration to match the\n  current qk/pv kernel ABI by passing block_table as a tensor plus\n  bt_offset as a scalar\n- rerun the device validation for both unroll cases after the fix\n- refresh the design doc with the rebase root cause and the new 4-way\n  benchmark results on device 4 with PTO-ISA d96c8784
@uv-xiao uv-xiao force-pushed the manual-dep-for-tensormap branch from 30b706a to e5fa1bc Compare April 10, 2026 02:39
@uv-xiao uv-xiao marked this pull request as ready for review April 10, 2026 03:05
uv-xiao added 12 commits April 10, 2026 13:12
- record the approved runtime-first optimization scope for manual mode\n- keep TensorMap semantics unchanged in this pass\n- define the common-case scope_end bypass and O(1) manual edge lookup\n- capture risks and validation before touching runtime code
- break the approved runtime-first optimization into staged tasks\n- keep risk controls explicit for stale lookup state and repair fallback\n- require device validation and fresh four-way benchmarks before PR update
- add failing perf-invariant coverage for manual-scope repair signaling
- align existing manual-scope hardware UTs to the current PTO-ISA pin
- verify the new file fails before runtime implementation while the existing
  boundary and guard UTs still pass on a healthy device
- add explicit manual-scope state for the later dep-pool repair bypass
- reset the repair flag conservatively at scope boundaries
- keep this step behavior-neutral so the new perf-invariant tests still fail
  only on the missing publish signal
- add manual submit helpers that accept explicit producer ids and use them in the paged-attention partial-manual hot paths
- keep explicit manual edges in payload metadata during submit, then link only the cached explicit tail range at manual scope_end
- preserve add_dependency semantics for manual scopes while cutting non-unroll partial-manual Case1 from about 32.76 ms to 32.01 ms on device 5
- rewrite the design note to match the cached explicit manual fanin range implementation
- replace stale benchmark tables with fresh device-5 four-way comparisons
- document which optimizations removed add_dependency cost and reduced scope_end overhead
- Precompute exact manual creator-retention, lookup, and insert masks\n- Skip Step 3 and Step 4 scans when manual submit has no TensorMap work\n- Drop dead local/boundary bookkeeping and only visit the remaining args\n\nHardware check: PTO_TEST_DEVICE_ID=5 pytest tests/ut/test_manual_scope_boundary.py tests/ut/test_manual_scope_guards.py -v\nPerf check: paged_attention_partial_manual on device 5 shows lower device-log medians for Case1 and neutral-to-slightly-better Case2 vs HEAD in the same session
- Populate Tensor::start_offset when external tensors and views are created\n- Keep fresh create-info outputs at zero and stop recomputing offsets in payload init\n- Add a focused C++ unit test that fails when external/view tensors carry stale offsets\n\nVerification:\n- ctest --test-dir tests/ut/cpp/build -R test_a2a3_tmr_tensor_offsets --output-on-failure\n- PTO_TEST_DEVICE_ID=5 python3 -m pytest tests/ut/test_manual_scope_boundary.py tests/ut/test_manual_scope_guards.py -v\n- python3 examples/scripts/run_example.py --build -k tests/st/a2a3/tensormap_and_ringbuffer/paged_attention/kernels -g tests/st/a2a3/tensormap_and_ringbuffer/paged_attention/golden.py -p a2a3 -d 5 --clone-protocol https -c d96c8784\n- Device-log medians for paged_attention_partial_manual improved from 31808.42us to 29144.02us (Case1) and from 31058.64us to 29514.62us (Case2)
- Replace the stale paged_attention comparison table with the fresh 7-round rerun\n- Record that the earlier large ABG gap on Case2 was not reproduced\n- Tie the new numbers back to commits a65894a and 6d33941 and the latest profiling shape
- Restore PR-only edits outside examples, src, tests, and tools/benchmark_rounds.sh.
- Preserve the original benchmark coverage while adding partial-manual benchmark support.
- add partial-manual scene variants for alternating_matmul_add,
  benchmark_bgemm, and batch_paged_attention by reusing the
  baseline kernels/golden data with dedicated manual-scope
  orchestrations
- extend tools/benchmark_rounds.sh so the partial-manual runtime
  benchmarks the full built-in scene set and no longer relies on
  example filtering via -e
- validate the new scenes with local syntax checks, simulation for
  alternating_matmul_add/benchmark_bgemm, and hardware runs for all
  three scenes
@uv-xiao uv-xiao requested review from ChaoWao and poursoul April 12, 2026 17:34
Comment thread src/a2a3/runtime/tensormap_and_ringbuffer/runtime/tensor.h
Comment thread src/a2a3/runtime/tensormap_and_ringbuffer/runtime/tensor.h
Comment thread src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto_orchestrator.cpp Outdated
@uv-xiao uv-xiao closed this Apr 15, 2026
@uv-xiao uv-xiao changed the title Add manual-scope dependency mode to tensormap runtime Add manual-scope v0 to tensormap runtime Apr 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants