[Bug] Flaky ST: test_dynamic_paged_attention[256-64-128-64-8192-32768] intermittently fails on a2a3 with AICore error 507018

[lyfne123]

Component

Other (please specify in description) — system test (tests/st/runtime/framework_and_models/) failing at on-device execution of the generated paged-attention kernel on the a2a3 backend/runtime.

Description

The system test tests/st/runtime/framework_and_models/test_dynamic_paged_attention.py::TestDynamicPagedAttentionKernels::test_dynamic_paged_attention[256-64-128-64-8192-32768] (test name dynamic_paged_attention_256bat_64h_128d_64bs; params batch=256, num_heads=64, head_dim=128, block_size=64, context_len=8192, max_model_len=32768) intermittently fails on a2a3 with AICore error code 507018.

When it fails, the device is marked unrecoverable and is force-reset. The failure is non-deterministic (~1/3 hit rate in local reproduction), so it occasionally turns the whole system-tests CI job red without any related code change.

This is not caused by a specific diff — it was hit while triaging PR #1834 (a Python-only runtime-default rename, zero .cpp/.h changes), and reproduces independently of that PR. The generated kernel is identical to main.

Steps to Reproduce

Run the single parametrization on an a2a3 device. A single run often passes; repeat it to surface the flakiness:

export PYTHONPATH=/path/to/pypto/python:$PYTHONPATH
python -m pytest \
  "tests/st/runtime/framework_and_models/test_dynamic_paged_attention.py::TestDynamicPagedAttentionKernels" \
  -k "256-64-128-64-8192-32768" \
  -v --device=<dev> --platform=a2a3 \
  --pto-isa-commit=016396b57e2c17093f1194e6acd89bb112b0ab24 \
  --count=3

Observed locally (2026-06-23):

• Single isolated run: PASSED (~72s).
• --count=3: 1 failed / 2 passed (failed on the 3rd repetition with the same 507018 signature).

Expected Behavior

The test passes deterministically on every run; the device is not poisoned.

Actual Behavior

One repetition out of three fails:

RuntimeError: run_prepared failed with code 507018
  File ".../runtime/device_runner.py", line 638, in execute_on_device
    return worker.run(cid, orch_args, cfg)
  File ".../simpler/worker.py", line 2587, in run
  File ".../simpler/task_interface.py", line 544, in _run_slot
    return self._impl.run(int(callable_id), args, config)

[ERROR] sync_run_streams: [device_runner_base.cpp:881]
        aclrtSynchronizeStreamWithTimeout (AICPU) failed: 507018
[ERROR] recover_device_or_mark_unusable: [device_runner.cpp:481]
        Device unrecoverable after AICore error 507018:
        aclrtSynchronizeDeviceWithTimeout failed. Marking DeviceRunner unusable;
        a soft in-process reset does not clear the poison.
[WARN]  force_reset_device: [device_runner.cpp:578]
        aclrtResetDeviceForce cleared the poisoned card

Git Commit ID

Reproduced on 1e18b0de (PR #1834 head, based on origin/main); not branch-specific — the kernel matches main.

NPU Kind

Other (please specify in description) — a2a3 (--platform=a2a3; npu-smi info reports the chip as Ascend910).

Host Platform

Linux (aarch64)

Additional Context

Related: #1388 (closed — flaky test_paged_attention_spmd_ptoas with the same 507018 device-unrecoverable signature), #1789 (open — same 507018 code but a different root cause). The shared 507018 + force-reset signature across multiple paged-attention ST variants suggests a common intermittent device/AICore fault rather than a per-test bug.

[Little-oil]

Root cause (device-verified): task ring overflow misreported as a deadlock

The 507018 here is not codegen / scheduler-dispatch / AICore compute — it is the AICPU orchestrator overflowing the per-ring task window and a normal "full ring, wait to drain" being misclassified as a fatal deadlock.

The numbers

• Tasks emitted by this config (256-64-128-64-8192-32768):
  batch * q_loop * kv_blocks * inner_kernels + init
  = 256 * 4 * 128 * 4 + 256 * 4
  = 525,312 tasks
  (q_loop = ceil(64/16) = 4, kv_blocks = ceil(8192/64) = 128, 4 inner kernels per (b,q,bn) + 1 init_inplace per (b,q)).
• Default per-ring task window: PTO2_TASK_WINDOW_SIZE = 16384 — i.e. ~32x smaller than the task count.
• Allocator spin limit: PTO2_ALLOC_SPIN_LIMIT = 100000 (pto_ring_buffer.h).

Mechanism

The orchestrator (producer) allocates into a bounded ring of 16384 slots; the scheduler (consumer) drains it by completing tasks and advancing last_task_alive. Since 525,312 >> 16,384, the orchestrator repeatedly fills the ring and spins in alloc() waiting for a slot. If the scheduler advances last_alive within 100,000 spins the counter resets and it continues; if it does not drain a single slot within 100,000 consecutive spins, the full ring is reported as fatal:

[pto_ring_buffer.h] FATAL: Task Allocator Deadlock - Task Ring Full!  (No progress after 100000 spins)
[scheduler_cold_path.cpp] Fatal error (code=3) ... total_tasks=0
[aicpu_executor.cpp]      PTO2 runtime failed with rc=-3   ->  host 507018

It is flaky because it is a producer/consumer timing race: with a 16384 window the orchestrator hits "ring full" thousands of times per run, and each episode has a small chance of exceeding the 100,000-spin patience (e.g. when AICore completion latency jitters, or the orchestrator's busy-spin starves the scheduler thread on the shared AICPU). The per-episode probability compounds over thousands of episodes into a ~1/3 per-run failure rate — which also explains why it is volume-sensitive (the non-flaky 256-16-128-128 config emits far fewer tasks) and diff-independent.

There is no real circular dependency — proven below by the fact that simply enlarging the ring removes it deterministically.

PR #1853 confirms it

PR #1853 sets ring_task_window = 1048576 (the smallest power-of-2 >= 525,312) via RunConfig, so the orchestrator never fills the ring and never spins. Device A/B on a2a3:

Window	Result
default (16384)	5/9 runs hit 507018
1048576	12/12 pass

Final solution — still to be discussed

PR #1853 is a front-end workaround (it pins the root cause and unblocks CI by trading memory for ring headroom). The proper fix is open for discussion:

1. Runtime fix — auto-size the task window to the program's task count; and/or make a full ring genuinely back-pressure/block instead of false-deadlocking after a fixed spin count; and/or avoid the orchestrator busy-spin starving the scheduler (yield instead of spin).
2. PyPTO exposes the knob to users — RunConfig.ring_task_window already exists for program.run(); PR fix(st): enlarge paged-attention task ring window to avoid flaky 507018 (#1840) #1853 also wires it through the ST harness. Could document it as the supported way for high-task-volume programs.
3. Drop/skip this ST case if it is not considered representative.

cc — happy to take direction on which route to pursue.