Skip to content

Fix: add store barriers in AICPU-AICore handshake on aarch64#521

Merged
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
hw-native-sys-bot:fix/handshake-store-barriers
Apr 12, 2026
Merged

Fix: add store barriers in AICPU-AICore handshake on aarch64#521
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
hw-native-sys-bot:fix/handshake-store-barriers

Conversation

@hw-native-sys-bot

Copy link
Copy Markdown
Collaborator

Summary

  • Add OUT_OF_ORDER_STORE_BARRIER() after volatile handshake stores (aicpu_ready, aicpu_regs_ready) in handshake_all_cores() across all five runtime variants
  • On aarch64's weak memory model, stores to volatile fields can remain in the CPU store buffer while the writing thread spins on a different field, invisible to the polling AICore thread → mutual deadlock
  • Single-device execution is unaffected (OS context switches drain the store buffer); the bug triggers under concurrent multi-device sim (4+ ChipWorkers with GIL released)

Root cause analysis

GDB core dump of a hung 4-worker sim process showed:

  • 78 AICore threads stuck at aicore_executor.cpp:69 — idle-polling read_reg(DATA_MAIN_BASE) for task dispatch
  • AICPU threads stuck at aicpu_executor.cpp:955 — idle spin in resolve_and_dispatch
  • Worker 14's aicpu_regs_ready was 1 in memory but the AICore thread never observed it (store buffer not drained)

The existing emergency_shutdown() path in the same files already uses the correct barrier pattern — the handshake path was the only omission.

Files changed

File Barrier added after
src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp aicpu_ready=1 loop, aicpu_regs_ready=1
src/a2a3/runtime/aicpu_build_graph/aicpu/aicpu_executor.cpp aicpu_ready=1 loop, aicpu_regs_ready=1
src/a2a3/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp aicpu_ready=1 loop, aicpu_regs_ready=1
src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp aicpu_ready=1 loop, aicpu_regs_ready=1
src/a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp aicpu_ready=1 loop, aicpu_regs_ready=1

Test plan

  • python ci.py -p a2a3sim — 20/20 pass (1 worker, baseline)
  • 4-worker sim stress: pass rate 35% → 80% (20 runs)
  • python ci.py -p a5sim — sim regression check
  • Hardware device tests unaffected (barrier is a no-op or benign on real NPU)

🤖 Generated with Claude Code

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces OUT_OF_ORDER_STORE_BARRIER() calls across several aicpu_executor.cpp files to ensure memory visibility of status flags before proceeding to synchronization loops. While these additions improve consistency, the reviewer identified a recurring issue where an additional barrier is needed between register initialization and the setting of the aicpu_regs_ready flag to prevent race conditions on architectures with weak memory models like aarch64.

Comment thread src/a2a3/runtime/aicpu_build_graph/aicpu/aicpu_executor.cpp
Comment thread src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp
Comment thread src/a2a3/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp
Comment thread src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp
Comment thread src/a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp
Add OUT_OF_ORDER_STORE_BARRIER() after volatile handshake stores
(aicpu_ready, aicpu_regs_ready) in handshake_all_cores() across all
five runtime variants.

On aarch64's weak memory model, stores to volatile fields can remain
in the CPU store buffer while the writing thread spins on a different
field, invisible to the polling AICore thread — causing mutual deadlock.

The barrier before aicpu_regs_ready ensures register initialization
stores from platform_init_aicore_regs() are globally visible before
the flag, so AICore does not proceed with uninitialized registers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ChaoWao ChaoWao force-pushed the fix/handshake-store-barriers branch from e30121d to 05428ba Compare April 12, 2026 04:28
@ChaoWao ChaoWao merged commit c0ff619 into hw-native-sys:main Apr 12, 2026
13 checks passed
@ChaoWao ChaoWao deleted the fix/handshake-store-barriers branch April 12, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants