Skip to content

perf(scheduler): avoid repeated route scans - #10

Open
Glitchfix wants to merge 1 commit into
cursor:mainfrom
Glitchfix:perf/linear-time-scheduler
Open

perf(scheduler): avoid repeated route scans#10
Glitchfix wants to merge 1 commit into
cursor:mainfrom
Glitchfix:perf/linear-time-scheduler

Conversation

@Glitchfix

@Glitchfix Glitchfix commented Aug 7, 2026

Copy link
Copy Markdown

Summary

This PR tightens the routed-token scheduler:

  • count routes once per local expert, peer, and logical scheduler thread
  • prefix the per-thread counts to recover the existing thread-major offsets
  • pad each expert segment and compute its base row
  • scatter each route once using the existing peer round-robin layout

Four 256-thread blocks cover each peer's fixed 1,024 logical scheduler threads. The output order and 256-token expert padding remain unchanged.

The old scheduler assigned a block to each expert/peer pair and scanned that peer's routes twice. For E global experts, P peers, and R routes per peer, this change reduces route reads from P*R + 2*E*R to 2*P*R. The added scratch space is E*1024 int32 counters.

For the 384-expert, EP8, 7,168-token, top-8 case, route reads fall from 44,498,944 to 917,504, plus a 393,216-counter prefix scan.

Benchmarks

Environment: H200 (SM90), torch==2.11.0+cu129. The scheduler was compiled separately for SM90 because the full MoK extension targets SM100/SM103.

EP8, 7,168 local tokens, median of 9 runs with 200 calls each:

Kimi K2.7 Code:       116.7 us -> 81.0 us  (1.44x)
GLM-5.2:               85.9 us -> 72.8 us  (1.18x)
Qwen3.5-397B-A17B:    179.3 us -> 99.3 us  (1.81x)
DeepSeek-V4-Pro:       91.6 us -> 68.2 us  (1.34x)

With 384 experts and 512 local tokens, EP8 through EP64 ranged from 0.97x to 1.03x. EP4 was slower at 32.1 us to 38.6 us (0.83x). Target-hardware performance still needs an SM100/SM103 run.

Tests

make ARCH=SM100 PYTHON=python3.12
ruff check tests/test_ops.py
python3.12 -m py_compile tests/test_ops.py
git diff --check
  • scheduler-only SM103 build passed
  • exact GPU output match against the previous scheduler for 16 realistic and adversarial cases
  • exact CPU reference match for 945 randomized and adversarial cases
  • SM100 and SM90 scheduler kernels compiled without warnings or spills

@Glitchfix
Glitchfix force-pushed the perf/linear-time-scheduler branch from 1eaa8dc to 274f83a Compare August 7, 2026 13:30
Count each peer route once and prefix per-thread counts to preserve the existing deterministic schedule order.

Signed-off-by: Shivanjan Chakravorty <schakravorty846@gmail.com>
@Glitchfix
Glitchfix force-pushed the perf/linear-time-scheduler branch from 274f83a to 79b320a Compare August 7, 2026 13:45
@Glitchfix

Copy link
Copy Markdown
Author

@StuartSul @HuiyingLi would appreciate a review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant