Skip to content

benchmarks.md

Latest commit

 

History

History
115 lines (78 loc) · 6.32 KB

benchmarks.md

File metadata and controls

115 lines (78 loc) · 6.32 KB
Raw

Benchmarks

dau-sim includes a benchmark suite under dau_sim/benchmarks/ that tracks simulation throughput across backends and measures internal compilation and evaluation performance. All numbers below were collected on an Apple M1 Pro (arm64, CPython 3.12) unless noted otherwise.

Benchmark suite

Cross-simulator comparison (bench_cross_simulators.py)

Compares dau-sim against Amaranth's native pysim simulator and Verilator on the same design: a 32-bit enabled counter running for a configurable number of cycles (set via DAU_BENCH_CYCLES, default 5,000).

Backends under test:

Backend What it measures
dau-sim from_amaranth → compile_module → cm.run(return_traces=False)
amaranth-sim Amaranth's built-in Python simulator (Simulator + generator process)
cxxsim Amaranth's optional CXXRTL-backed simulator (skipped when unavailable)
verilator-compile-run Full pipeline: write Verilog → verilator --binary → run executable
verilator-runtime Pre-compiled Verilator binary execution only (compile cost excluded)

Run the benchmark:

DAU_BENCH_CYCLES=100000 pytest dau_sim/benchmarks/bench_cross_simulators.py \
    --benchmark-only --benchmark-columns=mean,stddev,median

Results — 500k cycles

Backend Mean vs Verilator runtime vs Amaranth
verilator-runtime 93 ms 1.0× 87× faster
dau-sim 2.49 s 27× slower 3.2× faster
verilator-compile-run 4.85 s 52× slower 1.7× faster
amaranth-sim 8.08 s 87× slower 1.0×

Results — 100k cycles

Backend Mean vs Verilator runtime vs Amaranth
verilator-runtime 22.5 ms 1.0× 70× faster
dau-sim 154 ms 6.8× slower 10× faster
amaranth-sim 1,568 ms 70× slower 1.0×
verilator-compile-run 4,509 ms 200× slower 2.9× faster

Key observations:

  • dau-sim is 3–10× faster than Amaranth's pysim across cycle counts, with the gap widening at higher counts due to lower per-tick overhead.
  • Verilator runtime is the throughput ceiling — compiled C++ executing a simple counter at ~5M cycles/sec. dau-sim is 7–27× behind depending on cycle count.
  • Verilator compile+run is slower than dau-sim for small-to-medium workloads because compilation dominates. dau-sim's zero-compile-step workflow gives it a significant advantage for iterative development.
  • The Amaranth counter includes a reset signal (rst), which prevents dau-sim from using its fastest batch execution path. For reset-free IR designs, dau-sim achieves ~30 ms for 100k cycles (only 1.3× slower than Verilator runtime).

Compile partitioning (bench_compile_partitioning.py)

Measures compile_module time as a function of the number of independent combinational blocks in a design. Tests N = 16, 64, 256, 1024 blocks, each a simple assign o = a + const. This benchmark validates that the dependency-analysis and block-partitioning phase scales well.

pytest dau_sim/benchmarks/bench_compile_partitioning.py --benchmark-only

Selective settle (bench_selective_settle.py)

Measures runtime of a sequential design with N independent combinational components and configurable statements per component (1, 8, 32). Verifies that the selective-settle optimization — only re-evaluating combinational blocks whose inputs actually changed — keeps per-tick cost proportional to active components rather than total design size.

pytest dau_sim/benchmarks/bench_selective_settle.py --benchmark-only

Execution modes and optimization tiers

dau-sim uses several execution strategies depending on the design and whether trace output is needed:

CSP compiled path (default)

The compiler generates flat Python functions from the IR statement/expression trees (dau_sim/compiler/codegen.py) and executes them inside a single CSP node. Per-tick work:

  1. Toggle clock signals at the correct half-period
  2. Detect rising/falling edges via inlined comparisons
  3. Execute the compiled sequential block for each fired domain
  4. Re-evaluate affected combinational blocks (selective settle)
  5. Optionally emit trace output

This path supports all designs including those with resets, combinational logic, and memories.

Fast-tick path

For designs with no combinational logic, no memories, and no resets, the compiler generates a single _fast_tick(S, clock_arr, tc) function that inlines the clock toggle, edge detection, and sequential block body. This eliminates function-call overhead and changed-set tracking.

Batch no-trace path

When return_traces=False and the design qualifies for fast-tick, dau-sim bypasses the CSP engine entirely and runs all ticks in a pure Python for loop. This eliminates ~400k CSP scheduling events for a 200k-tick simulation.

Performance by execution mode (100k-cycle counter)

Mode Time Speedup vs interpreter Throughput
Interpreter (pre-optimization baseline) 420 ms 1.0× 238k cycles/sec
CSP compiled (with fast-tick) 71 ms 5.9× 1.4M cycles/sec
Batch no-trace 30 ms 13.9× 3.3M cycles/sec

Running benchmarks

# Run all benchmarks
pytest dau_sim/benchmarks/ --benchmark-only

# Cross-simulator comparison with custom cycle count
DAU_BENCH_CYCLES=100000 pytest dau_sim/benchmarks/bench_cross_simulators.py --benchmark-only

# Save results to JSON for tracking
pytest dau_sim/benchmarks/bench_cross_simulators.py --benchmark-only \
    --benchmark-save=cross-runtime --benchmark-storage=dau_sim/benchmarks/results

Stored results live in dau_sim/benchmarks/results/ for historical comparison.