wip #2693

FroyaTheHen · 2025-09-30T10:20:56Z

benchmarks below created by running the commands subsequently each one 4 times, in 5 and 2 packages workspace.

1.
BEFORE

alexandria git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build
 Time (mean ± σ):   18.649 s ±  0.110 s  [User: 57.170 s, System: 3.789 s]
 Range (min … max):  18.464 s … 18.765 s  10 runs
➜ alexandria git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w
 Time (mean ± σ):   18.669 s ±  0.313 s  [User: 57.080 s, System: 3.743 s]
 Range (min … max):  18.273 s … 19.388 s  10 runs
➜ alexandria git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build --test
 Time (mean ± σ):   46.206 s ±  0.678 s  [User: 175.756 s, System: 10.246 s]
 Range (min … max):  45.557 s … 47.938 s  10 runs
➜ alexandria git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test
 Time (mean ± σ):   42.926 s ±  0.381 s  [User: 163.054 s, System: 10.352 s]
 Range (min … max):  42.513 s … 43.764 s  10 runs (edited)

AFTER

Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build
 Time (mean ± σ):   14.262 s ±  0.241 s  [User: 47.060 s, System: 1.629 s]
 Range (min … max):  13.851 s … 14.568 s  10 runs
➜ alexandria git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w
 Time (mean ± σ):   14.804 s ±  0.637 s  [User: 47.771 s, System: 2.007 s]
 Range (min … max):  14.338 s … 16.430 s  10 runs
 Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
➜ alexandria git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build --test
 Time (mean ± σ):   61.446 s ±  1.054 s  [User: 306.312 s, System: 8.503 s]
 Range (min … max):  59.948 s … 62.926 s  10 runs
➜ alexandria git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test
 Time (mean ± σ):   60.363 s ±  0.596 s  [User: 304.159 s, System: 7.683 s]
 Range (min … max):  59.588 s … 61.467 s  10 run

2.
BEFORE

 ➜ cairo-contracts git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build
 Time (mean ± σ):   5.876 s ±  0.161 s  [User: 24.061 s, System: 3.060 s]
 Range (min … max):   5.718 s …  6.240 s  10 runs
➜ cairo-contracts git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w
 Time (mean ± σ):   62.003 s ±  1.005 s  [User: 239.437 s, System: 23.999 s]
 Range (min … max):  60.791 s … 64.260 s  10 runs
➜ cairo-contracts git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build --test
 Time (mean ± σ):   4.294 s ±  0.164 s  [User: 16.342 s, System: 2.281 s]
 Range (min … max):   4.138 s …  4.565 s  10 runs
➜ cairo-contracts git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test
 Time (mean ± σ):   71.009 s ±  1.017 s  [User: 265.053 s, System: 28.550 s]
 Range (min … max):  69.744 s … 73.247 s  10 runs

AFTER

➜ cairo-contracts git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build
 Time (mean ± σ):   3.643 s ±  0.085 s  [User: 15.488 s, System: 1.885 s]
 Range (min … max):   3.518 s …  3.816 s  10 runs
➜ cairo-contracts git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w
 Time (mean ± σ):   29.767 s ±  0.199 s  [User: 156.279 s, System: 8.768 s]
 Range (min … max):  29.472 s … 30.089 s  10 runs
➜ cairo-contracts git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build --test
 Time (mean ± σ):   4.327 s ±  0.127 s  [User: 16.255 s, System: 2.494 s]
 Range (min … max):   4.099 s …  4.613 s  10 runs
➜ cairo-contracts git:(main) ✗ hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test
 Time (mean ± σ):   61.297 s ±  0.543 s  [User: 224.448 s, System: 22.435 s]
 Range (min … max):  60.622 s … 62.580 s  10 runs

FroyaTheHen · 2025-10-01T18:21:18Z

scarb/tests/build_assets.rs

        .stdout_matches(indoc! {r#"
            [..] Compiling badpkg v0.1.0 ([..])
            error: package `badpkg v0.1.0 ([..])` asset is not a file: [..]/assets
+            error: could not compile `badpkg` due to previous error


are those changes acceptable? 👇

yes, they are

FroyaTheHen · 2025-10-03T10:10:54Z

additionally...
3.
BEFORE:

➜ garaga git:(main) hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build
 Time (mean ± σ):   5.385 s ±  0.175 s  [User: 19.707 s, System: 1.170 s]
 Range (min … max):   5.048 s …  5.662 s  10 runs
➜ garaga git:(main) hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w
 Time (mean ± σ):   5.332 s ±  0.170 s  [User: 19.924 s, System: 1.174 s]
 Range (min … max):   4.969 s …  5.521 s  10 runs
➜ garaga git:(main) hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build --test
 Time (mean ± σ):   15.061 s ±  0.708 s  [User: 44.595 s, System: 2.691 s]
 Range (min … max):  13.910 s … 16.230 s  10 runs
➜ garaga git:(main) hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test
 Time (mean ± σ):   14.841 s ±  0.362 s  [User: 44.069 s, System: 2.632 s]
 Range (min … max):  14.237 s … 15.377 s  10 runs

AFTER:

➜ garaga git:(main) hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build
 Time (mean ± σ):   5.273 s ±  0.154 s  [User: 17.709 s, System: 0.961 s]
 Range (min … max):   4.974 s …  5.468 s  10 runs
➜ garaga git:(main) hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w
 Time (mean ± σ):   5.352 s ±  0.090 s  [User: 17.600 s, System: 0.979 s]
 Range (min … max):   5.150 s …  5.454 s  10 runs
➜ garaga git:(main) hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build --test
 Time (mean ± σ):   14.853 s ±  0.553 s  [User: 41.802 s, System: 2.684 s]
 Range (min … max):  13.827 s … 15.506 s  10 runs
➜ garaga git:(main) hyperfine --warmup=1 '/Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test'
Benchmark 1: /Users/kingacepielik/starkware/scarb/target/release/scarb build -w --test
 Time (mean ± σ):   14.606 s ±  0.637 s  [User: 41.131 s, System: 2.501 s]
 Range (min … max):  13.719 s … 15.932 s  10 runs

FroyaTheHen · 2025-10-13T12:43:04Z

The second chnges benchmarks were done with changes in cairo
https://github.com/starkware-libs/cairo/blob/8b031889ea488c55817a53300c1c048a548d112c/crates/cairo-lang-test-plugin/src/lib.rs#L108-L113
👇

    let contracts = if tests_compilation_config.starknet { tests_compilation_config.contract_declarations.unwrap_or_else(|| {
        find_contracts(
            db,
            tests_compilation_config.contract_crate_ids.unwrap_or_else(|| db.crates()),
        )
    }) } else { vec![] };

1️⃣ cairo-contracts

/Users/kingacepielik/starkware/scarb/target_before/release/scarb build --test -w
 Time (mean ± σ):   69.337 s ±  0.525 s  [User: 264.522 s, System: 26.457 s]
 Range (min … max):  68.519 s … 70.030 s  10 runs

Benchmark 1: /Users/kingacepielik/starkware/scarb/target_before/release/scarb build --test
 Time (mean ± σ):   4.314 s ±  0.150 s  [User: 16.226 s, System: 2.460 s]
 Range (min … max):   4.177 s …  4.597 s  10 runs

Benchmark 1: /Users/kingacepielik/starkware/scarb/target_before/release/scarb build
 Time (mean ± σ):   5.779 s ±  0.120 s  [User: 24.010 s, System: 3.189 s]
 Range (min … max):   5.583 s …  5.966 s  10 runs

Benchmark 1: /Users/kingacepielik/starkware/scarb/target_before/release/scarb build -w
 Time (mean ± σ):   61.200 s ±  0.412 s  [User: 238.167 s, System: 24.746 s]
 Range (min … max):  60.568 s … 61.910 s  10 runs

Benchmark 1: /Users/kingacepielik/starkware/scarb/target_after/release/scarb build --test -w
 Time (mean ± σ):   52.952 s ±  0.556 s  [User: 200.225 s, System: 13.743 s]
 Range (min … max):  52.210 s … 53.840 s  10 runs

Benchmark 1: /Users/kingacepielik/starkware/scarb/target_after/release/scarb build --test
 Time (mean ± σ):   4.260 s ±  0.088 s  [User: 16.066 s, System: 2.211 s]
 Range (min … max):   4.138 s …  4.398 s  10 runs

Benchmark 1: /Users/kingacepielik/starkware/scarb/target_after/release/scarb build
 Time (mean ± σ):   3.603 s ±  0.121 s  [User: 15.497 s, System: 1.913 s]
 Range (min … max):   3.452 s …  3.813 s  10 runs

Benchmark 1: /Users/kingacepielik/starkware/scarb/target_after/release/scarb build -w
 Time (mean ± σ):   23.452 s ±  0.138 s  [User: 110.981 s, System: 5.554 s]
 Range (min … max):  23.282 s … 23.686 s  10 runs

2️⃣ alexandria

Benchmark 1: /Users/kingacepielik/starkware/scarb/target_before/release/scarb build --test -w
  Time (mean ± σ):     42.729 s ±  0.573 s    [User: 162.977 s, System: 9.683 s]
  Range (min … max):   41.986 s … 44.137 s    10 runs
 
Benchmark 1: /Users/kingacepielik/starkware/scarb/target_before/release/scarb build --test
  Time (mean ± σ):     42.673 s ±  0.261 s    [User: 162.824 s, System: 9.670 s]
  Range (min … max):   42.298 s … 43.061 s    10 runs
 
Benchmark 1: /Users/kingacepielik/starkware/scarb/target_before/release/scarb build
  Time (mean ± σ):     17.262 s ±  0.209 s    [User: 52.481 s, System: 3.296 s]
  Range (min … max):   16.906 s … 17.566 s    10 runs
 
Benchmark 1: /Users/kingacepielik/starkware/scarb/target_before/release/scarb build -w
  Time (mean ± σ):     17.307 s ±  0.127 s    [User: 52.455 s, System: 3.401 s]
  Range (min … max):   17.129 s … 17.477 s    10 runs
 
Benchmark 1: /Users/kingacepielik/starkware/scarb/target_after/release/scarb build --test -w
  Time (mean ± σ):     28.770 s ±  0.548 s    [User: 99.583 s, System: 3.972 s]
  Range (min … max):   27.938 s … 29.759 s    10 runs
 
Benchmark 1: /Users/kingacepielik/starkware/scarb/target_after/release/scarb build --test
  Time (mean ± σ):     29.910 s ±  0.689 s    [User: 100.699 s, System: 4.487 s]
  Range (min … max):   29.055 s … 31.164 s    10 runs
 
Benchmark 1: /Users/kingacepielik/starkware/scarb/target_after/release/scarb build
  Time (mean ± σ):     11.746 s ±  0.216 s    [User: 30.429 s, System: 1.452 s]
  Range (min … max):   11.423 s … 12.063 s    10 runs
 
Benchmark 1: /Users/kingacepielik/starkware/scarb/target_after/release/scarb build -w
  Time (mean ± σ):     11.602 s ±  0.167 s    [User: 30.267 s, System: 1.408 s]
  Range (min … max):   11.404 s … 11.951 s    10 runs

FroyaTheHen force-pushed the kinga/wip-share-compiler-db-across-targets branch 4 times, most recently from e5dad0a to 01eeb69 Compare October 1, 2025 18:20

FroyaTheHen commented Oct 1, 2025

View reviewed changes

FroyaTheHen self-assigned this Oct 1, 2025

FroyaTheHen marked this pull request as ready for review October 1, 2025 18:23

FroyaTheHen requested review from a team and Draggu as code owners October 1, 2025 18:23

FroyaTheHen force-pushed the kinga/wip-share-compiler-db-across-targets branch from 01eeb69 to 79e3b57 Compare October 2, 2025 06:51

share compiler db across targets

69b5f3e

FroyaTheHen marked this pull request as draft October 13, 2025 08:01

remove db.crates() usages

1566660

FroyaTheHen force-pushed the kinga/wip-share-compiler-db-across-targets branch from 79e3b57 to 1566660 Compare October 13, 2025 11:47

FroyaTheHen mentioned this pull request Oct 20, 2025

Kinga/share compiler db across pkgs #2764

Closed

cairo-fork

464f225

FroyaTheHen closed this Oct 29, 2025

FroyaTheHen deleted the kinga/wip-share-compiler-db-across-targets branch October 29, 2025 12:28

FroyaTheHen mentioned this pull request Oct 29, 2025

Consider sharing the compiler db across multiple compilation targets #2416

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

wip #2693

wip #2693

Uh oh!

FroyaTheHen commented Sep 30, 2025 •

edited

Loading

Uh oh!

FroyaTheHen Oct 1, 2025 •

edited

Loading

Uh oh!

mkaput Oct 2, 2025

Uh oh!

FroyaTheHen commented Oct 3, 2025 •

edited

Loading

Uh oh!

FroyaTheHen commented Oct 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wip #2693

wip #2693

Uh oh!

Conversation

FroyaTheHen commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FroyaTheHen Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkaput Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

FroyaTheHen commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FroyaTheHen commented Oct 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

FroyaTheHen commented Sep 30, 2025 •

edited

Loading

FroyaTheHen Oct 1, 2025 •

edited

Loading

FroyaTheHen commented Oct 3, 2025 •

edited

Loading