Investigate why build scripts are slower than presets

[jrhemstad]

I haven't done a deep enough dive on what the scripts are doing internally so I can't comment on the difference, but I've noticed empirically that it is much slower than just directly using the preset.

The feedback loop was really slow for me personally. If I made a small syntactical error in writing my code, it would take 2-3 minutes from running the build scripts before I got to see the error. That compounds really hard, where as running the preset build directly the feedback is nearly instantaneous.

Originally posted by @ZelboK in #647 (comment)

[jrhemstad]

@allisonvacanti I'd be curious to hear your thoughts on this.

[ZelboK]

https://discord.com/channels/1019361803752456192/1161051667945508884/1169068895500505150

^ from the conversation on discord

The way I'm running the tests for cccl has a slow feedback loop. I'm wondering if it can be optimized somehow. (btw, I haven't had time to look into the new presets yet. I still just run from the command line)

Let's say I'll make a small change in one test suite. PARALLEL_LEVEL=24 ./ci/test_cub.sh -cxx clang++-16 -std 17 -arch 86 running this for example will take ~190 seconds before it begins the build process.
I imagine i'm doing something inefficient. (Also I'm quite mentally tired so i apologize if im missing obvious 😂)

Any way to speed up this feedback loop?
Wes M — Yesterday at 1:01 AM
After having ran the script to produce a configuration you could probably get away with just issuing cmake --build build/latest.

Really the CI scripts are best used for checking that everything will pass in CI before submitting the changes to GitHub. Ideally you might test a few combinations locally and hence why they would go through all the setup and teardown that takes such a long time.

190 seconds for configuration is... Pretty massive though.

When I am done with work I can help provide more "data" to help diagnose.

[jrhemstad]

for example will take ~190 seconds before it begins the build process.

That definitely doesn't sound right. There must be something else going on here.

[ZelboK]

Sorry now that I think about it, it isn't a proper description of what happened.

For example, I ran the script with time when I had an error that I knew would fail, and it took 190 seconds to finish running(so the build finished at that point). The script only had to rebuild something like 34 files though, and failed at like 3/34. Does that make sense? Hopefully I described it better.

Also perhaps my network was slower than usual at the time, which would be unrelated to the script itself. Not sure if it downloads anything during the script.

When I have time later today I'll do some proper benchmarking and report back. I am doing too much context switching right now 😅

[alliepiper]

Is it taking 190s to see the error, or 190s for the script to finish after the error appears?

It may be that some of the compilations continue after the error is hit, and the script stays live until all of these child processes finish. Meanwhile, building the preset through an IDE or something would likely flag the error as soon as it appears, even if there's still compilation ongoing in the background.

[miscco]

Similar issue here, when using the test preset it is instantaneous starting lit, when only building it takes ages

[alliepiper]

The scripts force both the configure and build steps to run no matter what. That's likely where the overhead is coming from.

A do-nothing build should be practically free, but rerunning CMake can be somewhat expensive, especially on windows.

[miscco]

Its also like that on linux and I dont understand why configure + lit with NoopExecutor is so much slower than configure + lit

[alliepiper]

It might be the lit parallelism settings.