ctest is run for single tests, but with -j10

[guntern]

Hi,
having other issues with cmake-tools, I started to look close at what it is doing. When running tests I ran into #3082, which made it apparant to me, that each test appears to be run with a dedicated ctest-call with -R to filter the test. Nevertheless, the option -j10 is used for multithreading. Does this make sense? Especially, when running multiple tests at once (or even all) I would assume that it is faster to run all tests with -j10 (or a different number) in one ctest-call and filter the results from the output.
Or is the reason for the chosen approach, that different testing frameworks might have different output making the parsing difficult?
I think especially @xisui-MSFT might help to answer those questions.
Best regards

[xisui-MSFT]

This is by design for now (also stated in the CHANGELOG), but can certainly be improved.

When multiple tests are run in a single command (doesn't matter using -j or not), test result file is generated when all tests finish. So in order to know the process of the test run, we'll need to parse the ctest output. However, some ctest options like --output-on-failure prints out whatever printed out in the tests, and therefore, the parser can be easily fooled.

One workaround is to run <job number> of tests each time (i.e., ctest -R test1|test2|test3 -j 3), which isn't perfect, but mitigates this issue as long as test names are not too long, and most of the tests take roughly the same amount of time.

This isn't planned in the original implementation. We may work on this if we see related feedbacks, or contributions from the community are also welcome.

[guntern]

Thank you for your answer, That's the reason, why I started this as a discussion. It simply can be a design choice. Or there could be implications with other test setups, which I am not aware of.

From my point of view it simply appeared faster to run the tests multithreaded if multiple tests are run at once.

[rjra100]

Just to add my 2c, I think we really need a convenient way to run the tests in parallel - running ours sequentially is way too slow, and even if I multiselect the tests (which would in any case be rather inconvenient), I don't seem to get parallel execution. It's not even possible via the "Run CTest" button on the status bar, as that now redirects to the test pane.
I'm going to have to reinstall CMake Test Explorer (which already ran tests in parallel) until parallel execution via CMake Tools becomes a thing.

[gcampbell-msft]

@rjra100 Thanks for your feedback. To make sure that your request for parallel test execution is tracked, could you please make sure to thumbs up the issue here: #3091. This helps us easily see what we need to be prioritizing, thanks!

[bahmanrafatjoo]

In addition to the efficiency point of view, running tests in parallel is really useful in exposing weaknesses in the tests such as dependencies or collisions on shared resources (e.g. multiple tests using the same input/output files). Also on a slower machine the parallel load changes the timing of the tests, helping to weed out poorly written tests.
Have thumbed up #3091.