Shuffle tests for a single worker

[dhuang00]

Hi,

I'm using run-tests.php for testing in extension development. For my use case, I require the tests to be run one at a time and therefore cannot make use of the -j<workers> flag.

After a bit of investigation, I found that the --shuffle flag is hidden behind parallelization.

By moving shuffling to happen earlier in run_all_tests rather than run_all_tests_parallel, I was able to find some issues with my tests being order dependent.

Would this contribution be something of interest to the community?

[Girgias]

I think the change makes sense overall. But I know for sure that the tests of ext/pgsql are order dependent.

[iluuu1994]

LGTM.

But I know for sure that the tests of ext/pgsql are order dependent.

This would already be an issue, right? It's not a new flag.

[Girgias]

LGTM.

But I know for sure that the tests of ext/pgsql are order dependent.

This would already be an issue, right? It's not a new flag.

Right, but this is "prevented" by the use of the ext/pgsql/tests/CONFLICTS file to mark all tests in this folder as conflicting, which currently ensures they are run in order.

[cmb69]

But I know for sure that the tests of ext/pgsql are order dependent.

That should be fixed. :)

[Girgias]

But I know for sure that the tests of ext/pgsql are order dependent.

That should be fixed. :)

I do agree, but how it is actually setup currently, makes sense.

[nielsdos]

Looks like this was not merged yet but had an approval. I've merged this into master along with a NEWS entry.
Thanks for your contribution!

[cmb69]

@nielsdos, that patch may cause issues with ext/pgsql tests, which are order dependent according to @Girgias.

[nielsdos]

@cmb69 I interpreted "That should be fixed." as that it was already fixed.
Anyway, looking at the test code I don't think this can cause issues right now? If you ran tests in parallel with shuffle on, you would've experienced the issue already. And if you ran it in a single process then you couldn't shuffle.

[Girgias]

@cmb69 I interpreted "That should be fixed." as that it was already fixed. Anyway, looking at the test code I don't think this can cause issues right now? If you ran tests in parallel with shuffle on, you would've experienced the issue already. And if you ran it in a single process then you couldn't shuffle.

See one of my prior comments :)

But I know for sure that the tests of ext/pgsql are order dependent.

This would already be an issue, right? It's not a new flag.

Right, but this is "prevented" by the use of the ext/pgsql/tests/CONFLICTS file to mark all tests in this folder as conflicting, which currently ensures they are run in order.

Basically CONFLICTS says that all the tests in this folder conflict, and thus are never run in parallel.
I don't think it's a fundamental blocker if we add a way to prevent the tests from being shuffled (maybe adding a file ORDER_DEPENDENT ?). Would also make it easy to see which sets of tests are order dependent.

[nielsdos]

Ah okay, I can see this logic in the test runner indeed that it puts the conflicts at the end on a single worker.
It's probably going to be better to fix the test ordering instead of trying to patch it up though.