Consider moving slow tests to nightly test suite

[danielhollas]

Just noting where the current slowest tests are, perhaps some of them could be moved to nightly...

Here are six slowest test from presto tests (which take ~10 minutes overall)

31.25s call     tests/tools/archive/orm/test_links.py::test_high_level_workflow_links
16.24s call     tests/tools/archive/orm/test_links.py::test_link_flags
11.71s call     tests/tools/archive/orm/test_links.py::test_complex_workflow_graph_export_sets
11.46s call     tests/cmdline/commands/test_run.py::TestAutoGroups::test_autogroup_clashing_label
9.93s call     tests/cmdline/commands/test_group.py::TestVerdiGroup::test_add_remove_nodes
8.99s call     tests/cmdline/commands/test_group.py::TestVerdiGroup::test_move_nodes

Here are the slowest tests from the full test suite run with code coverage collection (full suite takes around 24 minutes)

  ============================= slowest 50 durations =============================
  53.37s call     tests/tools/archive/orm/test_links.py::test_high_level_workflow_links
  26.37s call     tests/tools/archive/orm/test_links.py::test_link_flags
  20.28s call     tests/tools/archive/orm/test_links.py::test_complex_workflow_graph_export_sets
  19.38s call     tests/cmdline/commands/test_run.py::TestAutoGroups::test_autogroup_clashing_label
  19.25s call     tests/cmdline/commands/test_run.py::TestAutoGroups::test_autogroup_filter_class
  17.13s call     tests/cmdline/commands/test_group.py::TestVerdiGroup::test_add_remove_nodes
  15.63s call     tests/cmdline/commands/test_group.py::TestVerdiGroup::test_move_nodes
  12.35s setup    tests/brokers/test_rabbitmq.py::test_duplicate_subscriber_identifier
  8.81s call     tests/cmdline/commands/test_group.py::TestVerdiGroup::test_delete
  8.36s teardown tests/engine/test_memory_leaks.py::test_leak_local_calcjob
  8.31s teardown tests/engine/test_memory_leaks.py::test_leak_run_process
  8.25s setup    tests/engine/test_memory_leaks.py::test_leak_local_calcjob
  7.98s call     tests/test_nodes.py::TestNodeDeletion::test_delete_cases
  7.85s setup    tests/engine/test_memory_leaks.py::test_leak_run_process
  7.70s call     tests/engine/processes/calcjobs/test_calc_job.py::test_restart_after_daemon_reset

In both cases tests/tools/archive/orm/test_links.py::test_high_level_workflow_links takes roughly 5% of overall time.

[unkcpz]

I think it is better to move those to another CI marked with slow tests, then we can have quick iteration on running all tests.

[danielhollas]

Hmm, but that's the idea of nightly tests already now? We already have admittedly too many CI jobs and test markers, I'd rather not introduce more.

In both cases tests/tools/archive/orm/test_links.py::test_high_level_workflow_links takes roughly 5% of overall time.

I think for this to decide if it should go to nightly we should investigate a bit more what this test is testing, and what is already covered in other tests. One quick dirty way to find this out would be to create a PR with this test disabled and see how it impacts the code coverage.

NOTE: I've opened #6689 which would increase the visibility of failing nightly workflows. Hopefully that should make moving slow tests to nightly less dangerous.

[danielhollas]

I think it is better to move those to another CI marked with slow tests, then we can have quick iteration on running all tests.

Perhaps we should also focus on speeding up the presto tests, to make local testing faster, and have one CI job that runs very quickly. So perhaps some tests can be moved to nightly, and some tests that we think really should run on each PR, but are slow, can be marked as slow and skipped during presto job.