Create a nightly long running CI job

[plemarquand]

Currently the CI tests take about 12-14 minutes on the 6.0 and main CI nodes. This approaches the upper bound of what should be acceptable for a PR CI run, and also approaches the current CI job timeout limit of 15 minutes.

Currently the longest running tests in CI are those that build/run tests and verify that they ran correctly. However, in order to stay under the 15 minute timeout, these tests are limited in scope. For instance there is no testing of Parallel or Debug build configurations and there are no tests that exercise build + run functionality. Some of these tests have already been written but are marked with @slow, which excludes them from PR CI runs.

In order to expand test coverage in these areas I propose a nightly CI job with a longer timeout and runs all tests.

• This job should run all tests including those marked with @slow by omitting the FAST_TEST_RUN environment variable.
• The nightly CI job should run across all of the same nodes as the current PR CI jobs
• An email or Slack notification should be sent to the CODEOWNERS if the job fails.

Alternatives Considered

• We could to parallelize the PR CI runs, breaking up the collection of tests in to groups and distributing them across multiple nodes.
  • This requires more maintenance than the proposed solution, as new tests would need to be grouped into a set of tests to run on a given node.
  • This makes gathering code coverage information difficult.
• Increasing the PR CI job timeout
  • This is easiest, but doesn't scale well as the suite grows. Ideally PR CI runs are < 15 minutes.