TRT-2292: Add support for test isolation #30478

xueqzhan · 2025-11-11T20:42:19Z

This PR adds a testScheduler that

Add conflict handler that will prevent tests with same conflict from running at the same time.
Conflicts are detected within conflictGroup, which is used to support future conflict modes: instance, bucket and exec
This also adds support for taint/toleration. The idea is to allow some tests running while forbidding all others from running at the same time.

If taint/toleration function looks ok, I will add the structure in the extension library and adjust here accordingly.

/hold for review and further test

openshift-ci-robot · 2025-11-11T20:42:23Z

@xueqzhan: This pull request references TRT-2292 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

This PR adds a testScheduler that

Add conflict handler that will prevent tests with same conflict from running at the same time.

Conflicts are detected within conflictGroup, which is used to support future conflict modes: instance, bucket and exec

This also adds support for taint/toleration. The idea is to allow some tests running while forbidding all others from running at the same time.

/hold for review and further test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-11-11T20:52:47Z

@xueqzhan: This pull request references TRT-2292 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

This PR adds a testScheduler that

Add conflict handler that will prevent tests with same conflict from running at the same time.

Conflicts are detected within conflictGroup, which is used to support future conflict modes: instance, bucket and exec

This also adds support for taint/toleration. The idea is to allow some tests running while forbidding all others from running at the same time.

If taint/toleration function looks ok, I will add the structure in the extension library and adjust here accordingly.

/hold for review and further test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-11-11T20:56:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xueqzhan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [xueqzhan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/test/ginkgo/queue.go

stbenjam

Looks really good, just a couple comments

pkg/test/ginkgo/queue.go

openshift-ci-robot · 2025-11-18T02:36:50Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

openshift-ci-robot · 2025-11-18T03:08:32Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

xueqzhan · 2025-11-18T13:47:21Z

/retest-required

openshift-trt · 2025-11-18T15:11:21Z

Job Failure Risk Analysis for sha: df32893

Job Name	Failure Risk
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6	IncompleteTests Tests for this run (22) are below the historical average (2192): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

xueqzhan · 2025-11-19T12:32:14Z

/retest-required

stbenjam · 2025-11-19T14:13:53Z

pkg/test/ginkgo/test_suite.go

+	// isolation defines conflict groups, mode, taints, and tolerations for test isolation
+	isolation extensiontests.Isolation


Do we need to copy this or could we just access it via spec?

Doesn't seem that we are copying spec in extensionTestSpecsToOriginTestCases. I can and use the spec instead if that is preferred.

We definitely should, we have a field for it. Retry copies it -- looks like maybe an oversight that it wasn't copied in the original creation of them

stbenjam · 2025-11-19T14:15:36Z

pkg/test/ginkgo/queue.go

+// getTestConflictGroup returns the conflict group for a test.
+// Conflicts are only checked within the same conflict group.
+func getTestConflictGroup(test *testCase) string {
+	return "default"


This needs to get replaced from the content in the spec right?

spec doesn't define group. Group is designed to support mode.

Sorry I'm not quite understanding what this function does. Isolation has two fields relevant, Conflicts and Mode. I think we agreed to only support mode=exec for now, but shouldn't we get the Conflicts out of the Isolation struct?

Why does this always return default?

The idea was to have a framework to support mode in the future. So conflict is only checked within a conflictGroup. Right now all tests belong to default group and therefore work like mode=exec. But just in case another mode is needed, more conflictGroup will be created for that purpose.

Is there a reason not to call it mode here instead of conflict group, and use the name we're implementing ("exec")? It's not clear to me "group" is linked to the "mode"

Mode is not clear to indicate a grouping mechanism. But conflictGroup is clear about its functionality of grouping. But configGroup can be used to implement mode. So I think conflictGroup is still better.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 11, 2025

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 11, 2025

openshift-ci bot requested review from deads2k and p0lyn0mial November 11, 2025 20:54

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 11, 2025

stbenjam reviewed Nov 12, 2025

View reviewed changes

pkg/test/ginkgo/queue.go Outdated Show resolved Hide resolved

pkg/test/ginkgo/queue.go Outdated Show resolved Hide resolved

stbenjam reviewed Nov 15, 2025

View reviewed changes

pkg/test/ginkgo/queue.go Outdated Show resolved Hide resolved

pkg/test/ginkgo/queue.go Outdated Show resolved Hide resolved

pkg/test/ginkgo/queue.go Outdated Show resolved Hide resolved

pkg/test/ginkgo/queue.go Show resolved Hide resolved

xueqzhan added 10 commits November 17, 2025 16:10

Schedule tests with conflicts configured

a6a923f

Fix some racing issues

333855c

Add test case for conflict tracker

43f600d

Add support for taint and toleration

14295f8

Rename conflictTracker to testScheduler

f264969

Add conflict group to support instance and bucket mode

6c929ff

Implement testQueue to keep test order even with conflict handling

b2c029a

Remove place holders for isolation mode handling

53b61e9

Address review comments

020c59e

Use taint and toleration from isolation struct

745cf1d

xueqzhan force-pushed the test-isolation branch from 109ff7f to 745cf1d Compare November 18, 2025 01:07

Simplify TestScheduler interface

4df7d1f

Code cleanup

1c730e3

Add context to GetNextTestToRun

df32893

stbenjam reviewed Nov 19, 2025

View reviewed changes

xueqzhan added 2 commits November 21, 2025 13:33

Address review comment about using spec

d0e1d91

Add comment for conflict group

e61f83c

		// isolation defines conflict groups, mode, taints, and tolerations for test isolation
		isolation extensiontests.Isolation

TRT-2292: Add support for test isolation #30478

Are you sure you want to change the base?

TRT-2292: Add support for test isolation #30478

Conversation

xueqzhan commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 11, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 11, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Nov 11, 2025

Uh oh!

Uh oh!

Uh oh!

stbenjam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 18, 2025

Uh oh!

openshift-ci-robot commented Nov 18, 2025

Uh oh!

xueqzhan commented Nov 18, 2025

Uh oh!

openshift-trt bot commented Nov 18, 2025

Uh oh!

xueqzhan commented Nov 19, 2025

Uh oh!

stbenjam Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

xueqzhan Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stbenjam Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stbenjam Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

xueqzhan Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

stbenjam Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

xueqzhan Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

stbenjam Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

xueqzhan Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xueqzhan commented Nov 11, 2025 •

edited

Loading

openshift-ci-robot commented Nov 11, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Nov 11, 2025 •

edited by openshift-ci bot

Loading

xueqzhan Nov 19, 2025 •

edited

Loading

stbenjam Nov 19, 2025 •

edited

Loading