TRT-2423: feat(tests): add BigQuery-based test duration tracking #30493
Conversation
Add test duration query functionality from BigQuery and implement duration-aware test sorting to optimize test suite execution order. Includes new "generate durations" command and required dependencies. Co-Authored-By: Claude <[email protected]>
|
Pipeline controller notification For optional jobs, comment |
|
Skipping CI for Draft Pull Request. |
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
vendor-update
stbenjam
changed the title
|
@stbenjam: This pull request references TRT-2423 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.21.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-reference
|
@stbenjam: This pull request references TRT-2423 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.21.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@stbenjam: This pull request references TRT-2423 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.21.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/test e2e-aws-ovn-fips |
| } | ||
|
|
||
| // buildQuery constructs the BigQuery SQL query | ||
| func (o *DurationsOptions) buildQuery() string { |
There was a problem hiding this comment.
Have you tested it with openshift-ci-data-analysis.ci_data.JobTestSummaryByDate noted in TRT-2389? Thought being that using the daily summary would be more cost effective over time. Also considered this being part of jobrunhistoricaldataanalyzer
There was a problem hiding this comment.
JobTestSummaryByDate appears to be job-level data? I only care about average test duration
There was a problem hiding this comment.
Oh, maybe TestsSummaryByDate? It is still grouped by job. I'm not sure I care about job-level data, I just need a rough estimate for test duration, and my JUnit query is only 13GB here and I'd guess we only care to run it at most once a day or week.
I'm a little concerned about building things in our repos on bespoke tables without schema management, especially if we'll plan to eventually try to change our data sources. Relying on JUnit makes various things easier to move en masse if we need to because its widely used
There was a problem hiding this comment.
Yeah, openshift-ci-data-analysis.ci_data.TestsSummaryByDate my bad.
We have a whole process setup that does this already for disruption and alert data. We have a job that runs weekly to update the data. It would make sense to me to have this logic running in the same process and if ci-tool already works with biq query then keeping the logic over there makes more sense to me as well.
| var testDurationsData []byte | ||
|
|
||
| // TestDurationData represents the duration information for a single test | ||
| type TestDurationData struct { |
There was a problem hiding this comment.
We have been talking about including more than just the duration data with the test names so we can consider things like pass rate over a period of time to consider dropping tests that never fail, etc. That would be good to consider here as well if possible.
|
/test e2e-aws-ovn-fips |
|
Interesting that 30493 - aws-ovn-fips ran for |
|
/test e2e-aws-ovn-fips |
|
@stbenjam: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Most recent aws-ovn-fips run for #30445 was for |
3 participants
Similar to #30445, but includes duration information for all tests from BigQuery. Not sure if the duration JSON generator should live here or elsewhere, probably elsewhere given the dependency size (ci-tools?)
Add test duration query functionality from BigQuery and implement duration-aware test sorting to optimize test suite execution order. Includes new "generate durations" command and required dependencies.