junit2jira

Utility tools for handling test failures

Build

go build  -o . ./...

Test

go test ./...

Usage

This repo provides two cli tools:

• junit2jira
• flakechecker

junit2jira

junit2jira supports conversion of test failures to jira issues. It also posts Slack messages for new failures and imports test results into DB.

Usage

Usage of junit2jira:
  -base-link string
    	Link to source code at the exact version under test.
  -build-id string
    	Build job run ID.
  -build-link string
    	Link to build job.
  -build-tag string
    	Built tag or revision.
  -csv-output string
    	Convert XML to a CSV file (use dash [-] for stdout)
  -debug
    	Enable debug log level
  -dry-run
    	When set to true issues will NOT be created.
  -enable-auto-priority
    	Enable automatic priority escalation based on comment count.
  -html-output string
    	Generate HTML report to this file (use dash [-] for stdout)
  -jira-url string
    	Url of JIRA instance (default "https://issues.redhat.com/")
  -job-name string
    	Name of CI job.
  -junit-reports-dir string
    	Dir that contains jUnit reports XML files
  -orchestrator string
    	Orchestrator name (such as GKE or OpenShift), if any.
  -priority-thresholds string
        Comma-separated minimum comment counts for priority escalation (Minor,Normal,Major,Blocker,Critical). (default "2,10,50,100,200")
  -priority-window-days int
        Number of recent days used for automatic priority escalation. (default 30)
  -slack-output string
    	Generate JSON output in slack format (use dash [-] for stdout)
  -threshold int
    	Number of reported failures that should cause single issue creation. (default 10)
  -timestamp string
    	Timestamp of CI test. (default "2023-09-04T17:50:36+02:00")
  -v	short alias for -version
  -version
    	print version information and exit

Auto-Priority Escalation

Automatic priority escalation is disabled by default. With --enable-auto-priority, junit2jira evaluates an existing issue after successfully adding a comment to it. It counts comments in one recent activity window, which defaults to 30 days and can be changed with --priority-window-days. Newly created issues are not evaluated for escalation during that run.

All available comment pages are fetched before counting. Comments from any author count if their creation timestamp is strictly after the window's start; comments exactly at the cutoff are excluded. Comments with missing or unparseable timestamps are skipped. Unparseable timestamps produce at most one warning per issue evaluation, including the timestamp and zero-based comment index.

The default thresholds map the comment count in that window to a target priority:

Window count	Priority
0-1	Undefined
2-9	Minor
10-49	Normal
50-99	Major
100-199	Blocker
200+	Critical

Priority is only raised: for example, an issue with 12 recent comments moves from Minor to Normal, but an issue already at Blocker stays at Blocker. Unrecognized Jira priority names are left unchanged with a warning.

--priority-window-days must be a positive integer. --priority-thresholds must contain exactly five comma-separated, non-negative integers in strictly ascending order. These are minimum comment counts for Minor, Normal, Major, Blocker, and Critical, respectively. For example, 12 comments in the selected window yields Normal with the defaults, but Minor with 10,50,100,200,400. Invalid configuration is rejected at startup, even when automatic escalation is disabled.

With --dry-run, no comment is added and automatic priority evaluation is skipped; the command does not preview the resulting priority.

For example, to count comments from the last 10 days and use thresholds of 10, 50, 100, 200, and 400:

junit2jira \
  --enable-auto-priority \
  --priority-window-days 10 \
  --priority-thresholds "10,50,100,200,400" \
  ...

Authentication

For Jira Cloud authentication, you need to provide:

• JIRA_USER: Your Jira account email address
• JIRA_TOKEN: Your Jira API token (get it from https://id.atlassian.com/manage-profile/security/api-tokens)

Example usage

JIRA_USER="user@example.com" \
JIRA_TOKEN="..." \
junit2jira \
  -jira-url "https://..." \
  -junit-reports-dir "..." \
  -base-link "https://..." \
  -build-id "$BUILD_ID|GITHUB_RUN_ID" \
  -build-link "https://..." \
  -build-tag "$STACKROX_BUILD_TAG|$GITHUB_SHA" \
  -job-name "$JOB_NAME|$GITHUB_WORKFLOW" \
  -orchestrator "$ORCHESTRATOR_FLAVOR" \
  -timestamp $(date --rfc-3339=seconds)
  -csv-output -

flakechecker

flakechecker helps prevent unnecessary CI pipeline failures by suppressing known flaky tests that are within the allowed failure thresholds.

flakechecker relies on several components:

• collected test results from junit2jira: we generate a table of flaky tests, including their failure ratios for the last 30 executions.
• flaky test configuration: we define and provide a flakechecker configuration with allowed failure ratio thresholds for known flaky tests.
• CI pipeline integration script: flakechecker is executed as the last step in a CI pipeline, and provided results allow the CI pipeline script to report success or failure.

The flakechecker expects at least one failed test. It will return an error if it is executed on test results without any failures.

flakechecker decision making:

• it checks if a failed test in a CI pipeline is listed as flaky in the provided configuration.
• if the test is not found in the flaky tests config -> it will cause the CI pipeline to fail. (test not found)
• if the test is found in the configuration, flakechecker will fetch information about the fail ratio for that test from the database. If we have fewer than 30 executions for that test -> it will cause the CI pipeline to fail. (insufficient historical test results)
• if the test's failure ratio in the database exceeds the threshold defined in the config -> it will cause the CI pipeline to fail. (flake ratio is above the allowed threshold)
• if a flaky test's failure ratio is below the defined threshold -> it will report the test as a success in the CI pipeline. (test suppression)

The flakechecker will apply this logic for each failed test in the CI pipeline.

Usage

Usage of flakechecker:
  -config-file string
        Config file with allowed flakes.
  -debug
        Enable debug log level.
  -job-name string
        Name of CI job.
  -junit-reports-dir string
        Directory containing JUnit report XML files.
  -v    short alias for -version
  -version
        print version information and exit

Example usage

flakechecker --config-file flake-config.yml --job-name "${JOB_NAME}" -junit-reports-dir "${ARTIFACT_DIR}"