Skip to content

Commit

Permalink
docs: align readme with main repo
Browse files Browse the repository at this point in the history
  • Loading branch information
willgeorgetaylor committed Dec 30, 2023
1 parent 5593b18 commit 7308927
Showing 1 changed file with 9 additions and 7 deletions.
16 changes: 9 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[![CodeQL](https://github.com/willgeorgetaylor/junit-reducer-action/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/willgeorgetaylor/junit-reducer-action/actions/workflows/codeql-analysis.yml)
[![Coverage](./badges/coverage.svg)](./badges/coverage.svg)

JUnit Reducer is a command-line tool that aggregates the [JUnit test XML reports](https://www.ibm.com/docs/en/developer-for-zos/14.1?topic=formats-junit-xml-format) from your CI runs and creates a single, averaged report set. This lets you download a smaller set of reports, during CI runs, to inform your test splitting.
JUnit Reducer Action is a GitHub Action wrapping a command-line tool that aggregates the [JUnit test XML reports](https://www.ibm.com/docs/en/developer-for-zos/14.1?topic=formats-junit-xml-format) from your CI runs and reduces them to a single, lighter set of reports to be downloaded later during CI, to steer your test splitting algorithm (e.g., [split_tests](https://github.com/marketplace/actions/split-tests)). The most typical use case is to regularly update a 'running average' of your recent test reports, which can be downloaded to your test runners in [less time](#faster-ci-) and without running an [ongoing race condition risk](#coverage-integrity-).

<picture>
<source media="(prefers-color-scheme: dark)" srcset="./diagram-dark.png">
Expand All @@ -17,7 +17,7 @@ JUnit Reducer is a command-line tool that aggregates the [JUnit test XML reports

## Quickstart

Typically, you'll be using `junit-reducer` within a scheduled cron task to reduce a trailing window of JUnit XML reports. From a speed and cost perspective, it's generally a good idea to retrieve and store both the inputs (JUnit XML reports) and outputs (averaged XML reports) in a cloud storage service like AWS S3 or Google Cloud Storage as opposed to the caching APIs available from the CI providers themselves.
Typically, you'll be using this Action within a scheduled [cron](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule) workflow to take the last `X` days of JUnit XML reports, reduce them and upload the results. It's recommended to accumulate the JUnit XML reports from individual CI runs in a cloud storage service like AWS S3 or Google Cloud Storage, as opposed to the caching API available within GitHub Actions, which is designed as an _overwrite_ key-value store.

## Inputs

Expand Down Expand Up @@ -95,17 +95,19 @@ jobs:
As your test suite grows, you may want to start splitting tests between multiple test runners, to be **executed concurrently.** While it's relatively simple to divide up your test suites by files, using lines of code (LOC) as a proxy for test duration, the LOC metric is still just an approximation and will result in uneven individual (and therefore overall slower) test run times as your codebase and test suites change.
### Faster CI
The preferable approach for splitting test suites accurately is to use **recently reported test times,** and the most popular format for exchanging test data (including timings) between tools is the [JUnit XML reports format](https://www.ibm.com/docs/en/developer-for-zos/14.1?topic=formats-junit-xml-format). While JUnit itself is a Java project, the schema that defines JUnit reports is equally applicable to any language and reports can be generated by most testing frameworks for JavaScript, Ruby, Python etc.
In busier projects, CI will be uploading reports frequently, so even if you take a small time window (for example, the last 24 hours), you could end up with 20MB+ of test reports. These reports need to be **downloaded to every runner in your concurrency set,** only to then perform the same splitting operation to **yield the exact same time estimates.** This means unnecessary and expensive work is being performed by each concurrent runner, potentially delaying the total test time by minutes and increasing CI costs.
### Coverage integrity
### Faster CI ✅
You can solve this speed issue by creating a set of reports that looks like the set produced by a single CI run. Importantly, the values for `time` taken by test suite (as well as other counts, like errors and tests) are reduced from the wider set of reports, typically by finding the `mean` of all of the aggregate `time` values. Other reducer operations, like `min` / `max` / `mode` / `median` / `sum`, are available to handle non-standard distributions.

### Coverage integrity ✅

In very busy projects, there is also a more **problematic race condition possible**, with larger downloads and test runners starting at different times. As CI runs from other commits upload their reports to the same remote source that you're downloading them from, if any of your concurrent runners download reports with different values, the input data is misaligned and the splitting operation is corrupted. However, because the download and splitting operation is being performed in a distributed manner (across all of the runners concurrently) this misalignment is not discoverable and it is likely that some tests in your run will be **skipped without you knowing it happened.**
In very busy projects, there is also a more **problematic race condition possible**, with larger downloads and test runners starting at different times. As CI runs from other commits upload their reports to the same remote source that you're downloading from, if any of your concurrent runners download reports with different values, the input data is misaligned and the splitting operation is corrupted. However, because the download and splitting operation is being performed in a distributed manner (across all of the runners concurrently) this misalignment will result in some tests in your run being **skipped.**

This risk can be mitigated by computing the averaged reports in one place, and updating that set as part of a scheduled job. This is exactly the approach outlined in the [quickstart](https://github.com/willgeorgetaylor/junit-reducer?tab=readme-ov-file#quickstart) section.
This risk is mitigated by computing the averaged reports in one place, and updating that set as part of a scheduled job. This is exactly the approach outlined in the [example workflow](#example-workflow) section.

## Dependencies

Expand Down

0 comments on commit 7308927

Please sign in to comment.