Improving Spring Boot Test efficiency

Problem statement

Spring test framework creates an application context according to test class configuration. The context is cached and reused for all subsequent tests. If there is an existing context with the same configuration, it will be reused. Otherwise, the new context will be created. This is a very efficient and flexible approach, but it has a drawback: eventually this may lead to out of memory errors if the number of unique configurations is too high and context has a lot of heavyweight beans like TestContainers. In many cases simple static bean definition can help, but this project suggests another approach: reordering test classes and eager context cleanup.

Consider a sample test suite of 8 classes that use 4 different configurations, classes that have the same configuration are marked with the same colour:

In a large test suites as well as in shared CI/CD environments with lots of test pipelines working simultaneously this may eventually lead to out of memory errors in Java process or Docker host.

Proposed solution

It's recommended to use statically-defined TestContainers beans, optimize reusing same configuration between tests e.g. via common test super-classes. But additionally this library makes two optimizations:

• test class execution is reordered to group tests with the same context configuration so they can be executed sequentially
• the order of tests is known, so if current test class is last per current configuration, the spring context will be automatically closed (it's called Smart DirtiesContext) and the beans will be disposed releasing resources

As a result, in a suite of single module there will always be not more than 1 active spring contexts:

This chart is done via calculating the number of active docker containers while executing a suite of 120 integration test classes that actively uses TestContainers for databases (several datasources simultaneously) and other services:

As shown on the chart, the suite just fails with OOM without the optimization. As an advantage, the total test execution time will also become less, because resource consumption (especially memory) will be reduced, hence tests are executed faster.

References

This idea was submitted to the Spring Framework team as a feature request:

• spring-projects/spring-framework#32289

Public presentation with AtomicJar (TestContainers creators):

• https://www.youtube.com/watch?v=_Vci_5nr8R0

Limitations

At the moment only single thread test execution per module is supported. Parallel test execution is work in progress. Also there can be problems with Jupiter Nested test classes if they declare own @ContextConfiguration or @Import of spring beans.

Supported versions

Java 8+ (Java 17+ for spring-boot 3.x projects)

Spring Boot 2.4+, 3.x as well as bare Spring framework

Supported test frameworks:

• JUnit 4 (via JUnit 5 junit-vintage-engine)
• JUnit 5 Jupiter
• TestNG (both bare TestNG and JUnit platform testng-engine)

Develocity Maven Extension (test execution caching) correctly supports changed behaviour

How to use

Add maven dependency (available in maven central):

<dependency>
    <groupId>com.github.seregamorph</groupId>
    <artifactId>spring-test-smart-context</artifactId>
    <version>0.8</version>
    <scope>test</scope>
</dependency>

Or Gradle dependency:

testImplementation("com.github.seregamorph:spring-test-smart-context:0.8")

Also it's recommended to configure "INFO" level for com.github.seregamorph.testsmartcontext logger. It's recommended to check Demo projects.

How it works

JUnit 5 Jupiter

Add the dependency to the library in test scope, it will automatically setup SmartDirtiesClassOrderer which will reorder test classes on each execution and prepare the list of last test class per context configuration. Then this test execution listener SmartDirtiesContextTestExecutionListener will be auto-discovered via spring.factories. Alternatively it can be defined explicitly

@TestExecutionListeners(SmartDirtiesContextTestExecutionListener.class)

or even inherited from AbstractJUnitSpringIntegrationTest

TestNG

Add the dependency to the library in test scope, it will automatically setup SmartDirtiesSuiteListener which will reorder test classes on each execution and prepare the list of last test class per context configuration. The integration test classes should add SmartDirtiesContextTestExecutionListener

@TestExecutionListeners(SmartDirtiesContextTestExecutionListener.class)

Note: the annotation is inherited, so it makes sense to annotate the base test class or use AbstractTestNGSpringIntegrationTest parent.

JUnit 4

Note: support of JUnit 4 is planned to be removed in version 1.0 (will stay available in 0.x versions).

The JUnit 4 does not provide standard way to reorder test class execution, but it's still possible via junit-vintage-engine. This dependency should be added to test scope of the module:

<dependency>
    <groupId>org.junit.vintage</groupId>
    <artifactId>junit-vintage-engine</artifactId>
    <scope>test</scope>
</dependency>

or for Gradle (see detailed instruction):

testRuntimeOnly('org.junit.vintage:junit-vintage-engine')
testRuntimeOnly('org.junit.platform:junit-platform-launcher')

Also the surefire/failsafe plugins should be configured to use junit-platform:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire.version}</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit-platform</artifactId>
            <version>${maven-surefire.version}</version>
        </dependency>
    </dependencies>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>${maven-surefire.version}</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit-platform</artifactId>
            <version>${maven-surefire.version}</version>
        </dependency>
    </dependencies>
</plugin>

or for Gradle:

tasks.named('test', Test) {
    useJUnitPlatform()
}

For projects with JUnit 4 it will automatically setup SmartDirtiesPostDiscoveryFilter which will reorder test classes on the level of junit-launcher and prepare the list of last test class per context configuration. Then this test execution listener SmartDirtiesContextTestExecutionListener will be auto-discovered via spring.factories. Alternatively it can be defined explicitly

@TestExecutionListeners(SmartDirtiesContextTestExecutionListener.class)

or even inherited from AbstractJUnit4SpringIntegrationTest

Additional materials

• See the online presentation of the project https://www.youtube.com/watch?v=_Vci_5nr8R0 hosted by AtomicJar, the creators of TestContainers framework.
• Presentation slides: Miro board

Known projects using library

Miro is using this approach to optimize huge integration test suites and it saved a lot of resource for CI/CD pipelines.