KAFKA-19174 POC for a new :distribution submodule

[dejan2609]

Prologue: #19513

This POC tries to explain why a new :distribution submodule is required for Gradle 8 -->> 9 upgrade

The thing is that Gradle 9 is very strict when it comes to a submodule boundaries: https://docs.gradle.org/9.1.0/userguide/viewing_debugging_dependencies.html#sub:resolving-unsafe-configuration-resolution-errors

Root cause (see this commit: 09c8d9f): :core module releaseTarGz task is not able to directly access other submodules runtimeClasspath configuration:

from(project(':trogdor').configurations.runtimeClasspath) { into("libs/") }

And that is why I tried to bypass that with:

subprojects {
...
  def  submodulesToIncludeInDistribution = ['tools', 'trogdor', 'shell', 'api', 'runtime', 'transforms',
                                              'json', 'file', 'basic-auth-extension', 'mirror', 'mirror-client',
                                              'streams', 'streams-scala', 'test-utils', 'examples', 'tools-api']
                                                
  def shouldModuleResolveRuntimeClasspath = (submodulesToIncludeInDistribution.contains(project.name))

    if (shouldModuleResolveRuntimeClasspath) {
        configurations.create('resolvedRuntimeClasspath') {
            extendsFrom(configurations.runtimeClasspath)
            canBeResolved = true
        }
    } 
}
...
project(':core') {
 ...
  tasks.create(name: "releaseTarGz", dependsOn: configurations.archives.artifacts, type: Tar) {
      ...
      from(project(':trogdor').configurations.resolvedRuntimeClasspath) { into("libs/") }
      ...
  }
}

but the build still fails ❌

---

Why build works in #19513 ? Because in addition to the changes above configuration for :distribution is extended: a new submodule :distribution expands Gradle setup like this:

project(':distribution') {
....
  configurations {
    includeIntoDistribution
  }

  dependencies {
    ...
    includeIntoDistribution project(path: ':trogdor', configuration: 'resolvedRuntimeClasspath')
    ...
  }

  tasks.create(name: "releaseTarGz", dependsOn: project(':core').configurations.archives.artifacts, type: Tar) {
    ...
    from(configurations.includeIntoDistribution) { into("libs/") }
    ...
  }

Pro et contra :distribution:

• build command stays the same: ./gradlew releaseTarGz ✔️
• :distribution Gradle submodule has no folder in Git ✔️ (🪧 folder gets created on ./gradlew releaseTarGz but Git ignores it)
• generated distributions (i.e. two .tgz files) paths also stays the same (they will be created in the same/previous :core path) ✔️
• some additional paragraph (no more than two lines) could be added into README.md ℹ️
• build does get slightly complicated ⚠️
• build works ✅

Pro et contra :core:

• build stays the same ✔️
• build fails (at the moment) ❌

Disclaimer: certainly, there is a way to do things differently.
Gradle provides some recommendations (not sure about effort/feasibility):
https://docs.gradle.org/9.1.0/userguide/how_to_share_outputs_between_projects.html

[dejan2609]

FYI @chia7712

[chia7712]

@dejan2609 Is this patch available? I encounter the following error

> Task :core:releaseTarGz FAILED

[Incubating] Problems report is available at: file:///home/chia7712/project/kafka/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':core:releaseTarGz'.
> Resolution of the configuration ':trogdor:resolvedRuntimeClasspath' was attempted without an exclusive lock. This is unsafe and not allowed.

[dejan2609]

@chia7712 Yes, a complete solution is available in KAFKA-19174 branch / #19513 PR; this branch/PR (KAFKA-19174-POC-for-distribution-submodule) is just an attempt to show why :distribution submodule is required.

KAFKA-19174 build works fine:

dejan@dejan-HP-ProBook-450-G7:~/kafka$ date
Thu 25 Sep 10:58:26 CEST 2025
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git status 
On branch KAFKA-19174
Your branch is up-to-date with 'origin/KAFKA-19174'.

nothing to commit, working tree clean
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git log -10 --oneline 
dcfda259ca (HEAD -> KAFKA-19174, origin/KAFKA-19174) KAFKA-19174 slightly changing configuration for `releaseTarGz` task (it should be clearly visible now which submodules configurations had to be extended)
41e2d73856 KAFKA-19174 settling down with `com.gradleup.shadow` version 8.3.9 (gradle task `:jmh-benchmarks:shadowJar` is failing with shadow plugin 9+ versions)
744d7bbcef KAFKA-19174 tidying up things: 'distribution' dummy folder is removed
44950bb6da KAFKA-19174 Gradle version upgrade: 9.0.0 -->> 9.1.0
62649e7f22 KAFKA-19654 do not fail Github Actions flaky/new build matrix build when no test are discovered (i.e. solution for KAFKA-16801 is expanded)
c754acd8ca KAFKA-16801 do not fail Gradle 9 build when no test are discovered in a `upgrade-system-tests*` submodules
04eeb043c1 KAFKA-19174 (note: optional commit) version upgrades for Gradle build dependencies/plugins and GitHub Actions
07cf3e9e5c KAFKA-19174 Gradle version upgrade (8 -->> 9); build logic is refactored
79f04159d9 KAFKA-19591 solving issues with `gradlew` content: an old template version (i.e. file `unixStartScript.txt`) was being referenced
5919762009 (gooler/trunk, gooler/HEAD) MINOR: Remove exitMessage.set() call in TopicBasedRemoteLogMetadataManagerTest (#20563)
dejan@dejan-HP-ProBook-450-G7:~/kafka$ ./gradlew clean releaseTarGz -i
...
> Task :distribution:releaseTarGz
Caching disabled for task ':distribution:releaseTarGz' because:
  Build cache is disabled
  Not worth caching
Task ':distribution:releaseTarGz' is not up-to-date because:
  Output property 'archiveFile' file /home/dejan/kafka/core/build/distributions/kafka_2.13-4.2.0-SNAPSHOT.tgz has been removed.

[Incubating] Problems report is available at: file:///home/dejan/kafka/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/9.1.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 4m 22s
228 actionable tasks: 228 executed
Consider enabling configuration cache to speed up this build: https://docs.gradle.org/9.1.0/userguide/configuration_cache_enabling.html
dejan@dejan-HP-ProBook-450-G7:~/kafka$