compression level difference on recent releases

[hboutemy]

Hi,
JPMML releases have been proven reproducible for some time now.
But lately, apparently related to switch from JDK 8 to 11 for releases, I'm having hard time to get exact same binaries:
see https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/content/org/jpmml/jpmml-model/README.md
(I have the same behaviour with other JPMML projects recent releases)

it seems there is "just" a compression difference:
https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/content/org/jpmml/jpmml-model/jpmml-model-1.7.1.diffoscope

Can you help me discover what could be the environment difference that leads to this?
My rebuild environment says:

Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /usr/local/apache-maven
Java version: 11.0.25, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-amd64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.8.0-1020-azure", arch: "amd64", family: "unix"

Can you share your release one?
Any idea?

[vruusmann]

But lately, apparently related to switch from JDK 8 to 11 for releases, I'm having hard time to get exact same binaries.

Thanks for doing the debugging work for me!

The Java requirement for JPMML-Model and JPMML-Evaluator projects was changed from JDK 8 to JDK 11 during the ongoing upgrade from 1.6-SNAPSHOT to 1.7-SNAPSHOT development branches. For example, JPMML-Model 1.6.X was JDK 8, but JPMML-Model 1.7.X is JDK 11 now.

I did not change anything about the Apache Maven configuration (apart from adding an extra configuration item to maven-enforcer-plugin). The builds continue to be fine locally (as in mvn clean install).

But I did notice that something is off with the builds when trying to build some downstream JPMML projects such as SkLearn2PMML or R2PMML using JDK 11 now.

For example, the R2PMML project builds fine with JDK 8:

cd r2pmml
export JAVA_HOME=/opt/jdk1.8.0_421/
mvn clean install

But when I switch to JDK 11, ...

export JAVA_HOME=/opt/jdk-11.0.24/
mvn clean install

... the build fails with a "Failed to create classpath: invalid entry compressed size (expected 795 but got 802 bytes)":

[INFO] --- packager:1.0.0:create-classpath (default) @ r2pmml ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.716 s
[INFO] Finished at: 2025-01-28T08:58:18+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jpmml:packager-maven-plugin:1.0.0:create-classpath (default) on project r2pmml: Failed to create classpath: invalid entry compressed size (expected 795 but got 802 bytes) -> [Help 1]

This error is raised by my Packager-Maven-Plugin custom plugin, probably somewhere in its copyArtifacts method:
https://github.com/jpmml/packager-maven-plugin/blob/1.0.0/src/main/java/org/jpmml/maven/plugins/CreateClasspathMojo.java#L185-L221

Can you help me discover what could be the environment difference that leads to this?

It seems like a JDK issue, not a Apache Maven issue. Meaning that JDK 11(.0.24) compresses files differently than JDK 1.8(.0-421) by default?

The bug is triggered when switching from JDK 8 to 11, and disappears when moving in the opposite direction.

Anyway, here is my Apache Maven configuration:

For JDK 1.8:

Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
Maven home: /opt/apache-maven-3.9.9
Java version: 1.8.0_421, vendor: Oracle Corporation, runtime: /opt/jdk1.8.0_421/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.12.9-200.fc41.x86_64", arch: "amd64", family: "unix"

For JDK 11:

Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
Maven home: /opt/apache-maven-3.9.9
Java version: 11.0.24, vendor: Oracle Corporation, runtime: /opt/jdk-11.0.24
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.12.9-200.fc41.x86_64", arch: "amd64", family: "unix"

[vruusmann]

Any idea?

Two other ideas come to mind:

1. Perhaps I need to explicitly set the "compression level" property with Apache Maven plugins such as the maven-jar-plugin?

2. Perhaps it's somehow related to Java Platform Module System (JPMS)? I'm halfway done with the JDK 1.8 to JDK 11 migration, meaning that the Java code is OK, but I haven't defined the JPMS descriptor module-info.java yet. Could a missing module-info.class file interfere with the maven-jar-plugin in some way? Perhaps it is creating an empty one (and stuffing it into the JDK 11 JAR)?

[hboutemy]

thanks for the feedback: I'll need time to test with your JDK, and also check other suprising issues you're having with JDK 11

Perhaps I need to explicitly set the "compression level" property with Apache Maven plugins

this is something that was never needed for any project I rebuilt: let's investigate deeper before going that route

Perhaps it's somehow related to Java Platform Module System (JPMS)?

diffoscope would report on missing file: it does not. It's reporting on zipdetails differences, that clearly show compression-only difference on META-INF/MANIFEST.MF file = the second entry in .jar (after META-INF/)

my best bet is to try with Oracle distribution: that's the only visible difference

[hboutemy]

@vruusmann I tested some 11.0.26 JDKs (not Oracle one, as it was not available in sdkman...) => same compression issue

the other visible difference in your setup is the Linux kernel: 6.12.9 for you vs 6.8.0 for me

may I ask what Linux distribution you're using?
I'm personally using Ubuntu 24.04

I'm searching difference in environment, as Maven itself is known as deterministic, I can't imagine difference from Maven plugin selection

[vruusmann]

I tested some 11.0.26 JDKs => same compression issue

OK, good to know that other JDK 11s are also affected.

I was almost ready to switch to a non-Orcale JDK for doing Maven Central Repository releases.

may I ask what Linux distribution you're using?

Here's my cropped hostnamectl output:

Operating System: Fedora Linux 41 (Workstation Edition)
CPE OS Name: cpe:/o:fedoraproject:fedora:41
Kernel: Linux 6.12.9-200.fc41.x86_64
Architecture: x86-64

I'm searching difference in environment, as Maven itself is known as deterministic, I can't imagine difference from Maven plugin selection

You said earlier that the only differently compressed file is META-INF/MANIFEST.MF?

Perhaps JDK 11 has some code (triggered in maven-jar-plugin) that causes this specific Jar entry to be re-compressed with a different compression level.

[hboutemy]

I tried using a Fedora 41 VM

Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
Maven home: /home/herve/.sdkman/candidates/maven/current
Java version: 11.0.26, vendor: Eclipse Adoptium, runtime: /home/herve/.sdkman/candidates/java/11.0.26-tem
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.11.4-301.fc41.x86_64", arch: "amd64", family: "unix"

but I don't get the same compression as you
and I don't have the same kernel: how did you upgrade from 6.11 to 6.12?

[vruusmann]

but I don't get the same compression as you

Do you think this could be an hardware issue then? Perhaps my CPU can use some extra instructions (during compression), which aren't available on verification env's CPU?

My lscpu output:

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   22
  On-line CPU(s) list:    0-21
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Core(TM) Ultra 9 185H
    CPU family:           6
    Model:                170
    Thread(s) per core:   2
    Core(s) per socket:   16
    Socket(s):            1
    Stepping:             4
    CPU(s) scaling MHz:   22%
    CPU max MHz:          5100.0000
    CPU min MHz:          400.0000
    BogoMIPS:             6144.00
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
                          pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe sysc
                          all nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bt
                          s rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_kno
                          wn_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ss
                          se3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe po
                          pcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 
                          3dnowprefetch cpuid_fault epb intel_ppin ssbd ibrs ibpb stibp 
                          ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase
                           tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap c
                          lflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves 
                          split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts 
                          hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umi
                          p pku ospke waitpkg gfni vaes vpclmulqdq rdpid bus_lock_detect
                           movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ib
                          t flush_l1d arch_capabilities
Virtualization features:  
  Virtualization:         VT-x
Caches (sum of all):      
  L1d:                    544 KiB (14 instances)
  L1i:                    896 KiB (14 instances)
  L2:                     18 MiB (9 instances)
  L3:                     24 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-21
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer saniti
                          zation
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB f
                          illing; PBRSB-eIBRS Not affected; BHI BHI_DIS_S
  Srbds:                  Not affected
  Tsx async abort:        Not affected

and I don't have the same kernel: how did you upgrade from 6.11 to 6.12?

The 6.12 is stable, and 6.11 is unstable. So, the question should be that why doesn't your system update to latest stable.

I got my setup by installing from FC 41 workstation ISO image, and have then kept it up to date by doing dnf update. That is, should be the most common FC 41 setup there is.

[hboutemy]

ok, I updated my system (did not update after initial install), now I have OS name: "linux", version: "6.12.11-200.fc41.x86_64", arch: "amd64", family: "unix" like you
does not solve the compression difference, sadly (I'm checking on -sources.jar, so no compiler complexity involved, pure basic https://maven.apache.org/plugins/maven-source-plugin/jar-mojo.html )

my lscpu:

lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   8
  On-line CPU(s) list:    0-7
Vendor ID:                GenuineIntel
  Model name:             12th Gen Intel(R) Core(TM) i9-12900K
    CPU family:           6
    Model:                151
    Thread(s) per core:   1
    Core(s) per socket:   8
    Socket(s):            1
    Stepping:             2
    BogoMIPS:             6374.39
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good no
                          pl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm a
                          bm 3dnowprefetch pti fsgsbase bmi1 avx2 bmi2 invpcid rdseed clflushopt arat md_clear flush_l1d arch_capabilities
Virtualization features:  
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):      
  L1d:                    384 KiB (8 instances)
  L1i:                    256 KiB (8 instances)
  L2:                     10 MiB (8 instances)
  L3:                     240 MiB (8 instances)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-7
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          KVM: Mitigation: VMX unsupported
  L1tf:                   Mitigation; PTE Inversion
  Mds:                    Mitigation; Clear CPU buffers; SMT Host state unknown
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Not affected
  Reg file data sampling: Vulnerable: No microcode
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Vulnerable
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline
  Srbds:                  Not affected
  Tsx async abort:        Not affected

I'm worried, all your recent jpmml releases have the same issue https://github.com/jvm-repo-rebuild/reproducible-central/pulls , it's creating quite a lot of noise

@algomaster99 any idea?

[vruusmann]

To rehash - the same Hardware/OS/Apache Maven compress "reproducibly" on JDK 8, but "irreproducibly" on JDK 11. The issue affects all distributable JAR files (source, binary, javadoc). It does not appear to affect non-JAR files (such as POM files).

[vruusmann]

@hboutemy The jar:jar goal of maven-jar-plugin does not seem to have a config option for customizing the compression level.

What's my best option to enforce "use maximum compression level (for all build artifacts)"?

[hboutemy]

"reproducibly" on JDK 8, but "irreproducibly" on JDK 11

in fact, your environment is the only one I know of that has different output from any other I know of (my Linux laptop x86_64, my Mac laptop aarm64, misc VMs I built, GitHub Actions runners, ...)

I don't know what makes your machine different from everything else, but the fact is that it's "reproducible" in the sense of stable: you personally have stable results, as everybody else also has stable results
the only question is: what makes your env different from every other env?

should that be sorted out by configuring a plugin: as a workaround, why not, every option is on the table :)

[vruusmann]

@hboutemy I'm not familiar with diffoscope, and I'm too lazy to begin mayor investigations on my own. What's my best option to run diffoscope locally, as part of Apache Maven build? Is there a Maven diffoscope plugin that would detect any irrelgularities, and fail my build fast (in order to make me aware of the problem)?

[hboutemy]

diffoscope is about trying to understand a binary difference = diff for archives: no reason to run it from Maven

and honestly, in the current case where issues are only on zip compression, instead of using the magic of diffoscope, I suggest working by hand when you have 2 jars that are different:
use zipdetails on the 2 jars and use basic diff: no magic, just simple clear tools

how to find what makes your environment different to others, I don't know
Did you try to rebuild from Docker? To see if building from Docker (or podman) gives you same result as building directly on your native OS?

[hboutemy]

you: Thread(s) per core: 2
me: Thread(s) per core: 1

could it be the difference that influences compression results?

[hboutemy]

do you have access to multiple machines to easily compare results yourself against the machine that yields surprising output?