Reproducible builds for plugins (and clients?)

[MaxKsyunz]

Is your feature request related to a problem? Please describe

Plugin builds depend on libraries from OpenSearch core. They also use a lot of the same packages. jackson-databind is a good example.

Whenever core libraries bump the version of any such package, plugin builds start to fail until they bump the version as well.

It's disruptive whenever this happens.

Describe the solution you'd like

Plugin builds to be reproducible.

Describe alternatives you've considered

No response

Additional context

No response

[gaiksaya]

[Triage] Hi @MaxKsyunz ,

Can you please elaborate more on what do you mean by "reproducible"? Also what is expected to happen more precisely?
cc: @dblock

[MaxKsyunz]

Reproducible builds mean that two builds of the same source will generate the same binary. There's a site with more info. The main goal is greater security but it also improves developer experience as well.

Here's a simplified of example how this affects development.

opensearch-sql plugin uses jackson-core and opensearch-rest-high-level-client which also depends on jackson-core:

graph LR;
  opensearch-sql ---> jackson-core;
  opensearch-sql ---> opensearch-rest-high-level-client;
  opensearch-rest-high-level-client ---> jackson-core;

Loading

Due to CVEs, jackson-core version gets bumped in opensearch-rest-high-level-client. This new version, with same version number, is published on Maven. As soon as that happens opensearch-sql CI actions start to fail because of the version clash of jackson-core`:

Now someone needs to update jackson-core version in opensearch-sql 2.x branch and everyone else working on a branch of opensearch-sql needs to do that as well.

As soon as this PR is merged, we'll run into this as well.

[MaxKsyunz]

Here's another issue in the same vane.
The error I'm seeing is:

> Could not find com.netflix.nebula:nebula-core:3.0.0.
     Searched in the following locations:
       - file:/home/runner/.m2/repository/com/netflix/nebula/nebula-core/3.0.0/nebula-core-3.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/com/netflix/nebula/nebula-core/3.0.0/nebula-core-3.0.0.pom
       - https://repo.maven.apache.org/maven2/com/netflix/nebula/nebula-core/3.0.0/nebula-core-3.0.0.pom
       - https://plugins.gradle.org/m2/com/netflix/nebula/nebula-core/3.0.0/nebula-core-3.0.0.pom
     Required by:
         project : > org.opensearch.gradle:build-tools:2.4.0-SNAPSHOT:20221105.061[41](https://github.com/Bit-Quill/opensearch-project-sql/actions/runs/3415472960/jobs/5684638758#step:4:42)6-1[46](https://github.com/Bit-Quill/opensearch-project-sql/actions/runs/3415472960/jobs/5684638758#step:4:47) > com.netflix.nebula:nebula-publishing-plugin:4.4.4
         ```

[gaiksaya]

Hi @MaxKsyunz ,

Is this issue still valid?

[peterzhuamazon]

Seems like resolved right here now:

• Apply reproducible builds to plugins OpenSearch#4746

Thanks.