[BUG] Plugin Build Fails Due to missing nebula Dependencies.

[cliu123]

Describe the bug
https://github.com/opensearch-project/security/actions/runs/3415191485/jobs/5684117326

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'opensearch-security'.
> Could not resolve all files for configuration ':classpath'.
   > Could not resolve com.netflix.nebula:nebula-core:3.0.0.
     Required by:
         project : > org.opensearch.gradle:build-tools:3.0.0-SNAPSHOT:20221103.083150-511 > com.netflix.nebula:nebula-publishing-plugin:4.4.4
      > Could not resolve com.netflix.nebula:nebula-core:3.0.0.
         > Could not get resource 'https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/com/netflix/nebula/nebula-core/3.0.0/nebula-core-3.0.0.pom'.
            > Could not GET 'https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/com/netflix/nebula/nebula-core/3.0.0/nebula-core-3.0.0.pom'. Received status code 403 from server: Forbidden

[cliu123]

I discussed this with @mch2 offline. Would you please provide more context on this?

[ylwu-amzn]

We see same issue in ml-commons Github CI today which passed last Friday.

[cliu123]

I'm tagging this for 2.4.0 since it's blocking the last PR of security plugin for 2.4.0.
@CEHENKLE @bbarani @peterzhuamazon

[peterzhuamazon]

Break all the build repo manifest checks:
https://github.com/opensearch-project/security/actions/runs/3415205437/jobs/5684070899

[MaxKsyunz]

Just seeing this now with SQL plugin.

[dblock]

The error in the original issue says org.opensearch.gradle:build-tools:3.0.0-SNAPSHOT which is 3.0, where are we seeing this for 2.4?

[cliu123]

@dblock PRs need to be merged to main(3.0.0) and then backported to 2.4.

[dblock]

@cliu123 Just make the PR on 2.x and backport it the other direction.

[mch2]

Looks like version 4.4.4 of the publish plugin depends on nebula-core 3.0.0 which is not in maven central. These are all very old versions of these plugins (latest is v19) but I think we can update to 4.6.0 or 4.10.0 and resolve the issue.

[dblock]

@mch2 The above seems to imply that it worked before. Was it removed from maven? Weird. But yes upgrading to a newer 4.x should be harmless. Are you on it?

[mch2]

@dblock Yeah it must have been removed but I'm not sure how to check that. Will PR to a 4.x version shortly.

[dblock]

It just failed in 2.4 here

[peterzhuamazon]

Issue was found as old as April this year:
#3097 (comment)

[peterzhuamazon]

Related dependabot:
#4810

[MaxKsyunz]

What do you think about updating nebula-publishing-plugin to 4.9.1? It no longer depends on nebula-core and the last available version in 4.x branch.

4.6.0. works because it specifies latest.release as required nebula-core version. They moved on from that in 4.7.0 -- 20 days later.

nebula-core itself is archived on GitHub. The only version I can find on Maven Central is 5.0.0 which was not released from the GitHub project.

[dblock]

@MaxKsyunz That's a good path. We'll work through this today. I tried this quickly and ran into more problems. If you want to attempt this in parallel, that would be greatly appreciated.

Do you have any idea why 4.4.0 stopped working?

[peternied]

The change hasn't made it into 2.4.0 snapshot builds, reopening this issue until builds in all active branches are fully unblocked

[dblock]

#5132 includes the nebula fix for 2.4

[peternied]

The last build for 2.4.0 is more than 3 days ago [1] so #5127 was not included.

@opensearch-project/engineering-effectiveness does another build need to be triggered manually?

[1] https://build.ci.opensearch.org/job/distribution-build-opensearch/6425/

[dblock]

I see 2.4.0 auto build disabled in opensearch-project/opensearch-build#2852. Already raised a while ago that we shouldn't be disabling that, so this is another data point. Who knows how to kick it? cc: @bbarani @peterzhuamazon

[prudhvigodithi]

Hey @dblock I see it was disabled because the Release candidate for 2.4.0 is locked.

[dblock]

@prudhvigodithi what bad thing would happen if we didn't?

[prudhvigodithi]

I see the RC is fixed to specific build numbers opensearch:2.4.0.6425, opensearch-dashboards:2.4.0.4275, we can still continue to run the auto builds, but only the downloads with /latest URL could break
Example https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-job-scheduler-2.4.0.0.zip
@peterzhuamazon @bbarani

[dblock]

Latest does not imply "release candidate", and I don't think it should.

[peterzhuamazon]

I see 2.4.0 auto build disabled in opensearch-project/opensearch-build#2852. Already raised a while ago that we shouldn't be disabling that, so this is another data point. Who knows how to kick it? cc: @bbarani @peterzhuamazon

We have to disable the build at this point for the release but we can introduce -rc input manifests later.

Thanks.

[MaxKsyunz]

@MaxKsyunz That's a good path. We'll work through this today. I tried this quickly and ran into more problems. If you want to attempt this in parallel, that would be greatly appreciated.

Yup. I'll take a look.

Do you have any idea why 4.4.0 stopped working?

No, I couldn't figure out why nebula-core 3.0.0 disappeared.

[gaiksaya]

I see 2.4.0 auto build disabled in opensearch-project/opensearch-build#2852. Already raised a while ago that we shouldn't be disabling that, so this is another data point. Who knows how to kick it? cc: @bbarani @peterzhuamazon

We have to disable the build at this point for the release but we can introduce -rc input manifests later.

Thanks.

@peterzhuamazon if we have release candidate ready, docker has the specific tag and urls have the buildNumber in their path, it should be okay to keep building right? Basically, whatever we will be shipping to customer or final product is locked now so whatever else it overrides shouldn't be a concern.

[peterzhuamazon]

We have published the latest snapshot.