[BUG] s3upload weirdly switch architecture value during upload sha512 for snapshots

[peterzhuamazon]

In this section, I have seen weird behavior happens over time that x64 is uploading arm64, while arm64 is upload x64.

Uploading file:/var/jenkins/workspace/distribution-build-opensearch/builds/opensearch/dist/opensearch-min-1.3.0-SNAPSHOT-linux-x64-latest.tar.gz.sha512 to s3://artifacts.opensearch.amazon.com/snapshots/core/opensearch/1.3.0-SNAPSHOT/opensearch-min-1.3.0-SNAPSHOT-linux-x64-latest.tar.gz.sha512 
Upload failed due to missing source file

https://github.com/opensearch-project/opensearch-build/blob/main/vars/uploadMinSnapshotsToS3.groovy#L35-L37

withAWS(role: "${ARTIFACT_PROMOTION_ROLE_NAME}", roleAccount: "${AWS_ACCOUNT_ARTIFACT}", duration: 900, roleSessionName: 'jenkins-session') {
        s3Upload(file: "${srcDir}/${baseName}-latest.tar.gz", bucket: "${ARTIFACT_PRODUCTION_BUCKET_NAME}", path: "${dstDir}/${baseName}-latest.tar.gz")
        s3Upload(file: "${srcDir}/${baseName}-latest.tar.gz.sha512", bucket: "${ARTIFACT_PRODUCTION_BUCKET_NAME}", path: "${dstDir}/${baseName}-latest.tar.gz.sha512")
 

This seems like a random error and it does not happen all the time.
But happens twice since yesterday morning already.

Acceptance Criteria

• Fix the library that has causes this issue. Refactoring var scopes in uploadMinSnapshotsToS3 libs #318
• Add unit tests part of the build jenkinsfile that can detect this error.
• Observe the error for upcoming builds.

Thanks.

[bbarani]

@peterzhuamazon Is this still a valid issue?

[jordarlu]

Hi, @peterzhuamazon .. just want to circle back with you if this issue still exists ,or if we can close it for now? As all the jennkins libs moved to opensearch-build-libraries repo and been thru some code changes from the team, maybe the issue is no longer exist? thanks!

[peterzhuamazon]

It is still an issue and it happens transiently.

[jordarlu]

Thank you, @peterzhuamazon , will you suggest adding some gardian code for it? or it will not help at all? maybe we validate the x64 file existinfg before uploading x64? same as arm64? thanks!!

[peterzhuamazon]

New: https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-build-opensearch/detail/distribution-build-opensearch/8497/pipeline/

+ cp -v C:/Users/Administrator/jenkins/workspace/distribution-build-opensearch/zip/builds/opensearch/dist/opensearch-min-1.3.13-SNAPSHOT-windows-x64.tar.gz C:/Users/Administrator/jenkins/workspace/distribution-build-opensearch/zip/builds/opensearch/dist/opensearch-min-1.3.13-SNAPSHOT-windows-x64-latest.tar.gz

cp: cannot stat 'C:/Users/Administrator/jenkins/workspace/distribution-build-opensearch/zip/builds/opensearch/dist/opensearch-min-1.3.13-SNAPSHOT-windows-x64.tar.gz': No such file or directory

script returned exit code 1

[peterzhuamazon]

This seems more like the scope of global var due to us not assigning type.

[peterzhuamazon]

Will issue a potential fix to check on this.

[peterzhuamazon]

Re-open as there is a PR missing on the build side.

[Divyaasm]

@peterzhuamazon , could you provide the latest update on this issue.

[zelinh]

I'm seeing the same issue on our recent snapshot publish. https://build.ci.opensearch.org/blue/organizations/jenkins/publish-opensearch-min-snapshots/detail/publish-opensearch-min-snapshots/129/pipeline/

cp -v /var/jenkins/workspace/publish-opensearch-min-snapshots/tar/builds/opensearch/dist/opensearch-min-2.12.0-SNAPSHOT-linux-x64.zip /var/jenkins/workspace/publish-opensearch-min-snapshots/tar/builds/opensearch/dist/opensearch-min-2.12.0-SNAPSHOT-linux-x64-latest.zip

cp: cannot stat '/var/jenkins/workspace/publish-opensearch-min-snapshots/tar/builds/opensearch/dist/opensearch-min-2.12.0-SNAPSHOT-linux-x64.zip': No such file or directory

script returned exit code 1

This is supposed to be tarball snapshot.

[prudhvigodithi]

Related issue: opensearch-project/opensearch-build#3097