Generalize Backwards Compatibility tests so we can test from any version to any version

[peternied]

Description

With an issue reported indicating that there are serialization issue between 1.3 and 2.X, making sure that we can reproduce the errors. This new workflow(s) will make sure that we aren't breaking BWC with changes we are adding to 2.X and will give us the flexibility to test certain migration workflows.

Fixing an issue where the BWC tests were not actually building or executing causing the clusters to spin up and then immediately spin down. We will need to invest more energy into running multiple kinds of security plugin specific scenarios through the test system.

Issues Resolved

• Resolves [BUG] Backward compatibility tests do not execute test cases #2250
• Resolves Update BWC tests to operate on Windows #2232
• Resolves Add Windows Support #1474
• Related [BUG] Errors/Broken operations during rolling upgrade of clusters from 1.3 to 2.0 #2228

Check List

• New functionality includes testing
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[peternied]

The build is totally broken because a JDK18 features is being used - waiting on the following issue to be merged to resolve this
opensearch-project/OpenSearch#5205

[codecov-commenter]

Codecov Report

Merging #2253 (7d7b262) into main (93fe633) will increase coverage by 0.09%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #2253      +/-   ##
============================================
+ Coverage     61.02%   61.12%   +0.09%     
- Complexity     3267     3271       +4     
============================================
  Files           259      259              
  Lines         18337    18337              
  Branches       3248     3248              
============================================
+ Hits          11191    11209      +18     
+ Misses         5561     5541      -20     
- Partials       1585     1587       +2     

Impacted Files	Coverage Δ
...iance/ComplianceIndexingOperationListenerImpl.java	63.23% <0.00%> (+1.47%)	⬆️
...ic/auth/http/jwt/AbstractHTTPJwtAuthenticator.java	59.30% <0.00%> (+3.48%)	⬆️
.../auth/http/jwt/keybyoidc/SelfRefreshingKeySet.java	68.30% <0.00%> (+8.45%)	⬆️
...t/keybyoidc/AuthenticatorUnavailableException.java	20.00% <0.00%> (+20.00%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[peternied]

Note; CI is blocked until the following issue(s) are resolved:

• Snapshot for OpenSearch 3.0.0 not getting updated on aws.oss.sonatype opensearch-build#2886

[cwperks]

Thank you @peternied! I took a first pass and really like the flexibility this gives us in BWC testing. Left a few thoughts. There's a few descriptions that look incomplete which should be addressed, but otherwise this looks good to me.

[DarshitChanpura]

Much cleaner and compartmentalized!!

[peternied]

@cwperks @DarshitChanpura @scrawfor99 CI has finally been sorted out since we've got a 2.5.0 build, please take another look.

[DarshitChanpura]

Ty @peternied for resolving this pain!

[cwperks]

@peternied Quick q: Is it safe to ignore these Windows errors? Failed to create working dir using hard links. Falling back to copy org.opensearch.gradle.testclusters.OpenSearchNode$LinkCreationException: Failed to create hard link D:\a\security\security\bwc-test\build\testclusters\securityBwcCluster0-2\distro\3.0.0-ARCHIVE\bin\opensearch pointing to C:\Users\runneradmin\.gradle\caches\transforms-3\c38e0a71e65414d6057b5385bc6304be\transformed\opensearch-3.0.0-windows-x64.zip\opensearch-3.0.0\bin\opensearch

[peternied]

@cwperks Interesting, since they are falling back seems safe to me. Took a look at the related code, seems like maybe its slowing the process down a little. I think we could dig in more in the future if we were looking for a speed boost

https://github.com/opensearch-project/OpenSearch/blob/a0d30739402ab361d2a4c0fc6970ffbd2fe07039/buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java#L1048

[stephen-crawford]

This looks really good to me!

[opensearch-trigger-bot]

The backport to 1.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.3 1.3
# Navigate to the new working tree
cd .worktrees/backport-1.3
# Create a new branch
git switch --create backport/backport-2253-to-1.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 dc7d21c716250001926af9ce5d5937d221122ff9
# Push it to GitHub
git push --set-upstream origin backport/backport-2253-to-1.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.3

Then, create a pull request where the base branch is 1.3 and the compare/head branch is backport/backport-2253-to-1.3.