[BUG] OpenSearch Devops Failed to validate on github actions with 3.0.0 version

[peterzhuamazon]

In this action, 3.0.0 always seem to fail the validation here:
https://github.com/opensearch-project/opensearch-devops/actions/runs/10495619352/job/29074434034

The issue is actually within the opensearch-tar-install.sh that calls $OPENSEARCH_HOME/plugins/opensearch-security/tools/install_demo_configuration.sh which located here:
https://github.com/opensearch-project/security/blob/06d8c29eea9a648c463e6063542283222e5b1805/tools/install_demo_configuration.sh

The problem is, when install_demo_configuration.sh calls org.opensearch.security.tools.democonfig.Installer that located here:
https://github.com/opensearch-project/security/blob/06d8c29eea9a648c463e6063542283222e5b1805/src/main/java/org/opensearch/security/tools/democonfig/Installer.java

It has different behavior between 2.16.0 (latest 2.x at the time of writing) and 3.0.0 (latest main), when dealing with jdk 11 vs jdk 17 or above.

/usr/lib/jvm/temurin-11-jdk-amd64/bin/java -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=OFF -cp '/local/home/zhujiaxi/test/opensearch-3.0.0/plugins/opensearch-security/tools/../*:/local/home/zhujiaxi/test/opensearch-3.0.0/plugins/opensearch-security/tools/../../../lib/*:/local/home/zhujiaxi/test/opensearch-3.0.0/plugins/opensearch-security/tools/../deps/*' org.opensearch.security.tools.democonfig.Installer /local/home/zhujiaxi/test/opensearch-3.0.0/plugins/opensearch-security/tools -y -i -s
Error: LinkageError occurred while loading main class org.opensearch.security.tools.democonfig.Installer
	java.lang.UnsupportedClassVersionError: org/opensearch/security/tools/democonfig/Installer has been compiled by a more recent version of the Java Runtime (class file version 65.0), this version of the Java Runtime only recognizes class file versions up to 55.0

Apparently 2.x security is having java compatibility at 11, while main change to use 21.
https://github.com/opensearch-project/security/blob/89ca6a8530bb0892140dbe9cf72f065dcb862b16/build.gradle#L95-L96
https://github.com/opensearch-project/security/blob/8fdc00dfe35078edce760d8019df86615a306eb4/build.gradle#L96-L97

It is because of this commit: opensearch-project/security@439606c

The problem is, even core itself did not update this value to 21, still on 17:
https://github.com/opensearch-project/OpenSearch/blob/b31627986fde9e9a8ed0cd1e94529df965acc5b1/buildSrc/src/main/resources/minimumCompilerVersion

---

Back to workflow:
The reason jdk11 was used here even though that is not defined in the workflow file here:

1. the install_demo_configuration.sh script prioritize OPENSEARCH_JDK_HOME before JAVA_HOME and the bundled jdk OPENSEARCH_HOME/jdk: https://github.com/opensearch-project/security/blob/06d8c29eea9a648c463e6063542283222e5b1805/tools/install_demo_configuration.sh#L34-L40
2. GitHub Actions ubuntu-latest at the time of writing is ubuntu-22.04 and its image include jdk11 as default JAVA_HOME (JAVA_HOME=/usr/lib/jvm/temurin-11-jdk-amd64): https://github.com/actions/runner-images/blob/69f706a1062a11c09911d22f8545a1cae5826bec/images/ubuntu/Ubuntu2204-Readme.md#java

This forced the script to use jdk11 to run org.opensearch.security.tools.democonfig.Installer and cause the failure above.

Thanks.

[peterzhuamazon]

We will unset the JAVA_HOME var in github actions for now in order to fix the issue:

• Unset JAVA_HOME to allow bundled JDK being picked up for security admin script pass #201

[peterzhuamazon]

The earliest failure of this exact issue is here:
https://github.com/opensearch-project/opensearch-devops/actions/runs/9424645727

2024/06/07 5:23 PST.
Run ./validation.sh --file-path opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/9938/linux/x64/tar/dist/opensearch/opensearch-3.0.0-linux-x64.tar.gz opensearch-dashboards=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/7701/linux/x64/tar/dist/opensearch-dashboards/opensearch-dashboards-3.0.0-linux-x64.tar.gz

However, upon manual testing it works fine with jdk11.

[gaiksaya]

cc: @cwperks @peternied @DarshitChanpura

[peterzhuamazon]

We are waiting for the 3.0.0 to have JDK21 set as default:

• [META] Set OpenSearch 3.0.0 baseline JDK version to JDK-21 OpenSearch#14011