Add appProtocol to cluster service definition

[tombojer]

Description

As discussed in issue #958, the appProtocol: "https" must be included to ensure compatibility with service meshes such as Istio.

Regarding the port name change in the service definition, this could be addressed in a future major release. However, if required, I can modify the pull request to reflect this change directly. That said, changing the port name now may break existing ingresses that reference the port by name.

As highlighted in issue #959, it is possible for the service to only listen on HTTP. Hardcoding the appProtocol to https would therefore break compatibility with service meshes in this scenario. I would appreciate some guidance on whether a global approach would be more suitable—one that accounts for services where the cluster might expose either HTTP or HTTPS. This would need to include health checks and any other components that need to determine the protocol being used by the cluster.

For testing, I created a KIND cluster, installed Istio, and applied the CR from the example:

istioctl install --set profile=demo -y
kubectl apply -f examples/2.x/opensearch-cluster.yaml

Additionally, an Istio-specific DestinationRule is required since the OpenSearch cluster uses a self-signed certificate:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: opensearch-destination-rule
spec:
  host: my-first-cluster.default.svc.cluster.local
  trafficPolicy:
    tls:
      mode: SIMPLE
      insecureSkipVerify: true

Now we are able to curl the the OpenSearch cluster.

# Without appProtocol
curl http://my-first-cluster:9200/_stats
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|33554536:system library:OPENSSL_internal:Connection reset by peer:TLS_error_enddebug-pod

# With appProtocol
curl http://my-first-cluster:9200/_stats
Unauthorized

Issues Resolved

Fixes #751
Fixes #958

Check List

• Commits are signed per the DCO using --signoff
• Unittest added for the new/changed functionality and all unit tests are successful
• Customer-visible features documented
• No linter warnings (make lint)

If CRDs are changed:

• CRD YAMLs updated (make manifests) and also copied into the helm chart
• Changes to CRDs documented

Please refer to the PR guidelines before submitting this pull request.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[tombojer]

#769

Is similar but it needs to be set in all services that point to the cluster to make it work.

[prudhvigodithi]

Adding @rootxrishabh to please take a look.

[rootxrishabh]

@tombojer, as I understand, hard coding appProtocol: https would break clusters running with http and service mesh. Assuming this is true, taking from #959. I believe most clusters with service mesh would be running https and so this change should be fine. WDYT?

I think once we fix #959, we can change the port name in a major release.
cc. @prudhvigodithi

[tombojer]

In our clusters, we use a service mesh to handle mTLS between the pods transparently. Running the OpenSearch cluster in HTTP mode would actually provide us with more visibility into the traffic, which can be quite helpful for debugging and observability.

That said, since I'm not sure whether #959 is fixable, this is sufficient to get OpenSearch running, especially if the rest of the community is aligned on using HTTPS.

[prudhvigodithi]

We should explore the setting plugins.security.ssl.http.enabled for OpenSearch http or https settings. Modifying the appProtocol to configure defaults might impact existing users.

@tombojer can you please check if we do this via CRD to define the cluster protocol?

[tombojer]

@prudhvigodithi

Hey,
afaik the plugins.security.ssl.http.enabled needs to be set via https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/opensearch-operator/api/v1/opensearch_types.go#L56 and is therefore not directly exposed via the crd.

[prudhvigodithi]

Hey,
afaik the plugins.security.ssl.http.enabled needs to be set via https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/opensearch-operator/api/v1/opensearch_types.go#L56 and is therefore not directly exposed via the crd.

Got it, looks like this PR #1000 should solve that and pass the security changes via additionalConfig.
CC @cwperks @rootxrishabh WDYT ?

[tombojer]

If the protocol can be configured via additionalConfig, is there a way for the operator to parse that and adjust service settings (like ports, probes, etc.) accordingly? Or should all relevant settings be explicitly exposed via the CRD instead?

For example, this issue would still persist unless we can either disable TLS in the CRD or update the health check behavior.