AutoOps ECK integration

[naemono]

Resolves #8789

What is this change?

This adds a new CRD AutoOpsAgentPolicy that allows Elastic AutoOps to be integrated into self-managed ECK clusters.

TODO

• Allow parts of the configuration (configmap) to be overridden
• Allow cleanup of orphaned Agents and their relevant data.

Implementation Notes

• For each ES cluster, the CA is copied to the namespace of the AutoOps Policy, and an API Key is created in the ES cluster for communication purposes and an additional secret is created that contains the API Key.
• Currently if the policy is in the same namespace as ECK operator the query for ES clusters is cluster-scoped, and if it's outside of the operator namespace, it's namespace scoped. This follows what we did for SSP, but recent discussions are questioning this behavior. (This behavior could quickly change and default to cluster-scoped always, which seems to make sense)

Needs testing

• All Helm Charts

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/api-reference/main.md

[kvalliyurnatt]

just checking for my understanding, if SSL is not enabled ca cert path is going to be "". is that fine from an autoops config stand point ?

[kvalliyurnatt]

why do we continue on errors in this function instead of re queuing, sorry if I missing something 🤔

[naemono]

I've compared the functionality of this onDelete with the other controllers, and this seems to be the pattern we follow. The onDelete is returned from the primary Reconcile function, which does an automatic retry on error.

[kvalliyurnatt]

But then do we need a finalizer to ensure that the autooops policy is not deleted until we have cleaned up the api keys ? sorry if we are already adding a finalizer and I missed it.

[naemono]

@kvalliyurnatt I think the "Todo" item in the readme covers this. Peter mentioned this previously I believe:

Also I don't think it handles the case where a user changes the label selector on the policy. See this section from the design:

Redefinition of selector. Problem: we now have orphaned AutoOps agents
Same approach as outlined below for deletion: add extra label metadata to track origin of deployments, diff
existing vs expected, delete unwanted, remove unnecessary API keys/users from no longer instrumented clusters.

Do you agree?

[kvalliyurnatt]

is there a way for us to test the negative cases here, like the ES API failing with an error other than not found.

[kvalliyurnatt]

I did not understand this, say we have three es instances.
The first one had no error, second one had an error and third one had no errors. In this case shouldn't the ready count be 2 ? but since we never reset errorCount between iterations, in this case the ready count will be 1 right ?

[kvalliyurnatt]

I think we should aa a unit test for this function