CMP write to container log

[peschmae]

Summary

Currently it's not possible to write any logs from within the commands run in the CMP sidecars.
Everything written to stdout is passed to kubectl while the things written to stderr are only visible if the commands fail.

Motivation

To enable easier debugging it would be great to be able to write some logs from the commands within CMP.

In our setup we had a sidecar that sometimes stopped processing the repos, and we don't have a way to debug it, beside writting into a log file within the sidecar and check that from a shell.
Beeing able to write some logs from a plugin sidecar command, would be really helpful.

Proposal

One way to implement this, would be to provide an additional command for the cmp-plugin-server, that can be called with the log level (and thus beeing able to respect the configured log level) and the message, eg
/var/run/argocd/argocd-cmp-server log --level=DEBUG --msg="Hello world"

Another alternative might be to have a pipe, that is always just written to the container logs by the cmp-server, but then the handling of the log level would need to be done by command itself, and the log level would need to be exposed to the commands somehow (or make the loglevel for the cmp-server configurable through an environment variable, that can then be read from the commands)

[peschmae]

This would also make development of sidecars easier and would support #15006

[crenshaw-dev]

How about logging to stdout? Argo CD could pick that up and write it directly to the sidecar logs, regardless of whether the generate command succeeds.

[peschmae]

How about logging to stdout? Argo CD could pick that up and write it directly to the sidecar logs, regardless of whether the generate command succeeds.

This would be the easiest solution, but currently CMP assumes that everything written to stdout is a kubernetes object, in either YAML or JSON, at least according to the documentation: https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/#write-the-plugin-configuration-file

Maybe logging to stderr and forwarding that to the sidecar logs, regardless of the command exit code, would be an alternative.

[crenshaw-dev]

Ah, yep, I meant stderr. Yay Friday brain. I think that would be a fine solution!

[peschmae]

Ah, yep, I meant stderr. Yay Friday brain. I think that would be a fine solution!

That would also be quite easy to implement I think.

Looking at the code in the cmpserver/plugin I think it just needs an additional log statement for the success cases around here: https://github.com/argoproj/argo-cd/blob/master/cmpserver/plugin/plugin.go#L123

If you're open to this change, I would create a pull request for it.
I might however need some pointers how to write a test for it, if that's needed. I couldn't see another test that handles stderr/stdout in the plugin_test.go file, but maybe I missed something

[crenshaw-dev]

Yep, please open a PR! For just some logging, I think we're okay without a test

[peschmae]

I've opened the pull request
@crenshaw-dev would it be possible to get that int 2.9 or is that too late as there are already RCs out?