Helm Chart: Add ServiceMonitor

[icelynjennings]

Proposed changes

Closes #4293

Testing: templated successfully via a local helm template cli run, but ideally should be further tested in Prometheus.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[nginx-bot]

Hi @icelynjennings! Welcome to the project! 🎉

Thanks for opening this pull request!
Be sure to check out our Contributing Guidelines while you wait for someone on the team to review this.

[github-actions]

✅ All required contributors have signed the F5 CLA for this PR. Thank you!
_{Posted by the CLA Assistant Lite bot.}

[icelynjennings]

I have hereby read the F5 CLA and agree to its terms

[sjberman]

Thanks for the contribution! Unfortunately this doesn't fully implement the desired solution. While this would create a ServiceMonitor for the control plane, it doesn't create one for the nginx data plane.

Since the data plane is provisioned by the control plane when a Gateway is created, all of the configuration for the data plane deployment is built into the control plane's code, and not through helm.

We would probably have to follow a similar implementation that we did with HPA (see here: #3702) to enhance our API and code to allow users to enable service monitor for the nginx data plane Deployment.

[icelynjennings]

Thanks @sjberman. Would there be benefit to adding a ServiceMonitor for the control plane now, and data plane later? Or do you think it would only make sense to develop this for both simultaneously?

[sjberman]

Honestly, metrics for the data plane are probably the more interesting ones for most users, since that's where the traffic is flowing. The control plane monitor could be an easy one to do (since it's just helm changes), but to truly close the linked issue, I think we would want data plane support.

[icelynjennings]

Thanks for linking a relevant PR @sjberman! That looks like mostly kubebuilder, so I should be able to finish that part.

[sjberman]

@icelynjennings No problem. Part of it is kubebuilder to define the API, but there are a decent amount of code changes internally too in order to provision the objects based on the API.