MON-4359: Add Monitoring Capability #2468

rexagod · 2025-09-03T22:14:27Z

Add a "Monitoring" capability to support optional monitoring.

openshift-ci · 2025-09-03T22:14:31Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2025-09-03T22:14:33Z

Hello @rexagod! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

Blocked by: openshift/api#2468 Signed-off-by: Pranshu Srivastava <[email protected]>

openshift-ci-robot · 2025-09-03T23:09:45Z

@rexagod: This pull request references MON-4359 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.21.0" version, but no target version was set.

In response to this:

Add a "Monitoring" capability to support optional monitoring.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

everettraven · 2025-09-29T13:12:45Z

config/v1/types_cluster_version.go

+
+	// ClusterVersionCapabilityOptionalMonitoring manages the cluster monitoring stack which is responsible for gathering and
+	// processing metrics from the in-house and user workloads. The following CRDs are constitute this capability:
+	// - TODO


Presumably we should not add this capability until we know the full scope of what this capability will and will not do?

everettraven · 2025-09-29T13:13:40Z

config/v1/types_cluster_version.go

+	// WARNING: This capability will drop all aforementioned CRDs, and may operational issues in the cluster.
+	// The only supported use-case for this capability is to reduce the monitoring stack's resource usage by only
+	// supporting telemetry.


What operational issues may happen here?

What does "only supporting telemetry" mean here?

Added context.

Operations relying the optional components, i.e., Alertmanager, and all UWM components will be disabled. For e.g., the "Alerts" tab under "Observe" will stop showing alerting (firing, pending, etc.).

Telemetry here refers to https://rhobs-handbook.netlify.app/products/openshiftmonitoring/telemetry.md that teams can use to gather telemetry metrics about their workloads from clusters.

everettraven · 2025-09-29T13:14:51Z

config/v1/types_cluster_version.go

 	ClusterVersionCapabilityCloudControllerManager ClusterVersionCapability = "CloudControllerManager"
+
+	// ClusterVersionCapabilityOptionalMonitoring manages the cluster monitoring stack which is responsible for gathering and
+	// processing metrics from the in-house and user workloads. The following CRDs are constitute this capability:


Is it only these CRDs that are removed when this capability is disabled? Are there default monitoring workloads that may no longer be running on the cluster when this is disabled?

Amended. Disabling the capability drops all optional monitoring components (AM and the UWM stack) and any manifests related to them. Do note that CRDs will persist as these are needed by the operator regardless of the capability's state.

config/v1/types_cluster_version.go

everettraven · 2025-09-29T13:17:04Z

config/v1/types_cluster_version.go

+		ClusterVersionCapabilityCloudCredential,
+		ClusterVersionCapabilityIngress,
+		ClusterVersionCapabilityCloudControllerManager,
+		ClusterVersionCapabilityOptionalMonitoring,


Why should this be included in the 4.19 set if the capability does not exist in 4.19?

Dropped, targeting 4.21 TechPreview now.

everettraven · 2025-09-29T13:17:29Z

config/v1/types_cluster_version.go

+		ClusterVersionCapabilityCloudCredential,
+		ClusterVersionCapabilityIngress,
+		ClusterVersionCapabilityCloudControllerManager,
+		ClusterVersionCapabilityOptionalMonitoring,


Why should this be included in the 4.20 capability list if this capability does not exist in 4.20?

Dropped, targeting 4.21 TechPreview now.

everettraven · 2025-09-29T13:55:38Z

/assign

rexagod · 2025-10-12T20:27:40Z

@everettraven Apologies! It seems there was some development before I got a chance to update this after marking it ready.

I'll make changes to implicitly enable the capability for TechPreview only in 4.21.

Add an "OptionalMonitoring" capability to support optional monitoring. Signed-off-by: Pranshu Srivastava <[email protected]>

openshift-ci · 2025-10-12T22:30:05Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from everettraven. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rexagod · 2025-10-12T22:42:11Z

@everettraven I'm probably missing something but I expected to find something similar to the feature-gates workflow (annotations/tags and such) to enable this just for TechPreview. Could you please drop some pointers on how I can achieve that? Thanks!

openshift-ci · 2025-10-12T22:58:49Z

@rexagod: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

everettraven · 2025-10-13T14:32:41Z

@rexagod As far as I am aware, the process for tech preview capabilities is the same as adding any new enum value under TechPreviewNoUpgrade, meaning:

An OpenShift Enhancement Proposal
A new feature gate
Adding the new enum using a feature-gate aware enum marker

I'm relatively new to capabilities for OpenShift so I read up on the original intention of them in https://github.com/openshift/enhancements/blob/df1fada1e045b2e81b677941414f3085c02945fd/enhancements/installer/component-selection.md

With my current understanding of how capabilities work it should be an all-or-nothing type configuration. It seems like an anti-pattern to have a partial capability.

I'd like to see answered in the EP you write up:

Why your cluster operator cannot be responsible for removing optional monitoring configurations
What is and is not included in this capability and why it is okay to exclude some monitoring components and not others
What other components may have a dependency on these now optional components

Thanks!

Edit to clarify: Re-reading the description of the capability, I'm not sure a capability is what you are actually looking for here. This seems more like an option that should be configured on and managed by the CMO itself.

rexagod · 2025-10-13T16:10:16Z

👋🏼 I'm a bit confused about the EP and feature-gate part, since https://github.com/openshift/enhancements/blob/df1fada1e045b2e81b677941414f3085c02945fd/enhancements/installer/component-selection.md#how-to-implement-a-new-capability doesn't mention those?

What is and is not included in this capability and why it is okay to exclude some monitoring components and not others

I've tried to answer this in the godoc for the capability but essentially this allows us to tune down the resource consumption of the in-cluster monitoring stack by not deploying optional components.

Why your cluster operator cannot be responsible for removing optional monitoring configurations
Re-reading the description of the capability, I'm not sure a capability is what you are actually looking for here. This seems more like an option that should be configured on and managed by the CMO itself.

We chose to go with capabilities to support configuring these at install time for clusters that are only concerned with forwarding telemetry metrics.

What other components may have a dependency on these now optional components

The optional components in question are Alertmanager and all UserWorkloadMonitoring components managed by CMO. The latter is opt-out by default, so no dependencies there. For AM, as I mentioned earlier, this could impact any areas that expect an AM presence 100% of the time (so far, monitoring-plugin, which we'll fix down the line to hide the "Alerts" tab when AM is absent).

everettraven · 2025-10-14T12:36:06Z

I'm a bit confused about the EP and feature-gate part, since https://github.com/openshift/enhancements/blob/df1fada1e045b2e81b677941414f3085c02945fd/enhancements/installer/component-selection.md#how-to-implement-a-new-capability doesn't mention those?

This document looks like it hasn't been updated in the last couple years. Our processes have changed throughout this time such that any API change, especially to a v1 API, must go behind a feature gate in TechPreviewNoUpgrade. Creating a feature gate requires an OpenShift enhancement proposal. This is meant to ensure that we:

Don't inadvertently ship something as GA without a backing implementation
Don't GA something without appropriate regression testing

This is all meant to ensure we are maintaining a quality bar for changes that go into the product.

I've tried to answer this in the godoc for the capability but essentially this allows us to tune down the resource consumption of the in-cluster monitoring stack by not deploying optional components.

I appreciate you taking the time to answer some of this in the GoDoc. I'm still not quite sure I have an understanding from reading that why some pieces of the monitoring stack we deploy by default is considered optional and why others are not.

We chose to go with capabilities to support configuring these at install time for clusters that are only concerned with forwarding telemetry metrics.

My understanding of capabilities is that they are all-or-nothing type mechanisms to tell the ClusterVersionOperator whether or not to stamp $thing out onto the cluster, not a way to tell cluster operators what they should and should not be doing. I don't think this partial use case is one that the "capability" concept was intended to support.

I'm not sure off the top of my head if there is another way to achieve what you are looking for at install time, but maybe @JoelSpeed would be more familiar with the options you have here.

I've got a few questions here that having an EP would be helpful to field these discussion:

Why does a user need to configure this at install time?
What is the impact to an end user if they can only configure this option as a "day 2" operation?
Should users be able to enable/disable the optional pieces of the monitoring stack at will?

simonpasquier · 2025-10-15T08:43:28Z

config/v1/types_cluster_version.go

+	// * ThanosRuler User Workload Monitoring
+	//
+	// NOTE: The only supported use-case for this capability is to reduce the monitoring stack's resource usage by only
+	// supporting telemetry (see https://rhobs-handbook.netlify.app/products/openshiftmonitoring/telemetry.md). Turning


I would rather link to the official docs: https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/support/remote-health-monitoring-with-connected-clusters

simonpasquier · 2025-10-15T08:55:51Z

config/v1/types_cluster_version.go

+	// ClusterVersionCapabilityOptionalMonitoring manages the enablement of the optional components of the in-cluster monitoring stack.
+	// The optional components are:
+	// * Alertmanager
+	// * Alermanager User Workload Monitoring


should we say "All user-defined monitoring components"?

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 3, 2025

openshift-ci bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 3, 2025

rexagod added a commit to rexagod/cluster-version-operator that referenced this pull request Sep 3, 2025

MON-X: Bump API version to include monitoring capability

949b387

Blocked by: openshift/api#2468 Signed-off-by: Pranshu Srivastava <[email protected]>

rexagod mentioned this pull request Sep 3, 2025

MON-4358: Bump API version to include monitoring capability openshift/cluster-version-operator#1229

Draft

rexagod changed the title ~~MON-X: Add Monitoring Capability~~ MON-4359: Add Monitoring Capability Sep 3, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 3, 2025

rexagod force-pushed the opt-monitoring-capability branch 3 times, most recently from 6d757d5 to 72b018d Compare September 29, 2025 02:26

rexagod marked this pull request as ready for review September 29, 2025 02:27

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 29, 2025

openshift-ci bot requested review from JoelSpeed and everettraven September 29, 2025 02:27

everettraven reviewed Sep 29, 2025

View reviewed changes

openshift-ci bot assigned everettraven Sep 29, 2025

rexagod marked this pull request as draft October 12, 2025 20:28

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 12, 2025

rexagod mentioned this pull request Oct 12, 2025

MON-4361,MON-4380: Optional Monitoring Capability openshift/cluster-monitoring-operator#2675

Open

2 tasks

MON-4359: Add Monitoring Capability

051d592

Add an "OptionalMonitoring" capability to support optional monitoring. Signed-off-by: Pranshu Srivastava <[email protected]>

rexagod force-pushed the opt-monitoring-capability branch from 72b018d to 051d592 Compare October 12, 2025 22:29

rexagod marked this pull request as ready for review October 12, 2025 22:29

openshift-ci bot requested a review from everettraven October 12, 2025 22:30

simonpasquier reviewed Oct 15, 2025

View reviewed changes

MON-4359: Add Monitoring Capability #2468

Are you sure you want to change the base?

MON-4359: Add Monitoring Capability #2468

Uh oh!

Conversation

rexagod commented Sep 3, 2025

Uh oh!

openshift-ci bot commented Sep 3, 2025

Uh oh!

openshift-ci bot commented Sep 3, 2025

Uh oh!

openshift-ci-robot commented Sep 3, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

everettraven commented Sep 29, 2025

Uh oh!

rexagod commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Oct 12, 2025

Uh oh!

rexagod commented Oct 12, 2025

Uh oh!

openshift-ci bot commented Oct 12, 2025

Uh oh!

everettraven commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rexagod commented Oct 13, 2025

Uh oh!

everettraven commented Oct 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

openshift-ci-robot commented Sep 3, 2025 •

edited by openshift-ci bot

Loading

rexagod commented Oct 12, 2025 •

edited

Loading

everettraven commented Oct 13, 2025 •

edited

Loading