PUCM Maintenance Signals

[niontive]

Which issue this PR addresses:

Implement maintenance signals for this: https://issues.redhat.com/browse/ARO-1933

What this PR does / why we need it:

1. Add pucmPending flag to cluster properties
2. Add maintenance task for pucm pending update. This is set by admin PATCH geneva action. It avoids the normal "admin update" code path by setting provisioning state to "Updating"
3. Use state machine in cluster monitoring stack to output PUCM signals

Test plan for issue:

• Unit tests
• Test in INT

Is there any documentation that needs to be updated for this PR?

[dem4gus]

Suggestion for the test cases

[cadenmarchese]

Looks great, just one adjustment suggested. Thanks!

[dem4gus]

Small suggestion but otherwise looks good.

[hawkowl]

@niontive We're planning on moving away from the "PUCM" terminology once the automation is going. Could we change this to a more generic "maintenancepending"?

[facchettos]

I think this is not the way it should be done, I believe we should emit the metrics directly when we are updating the cluster, rather than storing the state in the database and then polling it. This will increase the load on the monitor when we don't need to. We have access to the metrics emitter directly from the frontend (and most likely the backend too) and we should use that from there directly imho.

[AldoFusterTurpin]

Thank you for this. Great work so far!

I left some suggestions to simplify a bit a few things in case you find them useful.

Thanks! 😃

[niontive]

@facchettos I reviewed this design with @bennerv and @s-amann. Below is the justification for choosing using the cluster monitoring stack vs just the frontend:

• For resource health check, these metrics get picked up by a Geneva monitor. That Geneva monitor must emit resource impacting events once a minute. According to the resource health check docs, this is a "must" from the resource health team.
• For maintenance pending signal, we'll emit this two weeks before performing PUCM. This means we need to continuously generate that metric over the two week period. Geneva monitors are not designed to perform like step functions; they need the metric to be present continuously to emit the resource impacting event.
• PUCM can fail. After speaking with Spencer, Ben and Puneet, we decided to leave the cluster in a "pucm ongoing" state as ARO SREs finish maintenance work. This means if we rely solely on the frontend, we'll need some logic to check if the PUCM failed and continuously output the metrics.
• Any metrics outputted from ARO-RP would probably need some separate goroutine to emit the metric as the maintenance progresses. This puts additional load on the frontend and/or backend. The cluster monitor threads didn't get overloaded during INT testing, so I think we should be ok here.
• RP and monitoring services can crash. In that event, it's helpful to have the state saved such that upon restart, the monitoring service can know the last known maintenance state and emit the appropriate metric.

[petrkotas]

Hi @niontive I do see all remarks are answered, the design is thought through and the code is ok. Thanks for the PR.

[AldoFusterTurpin]

LGTM! 🚀 Thanks for applying the suggestions.