Remediator - #654
Draft
dciabrin wants to merge 6 commits into
Draft
Conversation
Introduce remediation.openstack.org PodRemediator CR and controller: when NHC/SNR are present, detect unhealthy nodes and delete bound local PVCs so StatefulSets can reschedule. Includes TopoLVM/LVMS-style PV node affinity, pending-deletion annotation for resumable deletes, and dynamic client for NHC/SNR discovery. Spec: optional disablePVCRemediation (default off = enabled), maxUnhealthyNodes disruption budget, multi-namespace watch. CRD, RBAC, samples, design doc and functional tests included. Co-authored-by: Cursor <cursoragent@cursor.com>
F1: Force-delete pods referencing a PVC before deleting it so the
kubernetes.io/pvc-protection finalizer is released and the PVC
can actually terminate (new deletePodsForPVC helper).
F2: Resume path (pending-deletion annotation) now re-checks node health
before deleting; removes the stale annotation if the node recovered.
F3: podFN and pvcFN now list PodRemediators cluster-wide ("") so a CR
in namespace A watching namespace B is enqueued by B's events.
F4: reconcileDelete strips pending-deletion annotations from all watched
namespaces before removing the CR finalizer, preventing orphan
annotations when the CR is deleted mid-remediation.
F5: nodeReadyChangedPredicate fires only when the NodeReady condition
status actually transitions, not on every kubelet heartbeat.
F6: hadError flag causes a requeue and sets Ready=False when any PVC
list or PV fetch fails, instead of silently reporting Ready=True.
F7: isLocalPV now requires a known node-pinning topology key for CSI and
HostPath volumes (new pvHasLocalTopologyKey helper); Cinder PVs with
zone affinity (topology.cinder.csi.openstack.org/zone) are excluded.
F8: getNodesWithActiveSNR filters the unhealthy-node set to only nodes
with an active SelfNodeRemediation CR, closing the gap between a
node going NotReady and NHC committing to remediate it.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
NHC names SelfNodeRemediation CRs with a random suffix (e.g. worker-0-6lwkb), not the bare node name. The previous implementation stored snr.GetName() as the map key, so snrNodes["worker-0"] was never true and PodRemediator skipped all unhealthy nodes even when an active SNR CR existed. Fix: read the authoritative node name from the medik8s label remediation.medik8s.io/node-name with a fallback to the CR name for forward compatibility. Discovered during RabbitMQ E2E testing on had-18: worker-0 went NotReady, NHC created worker-0-6lwkb SNR CR, but PodRemediator logged "No unhealthy nodes; monitoring" because the map lookup missed the label-keyed node name. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…or selfnoderemediations During RabbitMQ E2E testing two issues were found in the F8 SNR check: 1. The remediation.medik8s.io/node-name identifier is in the SNR CR annotation, not a label. The previous fix read GetLabels() and missed it. Now reads GetAnnotations() with label fallback. 2. The RBAC marker for selfnoderemediations (get;list;watch) existed in the controller code but was missing from config/rbac/role.yaml, causing a Forbidden error at runtime when listing SNR CRs at cluster scope. Added selfnoderemediations to the self-node-remediation.medik8s.io rule alongside selfnoderemediationtemplates. E2E result after both fixes: PASS (zero failures, full virsh destroy + PVC remediation + pod reschedule + node restore flow confirmed). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Adds test-kit/podremediator/ with: - Ansible runbook (runbook-podremediator-poc.yml): steps 1-7 to deploy custom infra-operator image, apply CRD/RBAC, install NHC/SNR via ci-framework, create PodRemediator CR, optional local-PVC test + smoke test - E2E playbooks: virsh-based node-failure scenarios for bundled lab STS, RabbitMQ, and Galera workloads; Galera wsrep monitoring and DR reporting - Lab helper playbooks: Galera worker rebalance, NHC/SNR standalone - Scripts: run-e2e.sh and run-runbook.sh (auto-source local.env for secrets) - Inventories: laptop (ProxyCommand) and jump-host variants with CHANGEME placeholders; .local.yml variants gitignored for real values - group_vars/all/local.yml.example for controller_user_home parameterization - local.env.example for HAD18, SSHPASS, CUSTOM_OPERATOR_IMAGE, etc. - Docs: POC runbook, test plan, Galera E2E scenarios, customer/operator guides All environment-specific hostnames, passwords, and personal image references are removed from committed files and stored in gitignored local.env / local.yml files instead. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Allow the application operator (e.g. mariadb-operator) to decide if the pod remediator is allowed to delete a PVC to unblock a pod stuck on an unhealthy node. The general mechanism is a three-phase handshake implemented via annotations: - pod remediator detects when a PVC with local storage is stuck on a unhealthy node and annotates it annotates with remediation.openstack.org/pvc-stuck-on-node - the application opeerator gets notified, and gets a chance to decide whether deleting this PVC is safe from a data's perspective. If so, the application operator annotate the PVC with remediation.openstack.org/safe-to-delete - when the pod remediator sees the safe-to-delete annotation, it proceeds as usual with the deletion of stuck pod and PVC, to allow k8s to recreate those on a healthy worker node.
Contributor
|
Skipping CI for Draft Pull Request. |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dciabrin The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
antonioromito
pushed a commit
to antonioromito/mariadb-operator
that referenced
this pull request
Aug 7, 2026
Watch annotations set by the PodRemediator, in order to allow it to delete a galera PVC with local storage, in case it's been created on a worker node that became unhealthy, preventing the galera replica to be rescheduled elsewhere until the PVC gets deleted. Depends-On: openstack-k8s-operators/infra-operator#654
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.