Dynamically Adjusting MySQL StatefulSet Partition to Control MySQL Pod Update Timing

[d-kuro]

What

This issue aims to address the situation of adjusting the update timing of MySQL pods by dynamically changing the partition of MySQL StatefulSet.

In the past, we encountered an issue where the MySQLCluster became unhealhty due to node failure resulting in a non-functional MySQL Pod. During this unhealthy state, an update to the MySQLCluster triggered a rolling update of the StatefulSet. The cluster became unavailable, as two MySQL Pods were down simultaneously due to the failure and update.

Even though PDB was set, the cluster state or existing pod status are not considered during a StatefulSet update. Therefore, the MySQL pod was updated despite the cluster being unhealthy.

How

We plan to dynamically adjust the partition of MySQL StatefulSet to modify the MySQL pod update timing.

The implementation steps are as follows:

• Add a reconciler to update the partition of MySQL StatefulSet.
• In general, set the partition to 99 (a value greater than the replicas that can be set by MOCO, it need not be 99).
• If there is an update to the podTemplate of the sts (currentRevision != updateRevision), look at the Ready state of MySQL Pods, decrease the partition, and update one MySQL Pod at a time.
• If there is a non-Ready MySQL Pod, do not decrease the partition and stop the sts update. We want this behavior equivalent to PDB + Eviction API.
• Aside from these, there are more detailed specifications that need to be considered.

Please note that this feature will be disabled from the initial MySQLCluster deployment until it first becomes Healthy.

Checklist

• Finish implementation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions