[Feature] Hold vtgate spec changes until vttablet spec changes are fully deployed (or vice-versa)

[bluecrabs007]

updateStrategy: Immediate of vitess-operator releases changes to all Vitess components simultaneously, including vtgate & vttablet. We have seen elevated impact when vtgate & vttablets are being rolled out concurrently. As PRS relies on vtgate query buffering to make the failover transparent to the client.

Vitess-Operator already handles staggered way of deploying changes by ensuring only one vttablet in a shard is updated at any point of time. So it makes sense to extend this pattern to ensure that vtgate changes are not released when vttablet changes are in progress or vice-versa.

Ask: Either modify the "updateStrategy: Immediate" to hold vtgate until all vttablets are upgraded OR add a updateStrategy strategy that does this.
This way, we can deploy changes to the VitessCluster CRD Spec and continue to use the "Immediate" way of automatically deploying the changes in a safe manner.

#473 proposes to address this in a different approach, bringing in user-input to the process. The feature request I am proposing is to improve the automated deployment process to be more safer.

[frouioui]

It does sound like a good idea. The use of a new VitessClusterUpdateStrategyType would be a better approach, leaving Immediate and External as they are today.

The VitessCluster reconciler, among other things, create and update the VitessKeyspace and VitessCell CRDs. The VitessKeyspace CRD indirectly manages the shards and tablets we have in the cluster by creating and updating another CRD VitessShard. The VitessCell reconciler manages the VTGate service and deployment among other things.

The use of a new VitessClusterUpdateStrategyType could trigger a requeue in the VitessCell reconciler to wait until the VitessCluster Status shows that all shards have been rolled out and the shards are healthy, by using the ReadyShards and UpdatedShards fields. One complexity here is to ensure that UpdatedShards is set to 0 before we attempt to reconcile the VitessCell.

The requeue delay of the VitessCell request, could be configured by a CLI flag.