Problematic sync order for force replaced resources

[tvandinther]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When using argocd.argoproj.io/sync-options: Replace=true,Force=true sync options on a resource, you can end up in a sync deadlock when dealing with the creation or deletion of dependant resources.

For example, if you have a statically provisioned PV and a PVC claiming that PV via the spec.volumeName field, you have to specify a sync order using sync waves to get it to work.

To Reproduce

Approach 1

Unable to update

Create

• Wave 1
  • Create PV
• Wave 2
  • Create PVC
  • PVC and PV are bound

Delete

• Wave 2
  • Delete PVC
  • PV is released
• Wave 1
  • Delete PV

Update

• Wave 1
  • Update PV
  • Stuck on immutable field
• Wave 1 (alternative)
  • Update PV (Replace=true,Force=true) [Delete -> Create]
  • PV pending delete (stuck on finalizer due to bound PVC)

Approach 2

Unable to delete

Create

• Wave 1
  • Create PVC
  • Use custom health check to consider Pending PVC as healthy
• Wave 2
  • Create PV
  • PVC and PV are bound

Delete

• Wave 2
  • Delete PV
  • PV pending delete (stuck on finalizer due to bound PVC)
• Wave 1 (never happens)
  • Delete PVC

Update

• Wave 1
  • Update PVC
  • Stuck on immutable field
• Wave 1 (alternative)
  • Update PVC (Replace=true,Force=true) [Delete -> Create]
  • New PVC Pending
  • PV is released
• Wave 2
  • Update PV (Replace=true,Force=true) [Delete -> Create]
  • New PVC and PV are bound

Expected behavior

It should be possible to sync new configuration on resources which are immutable by opting in to recreating them. However, if these resources have dependants, finalizers can deadlock the sync. This is because a recreation (i.e. Replace=true,Force=true) is a delete action followed by a create.

Sync waves are thought out in this regard such that during pruning, sync waves are run in reverse order. However, this logic also needs to apply to resources marked for recreation. If I have two resources marked for recreation, the one with the greater sync wave should be deleted first, but creation should then occur in ascending sync wave order.

In summary, resources marked with Replace=true,Force=true will be deleted and as such should be considered pruned (in effect) and therefore be taken care of during a pruning round prior to syncing. If this can be problematic, at least a new sync option could be added to handle recreation explicitly as a "delete in reverse order before performing the sync" functionality.

Version

2.12.2+560953c