Add immutability supporting limitation in documents.

[blackpiglet]

Thank you for contributing to Velero!

Please add a summary of your change

Does your change fix a particular issue?

Fixes #(issue)

Please indicate you've done the following:

• Accepted the DCO. Commits without the DCO will delay acceptance.
• Created a changelog file or added /kind changelog-not-required as a comment on this pull request.
• Updated the corresponding documentation in site/content/docs/main.

[codecov-commenter]

Codecov Report

Merging #6474 (67b14c8) into main (7deae4c) will decrease coverage by 0.38%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #6474      +/-   ##
==========================================
- Coverage   60.18%   59.81%   -0.38%     
==========================================
  Files         229      236       +7     
  Lines       24219    25088     +869     
==========================================
+ Hits        14577    15007     +430     
- Misses       8634     9041     +407     
- Partials     1008     1040      +32     

see 27 files with indirect coverage changes

[blackpiglet]

This address the need to document the data immutability limitation in issue #6422.

[blackpiglet]

@draghuram
Could you help PTAL?

[draghuram]

Sorry for the delay, will take a look.

[draghuram]

I made some changes to the text. Please feel free to make more changes as you see fit. It is a very difficult topic to document but it is better to have something and then we can improve it.

My version

Starting from 1.11, Velero's backups may not work as expected when the target object storage has some kind of an "immutability" option configured. These options are known by different names (see links below for some examples). The main reason is that Velero first saves the state of a backup as Finalizing and then checks whether there are any async operations in progress. If there are, it needs to wait for all of them to be finished before moving the backup state to Complete. If there are no async operations, the state is moved to Complete right away. In either case, Velero needs to modify the metadata in object storage and that will not be possible if some kind of immutability is configured on the object storage.

Note that backups may still work in some cases depending on specific provider and configuration.

• For AWS S3 service, backups work because S3's object lock only applies to versioned buckets and the object data can still be updated as the new version. But when backups are deleted, old versions of the objects will not be deleted.
• Azure Storage Blob supports both versioned-level immutability and container-level immutability. For the versioned-level scenario, data immutability can still work in Velero, but the container-level cannot.
• GCP Cloud storage policy only supports bucket-level immutability, so there is no way to make it work in the GCP environment.

The following are the links to cloud providers' documentation in this regard:

• AWS S3 Using S3 Object Lock
• Azure Storage Blob Containers - Lock Immutability Policy
• GCP cloud storage Retention policies and retention policy locks

[blackpiglet]

@draghuram
Thanks a lot! That looks better.
May just need to add some information from Velero's design perspective.

[draghuram]

Looks good.