Allow mutating some Bucket(Access)Class fields after create/grant

[BlaineEXE]

Initial discussion suggested that we can use Volume Attribute Class (or an analogue of it) for COSI

Use for changing vendor-specific quotas after 1st provisioning VAC KEP - kubernetes/enhancements#3751

The initial intent here is to allow updating vendor-specific parameters that might need to be updated after initial creation of a resource based on BClass/BAClass.

For example, the Ceph COSI driver wishes to allow setting bucket size quotas to prevent users from consuming too much of the limited, on-prem cluster storage. Apps might need to have their BC or BA upgraded to increase their quota space as their app grows.

Similarly, admins may wish to apply newly-available driver features to existing B/BC/BA resources.

This may also be important during the evaluation phase of an application. When an application with unique needs is being onboarded and evaluated, Class parameters may need to be modified somewhat frequently. Allowing these modifications without requiring application redeployment could save users time in this scenario.

[BlaineEXE]

@guymguym raised some concerns that seem most related to this topic:

I just see too many cases where you need to bring along the admin just to be able to set up cluster resources like these, where the entire application is still in evaluation stages. clearly moving to "production" also means moving the control from the user to the admin

Maybe this was discussed here before, or elsewhere in kubernetes, I am referring to the XXXClaim vs XXXClass pattern, like CSI and COSI have.
And I wondered, why shouldn't we have an option to specify an "inline" BucketClassSpec inside the BucketClaimSpec, just next to the BucketClassName reference...
Clearly portability is the drawback, but this is why it would be optional.
The benefit is ergonomics - I see many cases where the class is effectively a "one-off" set of parameters to the backend with some custom parameters, and I wish it was simpler to use for that.
Generally speaking, even setting a class-name is usually not a fully portable thing...
Anyway, I am not trying to say that setting inline class is good for everyone, but wouldn't it make sense for those who do want to do that?
Same thing goes for the BucketAccessClass...

[BlaineEXE]

I see many cases where the class is effectively a "one-off" set of parameters to the backend with some custom parameters, and I wish it was simpler to use for that.

This is a good statement as to what the pain points seem to be with the current COSI API. This is something that weighs on my mind also since I see and get feedback from Rook users about this, and the current OBC system.

Given the importance that the K8s and sig-storage communities have put on portability, I am reluctant to start with the standpoint that portability can be so easily thrown out the window for ease-of-use, but I also don't want to throw this valid concern.

One of the things we have talked about in the COSI meeting recently is potentially having XxClass as well as XxAttributeClasses in a way that mirrors VACs: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/

I think this is a way to still have portability as well as users who can have more control over their own destiny in a way that is gated by admins.

Another alternative that is more directly to the title of this issue is allowing XxClass parameters to be mutable (potentially even for pre-existing bucket/access resources?). For a trusted user who is trying to get things running preproduction, that could also allow for an easier time getting going with a backend that has many tunables, as I can imagine many backends will eventually have.

[BlaineEXE]

One of the corner cases we should plan for with this work is what to do about day-2 changes to Buckets/Accesses. Something I have seen often as a developer with many client installs is that end-user-admins will often begin with the minimal setup that gets them to a working state and then try to iterate on that much later into the future.

For ease of illustration, let's just talk about BucketClass for now.

For COSI, what I think this will mean is that an admin will create a simple BucketClass with few backend parameters set, and applications will begin using it.

After many months there will be an application that grows to have specific needs from the backend. This could be anything, but let's assume that it has high data needs and thus needs to use more of its backend quota (either more storage space, more storage bandwidth, or both).

At this point, the admin will effectively want to create a second BucketClass and move that application to the new BucketClass.

This is where problems start to appear. Does COSI allow BucketClaims to mutate their BucketClass to make the switch? If so, what should backend drivers do if the switch from one to another isn't possible? For example, what if the driver uses a totally different pool of backing storage for normal-quota vs. fast-quota buckets and can't seamlessly migrate? Do we (COSI) want to allow that challenge for drivers in the COSI API?

I think the idea Xing proposed a while back of looking into VolumeAttributesClasses could be an improvement in this case. In theory, the admin could easily create a new VAC for the application, and the user could modify BucketClaim's VAC selection to point to the new class. When defining VACs expectations, COSI could make it clear that the drivers are expected to be able to support mutation for all VAC params.

I still have similar concerns as before though. I can imagine backends that would have some mutable class params and some non-mutable class params. In Ceph, for example, allocating more space quota for a bucket would be easy, but I don't think it would be possible to move the bucket to a different speed tier of storage for more bandwidth quota.

For any application that needs to move BucketClaim configuration from one Class to another where in-place mutation isn't possible, I think the logical process would be to have the user create a new BucketClaim with the new class, and then copy data to the new claim and switch over during a maintenance window. COSI could require (or ask/suggest) backends to automatically do migrations, but I think this might be a bit too big of a burden for many driver implementations, and end users might be surprised if their data goes unavailable for a time during the internal switchover.

Would it be too much complexity for COSI drivers to be expected to separate their backend-specific parameters into mutable and non-mutable categories? Or too much complexity for end users?

Personally, I think having a VAC analogue for mutable params, and existing opaque params for non-mutable (or default) params allows for good flexibility as well as an ability for drivers distinguish for admins what flexibility can happen in-place and what cannot. It is also the more complicated option for COSI compared to mutable BucketClasses. I'm inclined to think the benefits outweigh the drawbacks here, but I'm interested in seeing how the conversation evolves from here.