bug: APIBinding deletion stuck when status.boundResources[].storageVersions contains a version the bound schema no longer serves #4466

Describe the bug

APIBinding deletion can get stuck forever with:

- type: BindingResourceDeleteSuccess
  status: "False"
  reason: ResourceDeletionFailed
  severity: Error
  message: the server could not find the requested resource

when status.boundResources[].storageVersions contains a version that the currently bound
APIResourceSchema no longer serves. This can happen even for APIExports managed through api-syncagent.

There are two parts to this:

1. The APIBinding reconciler only ever adds to storageVersions. It merges the bound CRD's
   status.storedVersions with the binding's existing storageVersions and never removes any:
   apibinding_reconcile.go.
   So when an APIExport moves a resource to a new schema that drops an old version, the old
   version stays recorded in existing bindings.

2. The APIBinding deletion controller uses every recorded storage version as an API version to
   request. deleteAllCRs lists and deletes collections through
   /apis/<group>/<version>/<resource> for each entry in StorageVersions:
   apibinding_deletor.go.
   For a version that is no longer served, the list returns 404. Any error from deleteAllCRs
   sets ResourceDeletionFailed and returns before the finalizer is removed:
   apibinding_deletion_controller.go.
   Retrying never helps, so apis.kcp.io/apibinding-finalizer is never removed.

A storage version (a version objects have been stored in) is not the same as a version the API
currently serves, but the deletion controller treats them as the same.

In our production installation this blocked deletion of 845 APIBindings to the same
APIExport, across 845 workspaces, for almost 3 months. All of them had the same state:

status:
  boundResources:
  - group: <group>
    resource: <resource>
    schema:
      name: <hash>.<resource>.<group>
      UID: ...
      identityHash: ...
    storageVersions:   # recorded on the binding
    - v1alpha1         # no longer served by the bound schema -> 404
    - v1alpha2
  # ...other resources of the same export, all fine

The bound APIResourceSchema has only:

spec:
  versions:
  - name: v1alpha2
    served: true
    storage: true

In the consumer workspace:

So the workspace had no objects left; only the stale version blocked deletion.

Steps To Reproduce

Not yet reproduced from scratch. These steps follow from the code paths above and from the
state we observed:

1. In a provider workspace, create an APIResourceSchema S1 for widgets.example.dev serving
   v1alpha1 (storage: true), and an APIExport that exports it with storage.crd: {}.
2. In a consumer workspace, create an APIBinding to that export. Wait until it is Bound
   (status.boundResources[0].storageVersions: [v1alpha1]).
3. Create an APIResourceSchema S2 for the same resource serving only v1alpha2
   (storage: true), and update the APIExport to use S2.
4. Wait for the APIBinding to re-bind. storageVersions becomes [v1alpha1, v1alpha2]: the old
   entry is kept, v1alpha1 is no longer served.
5. Delete the APIBinding.
6. Observe that the APIBinding keeps its deletionTimestamp and finalizer forever, with
   BindingResourceDeleteSuccess=False, reason ResourceDeletionFailed, message
   the server could not find the requested resource.

Expected Behaviour

Deletion should clean up the bound resource through a version the bound schema currently
serves, then remove the finalizer:

• Instances are listed and deleted (DeleteCollection), and remaining instances and their
  finalizers are counted, through one served version. All stored instances are readable
  through any served version of the CRD. If stored data can't be converted, the list fails, and
  blocking finalization is then correct.
• A historical, no-longer-served storage version alone should not block deletion forever.

Optionally, the APIBinding reconciler could stop recording (or prune) storage versions that are
no longer served. However, the deletion controller should handle this itself, because
already-affected bindings exist.

Additional Context

Version: v1.36.0+kcp-v0.33.1 (the build has local patches; the code paths above are the
same on main at 8a1a11ca0).

Workaround we used: for each stuck binding, patch apibindings/status to remove the
unserved versions, after confirming that listing through the served versions succeeds. A JSON
patch with a test op guards against concurrent changes:

kubectl patch apibinding <name> --subresource=status --type=json -p '[
  {"op":"test","path":"/status/boundResources/<i>/storageVersions","value":["v1alpha1","v1alpha2"]},
  {"op":"replace","path":"/status/boundResources/<i>/storageVersions","value":["v1alpha2"]}
]'

Within seconds the deletion controller removed the finalizer and the bindings were deleted.
This needs patch on apibindings/status, which tenant users usually don't have.

Possible fix: in deleteAllCRs, resolve the bound APIResourceSchema
(status.apiExportClusterName + boundResources[].schema.name, check the UID matches), then
list and delete through a single served version, preferring the storage version. If the schema
can't be resolved, fall back to the recorded StorageVersions. We have a draft of this with
unit tests and are happy to open a PR.

Possibly related: #1755 (same condition, but the cause there was TLS).