kcp v0.33.1: a wildcard partial-metadata LIST writes into the watch cache's objects — fatal error: concurrent map writes kills the whole process #4551

Summary

Two (or more) concurrent wildcard partial-metadata LIST requests (/clusters/*/…, Accept: application/json;as=PartialObjectMetadataList;g=meta.k8s.io;v=v1) that read the same object version can write into the same Go map. Go detects it and aborts the entire kcp process (fatal error: concurrent map writes, not recoverable by recover()), so every Workspace on the shard is down until the process restarts.

Observed on a test instance (kcp v0.33.1, official image) six times within one day once several clients issued this request on a schedule, and reproduced 3/3 on a disposable instance of the same release with the shape below (process gone within ~100 ms of the load starting). The same load with plain application/json (full objects, no as=) did not crash it in 5 minutes (256,864 wildcard LISTs, 70,020 concurrent updates).

Where

Pinned versions (kcp v0.33.1's go.mod replaces k8s.io/* with github.com/kcp-dev/kubernetes/staging/src/k8s.io/* v0.0.0-20260602065202-e006560fc76a):

• k8s.io/apiserver/pkg/endpoints/handlers/response_kcp.go:61 — setKCPOriginalAPIVersionAnnotation:

  annotations := partial.GetAnnotations()
  if annotations[KCPOriginalAPIVersionAnnotation] != "" { return }
  if annotations == nil { annotations = make(map[string]string) }
  …
  annotations[KCPOriginalAPIVersionAnnotation] = t.GetAPIVersion()     // line 61: map write
  partial.SetAnnotations(annotations)

  It only does anything for a wildcard request (request.ClusterFrom(ctx).Wildcard).
• k8s.io/apiserver/pkg/endpoints/handlers/response.go:474 (list; :432 for a single object, :453 for the v1beta1 variant) — asPartialObjectMetadataList calls it for every item after
  partial := meta.AsPartialObjectMetadata(m).
• k8s.io/apimachinery/pkg/api/meta/meta.go:112-137 — AsPartialObjectMetadata builds the PartialObjectMetadata with Labels: m.GetLabels(), Annotations: m.GetAnnotations(): it copies the map header, not the map. For a *metav1.ObjectMeta the whole struct (ObjectMeta: *t) is copied, with the same effect.
• The objects m come from the watch cache (the reproduction and the observed incidents use resourceVersion empty, i.e. MostRecent, which this build serves from the cache), i.e. they are shared between concurrent requests.
• Entry through kcp's own handler chain: pkg/server/filters/wildcard.go:59 (WithWildcardListWatchGuard) → … → handlers.ListResource → handleList → transformResponseObject → doTransformObject (target.Kind == "PartialObjectMetadataList") → asPartialObjectMetadataList.

Mechanism

1. Object O is in the watch cache with an annotations map A that does not yet contain kcp.io/original-api-version (every freshly written version; objects written before the request loop if annotations[...] != "" returns early only once a previous request has already written the key into the shared map).
2. Request 1 and request 2 (both wildcard, both partial-metadata) obtain O. AsPartialObjectMetadata gives each a PartialObjectMetadata whose Annotations is A.
3. Both execute annotations[KCPOriginalAPIVersionAnnotation] = … on A at the same time → fatal error: concurrent map writes.

A single request cannot trigger it (it is the only writer), which is why it hides until two clients list the same collection at the same moment. A plain-JSON LIST is not affected: doTransformObject with target == nil returns the objects unchanged and encodes them (read-only).

Minimal reproduction shape

Needs a kcp you may kill; nothing else.

1. One shard, any workspace ws with a few dozen ClusterRoles (any resource served to wildcard requests works; the one used had 50).
2. A credential allowed to LIST clusterroles in /clusters/* (e.g. system:masters).
3. Writers: 4 goroutines doing GET + UPDATE (a label) of those ClusterRoles in a loop, so that objects continually get new versions that have no kcp.io/original-api-version yet.
4. Listers: 8 goroutines, each in a loop:

   GET /clusters/*/apis/rbac.authorization.k8s.io/v1/clusterroles
   Accept: application/json;as=PartialObjectMetadataList;g=meta.k8s.io;v=v1
   

5. Result: kcp aborts within ~0.1 s (3 of 3 runs on a clean start). Stack (abridged):

   fatal error: concurrent map writes
   k8s.io/apiserver/pkg/endpoints/handlers.setKCPOriginalAPIVersionAnnotation  response_kcp.go:61
   k8s.io/apiserver/pkg/endpoints/handlers.asPartialObjectMetadataList.func2   response.go:474
   k8s.io/apimachinery/pkg/api/meta.eachListItem                               help.go:181
   k8s.io/apiserver/pkg/endpoints/handlers.asPartialObjectMetadataList         response.go:467
   k8s.io/apiserver/pkg/endpoints/handlers.doTransformObject                   response.go:262
   k8s.io/apiserver/pkg/endpoints/handlers.transformResponseObject             response.go:335
   k8s.io/apiserver/pkg/endpoints/handlers.handleList                          get.go:323
   github.com/kcp-dev/kcp/pkg/server/filters.WithWildcardListWatchGuard        wildcard.go:59
   

6. Same listers with Accept: application/json: no crash in 5 min (256,864 LISTs of 50 objects each, 70,020 updates).

(A realistic trigger is two controllers/clients that each confirm the state of a cross-workspace collection with a partial-metadata wildcard LIST on a schedule, while the collection receives updates; kcp's own kcp-dev/kcp/pkg/metadata dynamic metadata client uses exactly this request for wildcard lists/watches.)

Why this is not just a client problem

Anything that sends the request above can take the shard down; the response is also a write on shared cache state from a read request. kcp's own metadata client (pkg/metadata/dynamic.go, metadataTransport) and informers built on it issue this request for wildcard lists and watches (the watch path goes through the same doTransformObject, see the comment at response.go:249, and so reaches the per-object variant at :432; only the list variant was exercised by the reproduction), so any controller using it is exposed.

Suggested fix directions (for the maintainers to choose)

• In setKCPOriginalAPIVersionAnnotation never mutate the aliased map: copy it before writing (maps.Clone then set), or build the PartialObjectMetadata with a deep copy of Labels/Annotations in AsPartialObjectMetadata's kcp variant.
• Or compute the annotation once when the object enters the cache.

Versions

kcp v0.33.1 (github.com/kcp-dev/kcp@v0.33.1), apiserver/apimachinery from github.com/kcp-dev/kubernetes pinned at e006560fc76a (module version v0.0.0-20260602065202-e006560fc76a), Go 1.26.x runtime. Exact pin of the observed build: kcp v0.33.1's go.mod → github.com/kcp-dev/kubernetes e006560fc76a; the code quoted above is read there. The same carry also exists, unchanged in the relevant lines, as commit daae13cf03971c6cc5fb869677bb1bd7692df880 ("UPSTREAM: : endpoints: set kcp.io/original-api-version on wildcard partial metadata requests", latest commit of response_kcp.go on branch kcp-1.37.0), so the defect is likely not fixed there; newer kcp releases (v0.33.4–v0.33.6) were not tested.