Skip to content

Investigate EtcdRangeStream for LogicalClusterMigration etcd operations and LogicalClusterDump #4519

Open

Description

@mjudeikis-bot

Context

Kubernetes 1.37 adds the EtcdRangeStream feature gate, which the kube rebase enables by default (Beta) in pkg/features/kcp_features.go. Requested in #4514 (comment).

The logical cluster migration currently pages through etcd with plain paginated Get calls. We should investigate whether range streaming can replace or simplify that, and whether the LogicalClusterDump ephemeral resource can be streamed instead of paged.

What exists today

  • pkg/server/migrationdump/server.go (scanEtcdEntries, nextClusterRange): origin shard scans etcd with kv.Get(..., WithRange, WithLimit(etcdScanLimit)) and returns pages in LogicalClusterDump.status with a continue token, bounded by spec.limit and spec.maxBytes.
  • pkg/reconciler/migration/logicalclustermigration/datacopy.go (copyPageFromOriginViaHTTP, writeDumpEntries): destination shard requests one dump page per call over HTTP and writes the entries to its own etcd.
  • pkg/reconciler/migration/logicalclustermigration/datacleanup.go (deleteOriginData, scanEtcdKeys): origin cleanup pages through keys with WithLimit(pageSize), then deletes by prefix.
  • API: staging/src/github.com/kcp-dev/sdk/apis/migration/v1alpha1/types_logicalclusterdump.go (create-only, non-persisted, spec.continue/limit/maxBytes).
  • Docs: docs/content/concepts/sharding/migration.md, pkg/reconciler/migration/logicalclustermigration/doc.go.

To investigate

  • Does EtcdRangeStream apply to the direct etcd client calls kcp makes here, or only to the kube-apiserver storage layer? Which etcd client/server versions does it need?
  • Could the dump scan on the origin and the cleanup scan use it, reducing round trips and memory?
  • Could LogicalClusterDump be streamed (chunked response or watch-like) instead of continue pages? What would that mean for the v1alpha1 API, which is alpha but already has continue/limit/maxBytes?
  • Interaction with the transformers (decryption on dump, re-encryption on write), the lease handling in writeDumpEntries, and error/resume behaviour when a stream breaks mid-copy.
  • Behaviour when the gate is disabled, and a mixed-version shard setup (origin and destination on different versions).

Steps

  • Read the upstream EtcdRangeStream KEP/implementation and confirm what it covers
  • Decide whether it applies to the migration scan/copy/cleanup paths
  • Prototype the origin dump scan with range streaming and compare memory and latency on a large logical cluster
  • Evaluate a streaming LogicalClusterDump API and its compatibility impact
  • Write up the findings and, if worthwhile, follow-up implementation issues
  • Update docs/content/concepts/sharding/migration.md if behaviour changes

Related: PR #4514, comment #4514 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      New

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions