[CORE-15110] `proto/admin`: Add `CloudStorageStatusService` #29148

WillemKauf · 2026-01-05T15:46:31Z

A port of the existing shadow_indexing.json API to Protobuf for our AdminV2 migration project, performed 99% by Claude.

Only the Protobuf spec is committed here, handler implementation will follow after the spec is accepted.

Human thoughts upfront

Claude was pretty good at this, I thoroughly reviewed the changes and didn't find anything lost in translation between the Swagger spec and the Protobuf spec.

Claude commit message

Port all cloud storage admin endpoints from shadow_indexing.json to protobuf-based admin API v2. Includes 12 RPCs:

Cluster and topic recovery operations

Partition status and manifest retrieval

Lifecycle marker management

Cache trimming and anomaly detection

Metadata reset operations

All messages include comprehensive field documentation for OpenAPI spec generation.

Claude session output, in Markdown

https://gist.github.com/WillemKauf/cf785b135c2aa581a4ebae5ad3b8ecce

Backports Required

Release Notes

none

Copilot

Pull request overview

This PR ports cloud storage admin endpoints from the existing Swagger-based API (shadow_indexing.json) to the new protobuf-based admin API v2. The implementation adds 12 RPCs covering cluster recovery, partition status retrieval, lifecycle marker management, cache operations, and metadata operations.

Key Changes:

Added comprehensive protobuf service definition for cloud storage status operations
All 12 RPCs include detailed field documentation for OpenAPI spec generation
Integrated with existing common protobuf types (e.g., TopicPartition)

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
`proto/redpanda/core/admin/v2/cloud_storage_status.proto`	New protobuf service definition with 12 RPCs and 30+ message types for cloud storage operations
`proto/redpanda/core/admin/v2/BUILD`	Bazel build configuration for the new proto library and its dependencies

Port all cloud storage admin endpoints from shadow_indexing.json to protobuf-based admin API v2. Includes 12 RPCs: - Cluster and topic recovery operations - Partition status and manifest retrieval - Lifecycle marker management - Cache trimming and anomaly detection - Metadata reset operations All messages include comprehensive field documentation for OpenAPI spec generation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

dotnwat

is it intentional that the service implemetnation isn't included in the PR?

WillemKauf · 2026-01-05T17:28:56Z

is it intentional that the service implemetnation isn't included in the PR?

Yes, we plan to propose the protos first, then implement the handlers for migration.

https://redpandadata.atlassian.net/browse/CORE-15102

michael-redpanda · 2026-01-05T19:22:54Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

@@ -0,0 +1,493 @@
+// Copyright 2025 Redpanda Data, Inc.


chore: year

michael-redpanda · 2026-01-05T19:25:22Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

+    // Retention milliseconds for recovered topics.
+    optional int64 retention_ms = 3;


nit/question: Why not a google.protobuf.Duration for time based values?

ah n/m it replaces a topic property

michael-redpanda · 2026-01-05T19:26:44Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

+// TopicRecoveryStatus contains the status of a topic recovery operation.
+message TopicRecoveryStatus {
+    // Current state of topic recovery.
+    string state = 1;
+    // Download counts per topic.
+    repeated TopicDownloadCounts topic_download_counts = 2;
+    // Original recovery request parameters.
+    optional RecoveryRequestParams request = 3;
+}


question: Why is this its own message when it's only used in QueryAutomatedRecoveryResponse?

michael-redpanda · 2026-01-05T19:27:25Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

+    // Original recovery request parameters.
+    optional RecoveryRequestParams request = 3;


Why is this optional in the response? Should it always be provided? When is it or is not provided?

michael-redpanda · 2026-01-05T19:30:48Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

Overall:

Can we do a little reorganization of the file, a la AIP-191

I wonder if we shouldn't do a 1:1 conversion of v1 endpoints to v2 and instead treat topics/partitions as resources to be operated on.

Have some sort of Partition or Topic service that provides different services to control the ntp?

@rockwotj not sure if you have thoughts on this?

I wonder if we shouldn't do a 1:1 conversion of v1 endpoints to v2 and instead treat topics/partitions as resources to be operated on.

Appreciate the thorough review, I tasked Claude with a direct port of the existing endpoint, but I'm happy to make whatever improvements we would benefit from and include them in this migration. And obviously carry forward those improvements/learnings to future migrations.

We (well I) do not want a direct port. Let me port a couple as an example when I am back and we can go from there.

We (well I) do not want a direct port.

thanks, this is helpful context for these ports that I didn't realize we were also cleaning them up but +1.

michael-redpanda · 2026-01-05T20:12:51Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

+message PartitionCloudStorageStatus {
+    // The partition's cloud storage mode (one of: disabled, write_only,
+    // read_only, full and read_replica).
+    string cloud_storage_mode = 1;


michael-redpanda · 2026-01-05T20:13:06Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

+    // Delta in milliseconds since the last upload of the partition's manifest.
+    optional int64 ms_since_last_manifest_upload = 2;
+    // Delta in milliseconds since the last segment upload for the partition.
+    optional int64 ms_since_last_segment_upload = 3;
+    // Delta in milliseconds since the last manifest sync (only present for read
+    // replicas).
+    optional int64 ms_since_last_manifest_sync = 4;


michael-redpanda · 2026-01-05T20:15:41Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

+    // Lifecycle status.
+    string status = 4;


michael-redpanda · 2026-01-05T20:16:24Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

+    // DeleteCloudStorageLifecycle
+    //
+    // Forcibly drop a lifecycle marker for a topic, this may leave data behind
+    // in the tiered storage bucket.
+    rpc DeleteCloudStorageLifecycle(DeleteCloudStorageLifecycleRequest)
+        returns (DeleteCloudStorageLifecycleResponse) {
+        option (pbgen.rpc) = {
+            authz: SUPERUSER
+        };
+    }


should this be treated internal and not documented?

michael-redpanda · 2026-01-05T20:26:02Z

proto/redpanda/core/admin/v2/cloud_storage_status.proto

+    // Type of anomaly.
+    string type = 1;


cleaning up service api too

Copilot AI review requested due to automatic review settings January 5, 2026 15:46

WillemKauf requested review from a team, michael-redpanda and rockwotj as code owners January 5, 2026 15:46

github-actions bot added the area/build label Jan 5, 2026

Copilot AI reviewed Jan 5, 2026

View reviewed changes

WillemKauf force-pushed the cloud_storage_api_proto branch from bec1e41 to 79eb957 Compare January 5, 2026 15:49

WillemKauf changed the title ~~proto/admin: Add CloudStorageStatusService~~ [CORE-15110] proto/admin: Add CloudStorageStatusService Jan 5, 2026

dotnwat reviewed Jan 5, 2026

View reviewed changes

dotnwat previously approved these changes Jan 5, 2026

View reviewed changes

michael-redpanda reviewed Jan 5, 2026

View reviewed changes

		// Retention milliseconds for recovered topics.
		optional int64 retention_ms = 3;

		// Original recovery request parameters.
		optional RecoveryRequestParams request = 3;

[CORE-15110] proto/admin: Add CloudStorageStatusService #29148

Are you sure you want to change the base?

[CORE-15110] proto/admin: Add CloudStorageStatusService #29148

Uh oh!

Conversation

WillemKauf commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Human thoughts upfront

Claude commit message

Claude session output, in Markdown

Backports Required

Release Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

dotnwat left a comment

Choose a reason for hiding this comment

Uh oh!

WillemKauf commented Jan 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[CORE-15110] `proto/admin`: Add `CloudStorageStatusService` #29148

[CORE-15110] `proto/admin`: Add `CloudStorageStatusService` #29148

WillemKauf commented Jan 5, 2026 •

edited

Loading