[improve][pip] PIP-444: Rate limit for deleting ledger to alleviate the zk pressure.

pip/pip-444.md

@@ -0,0 +1,55 @@
+
+# PIP-444: Rate limit for deleting ledger to alleviate the zk pressure.
+
+# Background knowledge
+
+Apache Pulsar uses Apache BookKeeper as its storage layer. 
+Each topic consists of multiple ledgers, which are created periodically based on the configuration.
+Each ledger corresponds to a ZooKeeper (ZK) node for metadata storage. When a ledger is deleted, the corresponding ZK node will also be deleted.
+Retention policies determine how long messages are retained in a topic. When the retention period is reached, the corresponding ledgers will be deleted.
+
+# Motivation
+
+When the retention of a large topic is reduced, a significant number of ledgers need to be deleted.
+Since this deletion operation is not rate-limited, it could results in ZooKeeper (ZK) latency of several minutes,
+which threatens the stability of the entire Pulsar cluster.
+
+# Goals
+
+Add rate limit feature for deleting ledgers.
+
+# Detailed Design
+
+Introduce a new configuration parameter:
+```
+    @FieldContext(
+            category = CATEGORY_STORAGE_ML,
+            doc = "Max number of concurrent requests for deleting ledgers at broker level"
+    )
+    private int managedLedgerDeleteMaxConcurrentRequests = 1000;
+```
+These parameters can be configured in the `broker.conf` file.
+
+When the `managedLedgerDeleteMaxConcurrentRequests` parameter is set to a value greater than 0,
+the concurrency of ledger deletions will be limited to the specified value.
+
+# Backward & Forward Compatibility
+
+Fully backward compatible and forward compatible.
+
+## Upgrade
+
+New feature is enabled by default with a rate limit of 1000 deletions per second and concurrency of 1.
+
+## Downgrade / Rollback
+
+To downgrade or rollback, simply revert the configuration changes in the `broker.conf` file and restart the broker.
+
+
+# Links
+
+<!--
+Updated afterwards
+-->
+* Mailing List discussion thread: https://lists.apache.org/thread/ppmvm4noowlrhr229zf65mc44xqhj8k1
+* Mailing List voting thread: https://lists.apache.org/thread/2dop6kyvrpt6ztz9pvylyl8o2syvh9kw