[fix][broker] Fix repeatedly acquired pending reads quota

[poorbarcode]

Motivation

Background

• Broker will limit the pending reading to Bookies if enabled managedLedgerMaxReadsInFlightSize
• [managed-ledger] Do not send duplicate reads to BK/offloaders #17241 introduced a mechanism that merges BK reading requests if we can, for example:
  • Request-1: Read entry 50~59
  • Request-2: Read entry 50~69
  • The Request-2 will wait for Request-1, and only send a real request that reads 60~69 after Request-1 is finished.

Issue

• The Request-2 above requests repeatedly
  • It requests 50~70 when creating.
  • It requests another range 61~70 after Request-1 is finished.
• Expected: Request-2 acquire 20 quota.
• Exact: Request-2 acquire 40 quota.
• You can call testPreciseLimitation to reproduce the issue

Modifications

• Fix the issue.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[lhotari]

I agree that this issue exists, however there are broader issues in the managedLedgerMaxReadsInFlightSizeInMB solution.

Sharing some context about the issues I have found and what I have currently in progress:

I have reported issues #23482, #23504, #23505 and #23506 .
These issues are related to the broker cache and managedLedgerMaxReadsInFlightSizeInMB and dispatcherMaxReadSizeBytes limits.

I have already changes to address these issues in an experimental branch, pending the submission of individual PRs. Addressing the lack of caching for replay queue messages requires broader changes to the the broker cache and those will be covered with a new PIP.

Some earlier details shared in this comment: #23524 (comment) .

Due to issues #23482 and #23506, I don't think that this PR would resolve problems alone. Regarding broker OOMEs, issue #23504 would also need to be resolved. It's possible that dispatcherDispatchMessagesInSubscriptionThread=false and managedLedgerReadEntryTimeoutSeconds=15 (some reasonable value) could be a mitigation to some issues.

[poorbarcode]

@lhotari

I have already changes to address these issues in an experimental branch, pending the submission of individual PRs. Addressing the lack of caching for replay queue messages requires broader changes to the the broker cache and those will be covered with a new PIP.

Some earlier details shared in this comment: #23524 (comment) .

Due to issues #23482 and #23506, I don't think that this PR would resolve problems alone. Regarding broker OOMEs, issue #23504 would also need to be resolved. It's possible that dispatcherDispatchMessagesInSubscriptionThread=false and managedLedgerReadEntryTimeoutSeconds=15 (some reasonable value) could be a mitigation to some issues.

Agree with you, let us fix them one by one, which is easier to review, and we should add tests for each case.

[heesung-sn]

I think we should use AsyncTokenBucket or Guava.RateLimiter.tryAcquire for this rate limiter.

IMHO, the current logic, requiring acquiredPermit to release is error-prone, when the caller forgets to release it. Instead, I think we better use token bucket-based one, which can automatically fill the bucket.

[lhotari]

I think we should use AsyncTokenBucket or Guava.RateLimiter.tryAcquire for this rate limiter.

IMHO, the current logic, requiring acquiredPermit to release is error-prone, when the caller forgets to release it. Instead, I think we better use token bucket-based one, which can automatically fill the bucket.

@heesung-sn I agree, the current solution is problematic. However, a token bucket isn't most optimal for this use case. I have a redesigned solution the managedLedgerMaxReadsInFlightSize solution in my experimental branch (example). There are a lot of different changes in branch so it might be hard to see what it's about. I'll extract that into a clear PR when time comes.
There's first #23892 and #23894 which need reviews so that I could make further progress. Do you have a chance to review them?

[lhotari]

LGTM

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.24%. Comparing base (bbc6224) to head (977b33a).
Report is 871 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23869      +/-   ##
============================================
+ Coverage     73.57%   74.24%   +0.66%     
+ Complexity    32624    32222     -402     
============================================
  Files          1877     1853      -24     
  Lines        139502   143618    +4116     
  Branches      15299    16310    +1011     
============================================
+ Hits         102638   106622    +3984     
+ Misses        28908    28610     -298     
- Partials       7956     8386     +430     

Flag	Coverage Δ
inttests	26.78% <60.00%> (+2.19%)	⬆️
systests	23.16% <60.00%> (-1.16%)	⬇️
unittests	73.75% <100.00%> (+0.90%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...keeper/mledger/impl/cache/PendingReadsManager.java	86.80% <ø> (+0.14%)	⬆️
...keeper/mledger/impl/cache/RangeEntryCacheImpl.java	63.48% <100.00%> (+4.73%)	⬆️

... and 1025 files with indirect coverage changes

[lhotari]

I think we should use AsyncTokenBucket or Guava.RateLimiter.tryAcquire for this rate limiter.
IMHO, the current logic, requiring acquiredPermit to release is error-prone, when the caller forgets to release it. Instead, I think we better use token bucket-based one, which can automatically fill the bucket.

@heesung-sn I agree, the current solution is problematic. However, a token bucket isn't most optimal for this use case. I have a redesigned solution the managedLedgerMaxReadsInFlightSize solution in my experimental branch (example). There are a lot of different changes in branch so it might be hard to see what it's about. I'll extract that into a clear PR when time comes. There's first #23892 and #23894 which need reviews so that I could make further progress. Do you have a chance to review them?

@heesung-sn @poorbarcode To continue the improvements to address issues with managedLedgerMaxReadsInFlightSize, I have created #23901 which addresses the problems I have referred to earlier. Please review