[fix][broker]If there is a deadlock in the service, the probe should return a failure because the service may be unavailable #10
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…return a failure because the service may be unavailable
15 tasks
…return a failure because the service may be unavailable
…e should return a failure because the service may be unavailable
…e should return a failure because the service may be unavailable
…e should return a failure because the service may be unavailable
…e should return a failure because the service may be unavailable.
…e should return a failure because the service may be unavailable.
…e should return a failure because the service may be unavailable. Add lastPrintThreadDumpTimestamp field to control the interval time for printing complete thread stack information.
…e should return a failure because the service may be unavailable. Add unit testing code.
…e should return a failure because the service may be unavailable. Add unit testing code.
…e should return a failure because the service may be unavailable. Add unit testing code.
…luster migration (apache#23767) Co-authored-by: ruihongzhou <[email protected]>
…e should return a failure because the service may be unavailable. Add unit testing code, shutdown deadlock thread.
…e should return a failure because the service may be unavailable. Add unit testing code, shutdown deadlock thread.
…pache#23718) Co-authored-by: anurag.reddy <[email protected]>
…e should return a failure because the service may be unavailable. Modify unit testing code, use org.mockito.Mockito replaces MockServletContext.
…hes (apache#23772) ### Motivation **Background** There is a mechanism that repeatedly prevents the callback of ML data ledger creation: - Start a scheduled task to check whether the creation will be timeout. - Received a callback - Check whether the future(`@param ctx` of `BK.createAsync`) has been done or not. - If done: it means the creation has timeout before the creation is completed - Otherwise: it is a real callback from BK. **Issue:** But the timeout event will call the same callback as above, then the steps are as follows, which you ca reproduce by the test `testCreateDataLedgerTimeout`: - Start creating a data ledger - Call `BK.createAsync` - Timeout - Mark the future(`@param ctx` of `BK.createAsync`) as completed exceptionally. - Trigger the callback related to ledger creation. - Check whether the future(`@param ctx` of `BK.createAsync`) has been done or not. - If done: do nothing. - Creation is compelled. - Trigger the callback related to ledger creation. - Check whether the future(`@param ctx` of `BK.createAsync`) has been done or not. - If done: do nothing. - Issue: The callback for ledger creation will never be called.   ### Modifications Fix the issue ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> ### Matching PR in forked repository PR in forked repository: x
Signed-off-by: Zixuan Liu <[email protected]>
…oo large to persist (apache#23652)
…nSendAcknowledgement (apache#23791)
Signed-off-by: Zixuan Liu <[email protected]>
…one fails (apache#23797) Signed-off-by: Zixuan Liu <[email protected]>
…ased naming scheme (apache#23620) Co-authored-by: zjxxzjwang <[email protected]>
…centlyJoinedConsumers are out-of-order (apache#23795)
Signed-off-by: Zixuan Liu <[email protected]>
…ement (apache#23600) Co-authored-by: Lari Hotari <[email protected]>
…hile using reverse proxy (apache#23974)
…metheus 3.x compatibility (apache#24060)
…ead letter topic producer (apache#24020) Co-authored-by: Lari Hotari <[email protected]>
…acklogQuotaDefaultLimitBytes (apache#24066) Co-authored-by: nuolin <[email protected]>
…sage metadata contains the same key but with different values (apache#23927)
…mentPartitionsTest (apache#24056)
…for Pulsar Functions&Sources (apache#23860)
…ecks and parameter reuse (apache#24064)
…udes the topic domain. (apache#24072)
…s be 0 (apache#24067) Co-authored-by: 张浩 <[email protected]>
…ded storage by the write quorum (apache#24054)
Co-authored-by: maruimin <[email protected]> Co-authored-by: crossoverJie <[email protected]>
…ption level dispatch rate policy (apache#24048)
… so that the producer gets created eventually (apache#24081)
…ot set in publishing (apache#24087)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…return a failure because the service may be unavailable
Fixes #xyz
Main Issue: #xyz
PIP: #xyz
Motivation
In some special scenarios, when the broker service has a deadlock, it needs to be able to automatically recover instead of requiring manual intervention. For example, when the service is deployed in a customer environment, we cannot directly manage it. If the service has a deadlock, the probe should return a failure because the service may be unavailable. The probe failure triggers a node restart to resolve the deadlock.
Modifications
Add deadlock detection in the probe. If a deadlock exists, print the thread stack and return a service unavailable exception
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
Documentation
docdoc-requireddoc-not-neededdoc-completeMatching PR in forked repository
PR in forked repository: