feat: graceful worker shutdown #2274
Closed
+257
−89
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
How to use the Graphite Merge QueueAdd the label merge-queue to this PR to add it to the merge queue. You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
Deploying rivet with
|
| Latest commit: |
580531f
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://0fcd18b8.rivet.pages.dev |
| Branch Preview URL: | https://03-26-feat-graceful-worker-s.rivet.pages.dev |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR implements graceful worker shutdown functionality across the workflow system, focusing on safe termination of running workflows and proper resource cleanup.
- Added stop channel and check_stop() method in
WorkflowCtxto enable interrupting workflows at any point during execution - Modified
ListenCtxto use immutable reference toWorkflowCtx, improving signal handling during shutdown - Added shutdown sequence in
worker.rsthat waits for running workflows with timeout and handles SIGINT/SIGTERM - Implemented parallel execution using
ctx.joinin pegboard client workflow, replacing sequential operations that had deadlock issues - Added max_retries=15 for drain/undrain pegboard client activities to handle transient failures during shutdown
15 file(s) reviewed, 7 comment(s)
Edit PR Review Bot Settings | Greptile
MasterPtato
force-pushed
the
03-26-feat_graceful_worker_shutdown
branch
from
d104e87 to
f6a4e70
Compare
MasterPtato
force-pushed
the
03-25-feat_workflows_defererred_sqlite_tasks
branch
from
45c6cc6 to
ceec131
Compare
MasterPtato
force-pushed
the
03-26-feat_graceful_worker_shutdown
branch
from
f6a4e70 to
eec0c12
Compare
MasterPtato
force-pushed
the
03-25-feat_workflows_defererred_sqlite_tasks
branch
from
ceec131 to
4739210
Compare
MasterPtato
force-pushed
the
03-26-feat_graceful_worker_shutdown
branch
from
eec0c12 to
9dcc276
Compare
MasterPtato
force-pushed
the
03-25-feat_workflows_defererred_sqlite_tasks
branch
from
4739210 to
023537c
Compare
MasterPtato
force-pushed
the
03-26-feat_graceful_worker_shutdown
branch
from
9dcc276 to
580531f
Compare
Deploying rivet-hub with
|
| Latest commit: |
580531f
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://eb1573f9.rivet-hub-7jb.pages.dev |
| Branch Preview URL: | https://03-26-feat-graceful-worker-s.rivet-hub-7jb.pages.dev |
Merge activity
|
graphite-app bot
pushed a commit
that referenced
this pull request
Mar 28, 2025
<!-- Please make sure there is an issue that this PR is correlated to. --> Fixes RVT-4594 ## Changes <!-- If there are frontend changes, please include screenshots. -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes RVT-4594
Changes