Skip to content

Commit

Permalink
Reset state to READY before completing futures
Browse files Browse the repository at this point in the history
This should fix a race condition where we:

1. say we're ready for new work by completing the predict future
2. this updates healthcheck status by setting runner back to "not busy"
3. we haven't updated worker state to READY yet, so the next prediction
   fails
  • Loading branch information
nickstenning committed Aug 14, 2024
1 parent aa6c095 commit 9563ef1
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions python/cog/server/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -185,9 +185,13 @@ def _consume_events_inner(self) -> None:
return

assert self._result
self._result.set_result(done)

# We capture the setup future and then set state to READY before
# completing it, so that we can immediately accept work.
result = self._result
self._result = None
self._state = WorkerState.READY
result.set_result(done)

# Predictions
while self._child.is_alive():
Expand All @@ -213,12 +217,15 @@ def _consume_events_inner(self) -> None:
if not done:
break

# We capture the predict future and then reset state before
# completing it, so that we can immediately accept work.
result = self._result
self._predict_payload = None
self._predict_start.clear()
self._result.set_result(done)
self._result = None
self._state = WorkerState.READY
self._allow_cancel = False
result.set_result(done)

# If we dropped off the end off the end of the loop, it's because the
# child process died.
Expand Down

0 comments on commit 9563ef1

Please sign in to comment.