Skip to content

Add keep_alive() call when an error occurs in fetch_next(), to prevent permanent failure when redis restarts#492

Merged
geofmureithi merged 3 commits intoapalis-dev:mainfrom
zakstucke:main
Jan 15, 2025
Merged

Add keep_alive() call when an error occurs in fetch_next(), to prevent permanent failure when redis restarts#492
geofmureithi merged 3 commits intoapalis-dev:mainfrom
zakstucke:main

Conversation

@zakstucke
Copy link
Copy Markdown
Contributor

As per title, without this change when redis restarts, it seems apalis get's stuck repeating this error over and over:

WARN  An error occurred during streaming jobs: An error was signalled by the server - ResponseError: user_script:15: consumer not registered script: e3ac73ae44f02d04176688b08c26ed751417cdb0, on @user_script:15.

After this change, on error keep_alive() will re-register, so it's fixed on the next call.

@geofmureithi
Copy link
Copy Markdown
Member

I think this should only happen when one encounters a specific error, could you provide some logs or more context of what you are encountering?

@zakstucke
Copy link
Copy Markdown
Contributor Author

You can reproduce by:

  • Starting redis
  • Starting apalis job queue
  • send and process some jobs
  • Kill redis
  • Restart redis
  • Send another job
  • Apalis endlessly emits WARN An error occurred during streaming jobs: An error was signalled by the server - ResponseError: user_script:15: consumer not registered script: e3ac73ae44f02d04176688b08c26ed751417cdb0, on @user_script:15.

This error is coming from packages/apalis-redis/lua/get_jobs.lua line 15

By recalling keep_alive(), apalis can then recover and continue processing jobs, otherwise a single redis failure breaks apalis permanently

@geofmureithi geofmureithi self-requested a review January 11, 2025 09:04
Comment thread packages/apalis-redis/src/storage.rs Outdated
@zakstucke
Copy link
Copy Markdown
Contributor Author

@geofmureithi updated

@geofmureithi geofmureithi merged commit 132c581 into apalis-dev:main Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants