Skip to content

Conversation

aramissennyeydd
Copy link
Contributor

Summary

We've been seeing intermittent CI failures caused by messages like,

==[ e (build) ]==================================================[ 10 of 10 ]==
node:events:496
      throw er; // Unhandled 'error' event
      ^

SocketClosedUnexpectedlyError: Socket closed unexpectedly

We know that the underlying library @redis/client handles retries but it looks like there is no .on('error' handler to listen for retryable errors so it is crashing the process. This PR addresses that by adding an .on('error' handler and configuring the retry strategy to give up after a couple of failed connection attempts (I chose 5 as a good number). This also necessitated bumping the package version to latest.

Details

Added an 'error' handler to prevent the entire process from crashing due to unhandled errors. This is hidden in the docs, https://github.com/redis/node-redis#events but is definitely causing issues for us.

How it was tested

Ran the rush-redis-cobuild-integration-test locally and docker-compose down-ed the redis server halfway through a run. Before this change, the Rush process would immediately crash - with this change, if you docker-compose up -d again within the retryStrategy timeout, the process continues until success. If you keep the redis server off, the process will die with a message like,

--[ FAILURE: b (build) ]-------------------------------------[ 2.09 seconds ]--

Invoking: node ../build.js b 
start b
done
Failed to set completed_state(cobuild:completed:foo:56ea08cd70a55cacbb53405171864ebd5cf2ad0b)_package(b)_phase(_phase:build): 


Operations failed.

rush cobuild (11.97 seconds)

ERROR: Failed to disconnect to redis server: The client is closed

Impacted documentation

@github-project-automation github-project-automation bot moved this to Needs triage in Bug Triage Aug 27, 2025
@aramissennyeydd aramissennyeydd changed the title fix(redis): treat intermittent errors as warnings fix(redis): treat ignorable Redis errors as warnings Aug 27, 2025
@iclanton iclanton moved this from Needs triage to In Progress in Bug Triage Aug 27, 2025
aramissennyeydd and others added 8 commits September 2, 2025 09:03
Signed-off-by: Aramis Sennyey <[email protected]>
Signed-off-by: Aramis Sennyey <[email protected]>
Signed-off-by: Aramis Sennyey <[email protected]>
Signed-off-by: Aramis Sennyey <[email protected]>
Signed-off-by: Aramis Sennyey <[email protected]>
@aramissennyeydd aramissennyeydd force-pushed the sennyeya/fix-redis-error branch from 2c7aaf5 to 85d78ef Compare September 2, 2025 13:04
@aramissennyeydd
Copy link
Contributor Author

@iclanton Is this good to merge?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

2 participants