RedisConnectionException in StackExchange.Redis when using ElastiCache Cluster "No connection is available to service this operation"

[ParveenDotNet]

I am encountering an issue with the StackExchange.Redis library while using an ElastiCache cluster for caching. Below are the details:

StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: GET {41490_SessionContext}:yIXd8G+87efOjopAAYMRPm/MDYHMNxhhmpKphKnDpZQ=; IOCP: (Busy=0,Free=1000,Min=32,Max=1000), WORKER: (Busy=0,Free=32767,Min=32,Max=32767), Local-CPU: n/a at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultiplexer.cs:line 2194
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisBase.cs:line 54 at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisDatabase.cs:line 2251 at StackExchange.Redis.Extensions.Core.StackExchangeRedisCacheClient.Get[T](String key, CommandFlags flag)

Environment:

• StackExchange.Redis Version: 2.0.519
• StackExchange.Redis.Extensions.Binary Version: 4.0.5
• StackExchange.Redis.Extensions.Core Version: 4.0.5
• Redis Provider: AWS ElastiCache
• Application Details: We have multiple workflow jobs that create their own connections to Redis. The number of keys could be in the millions.

Issue Details:

The issue seems to occur when multiple workflows are trying to access Redis simultaneously. Here are a few observations:

• The error occurs intermittently but is frequent enough to affect the application's reliability.

• We have looked into the StackExchange code as well the line that is mentioned in the error is never expected to through error, the error can occur inside the function "TryPushMessageToBridge" while getting the Server
  

• To get the same exception we set Server to null then we got the same error, however the line number is not same
  at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultiplexer.cs:line 2197`
  which is as per the code is correct, but not matching with the exact stack trace we got on production.
  

• We have seen multiple errors related to "No connection is available to service this operation:" but they are different as they include some more information about the error like:

  • SocketClosed

  • It was not possible to connect to the redis server(s

    No connection is available to service this operation: SETEX {Testing}:A0; It was not possible to connect to the redis server(s). To create a disconnected multiplexer, disable AbortOnConnectFail. ConnectTimeout; IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=59,Free=1988,Min=8,Max=2047), Local-CPU: n/a

Request for Assistance:

• Diagnosis: Could you help diagnose the root cause of this issue? Are there any known issues or limitations with the library version we are using, especially in a high-concurrency environment with ElastiCache?

• Best Practices: What are the recommended best practices for using StackExchange.Redis in a scenario with high concurrency and a large number of keys? Are there specific configuration settings we should use?

• Potential Fixes: Are there any fixes or updates available that address this issue? Would upgrading to a newer version of StackExchange.Redis or its extensions help mitigate this problem?

[ParveenDotNet]

@mgravell Can you please help if possible?

[NickCraver]

This looks like a version from over 5 years ago, and we've made many, many enhancements since then around all these issues. Definitely upgrade to latest and see if you're still having issues. Newer releases have a lot more debug data in the logging to help us advise if so :)

[ParveenDotNet]

Thanks for the response @NickCraver

In our project, we have a limitation that we can only upgrade to version 2.1.58. Will upgrading to this be helpful?