Skip to content

Commit

Permalink
Adjusting disconnectedBehavior Option to Prevent Timeout During Redis…
Browse files Browse the repository at this point in the history
… Shutdown

We often need to shut down Redis for maintenance, such as version upgrades. During these times, requests do not fail immediately but instead experience timeouts, increasing application latency. This issue can be resolved by adjusting some options.

Currently, our Redis client options are configured as follows:

```java
ClientOptions options = ClientOptions.builder()
.autoReconnect(true)
.disconnectedBehavior(DisconnectedBehavior.REJECT_COMMANDS)
```

The DisconnectedBehavior.REJECT_COMMANDS option appears to cancel commands when the connection is lost. However, if autoReconnect is not set to false, commands in the CommandHandler.stack are not canceled but are placed into the disconnectedBuffer. Therefore, ongoing commands are not rejected if autoReconnect is true, even with the client option modified.

For services heavily relying on Redis, latency is crucial. Additionally, we want to avoid writing custom code for reconnections by using the auto-reconnect feature. Adjusting the autoReconnect option can solve this issue immediately, but it would require significant changes to implement automatic reconnection.

Proposal

We propose that the condition for canceling commands in the CommandHandler.stack should be based solely on rejectCommandsWhileDisconnected and not combined with autoReconnect.

I've submitted a PR with a simple code change. Please let me know your thoughts.

===

Additionally our applications, to maintain low latency and avoid queuing many requests in the buffer when the connection is down, we have configured the client to execute commands only when the connection is active. As a result, only a few requests in the CommandHandler.stack encounter timeouts, but this still negatively impacts the application.
  • Loading branch information
하원호 committed May 29, 2024
1 parent 522abfd commit 21f6f5c
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -678,7 +678,7 @@ public void notifyDrainQueuedCommands(HasQueuedCommands queuedCommands) {
cancelCommands("Connection closed", queuedCommands.drainQueue(), it -> it.completeExceptionally(lazy.get()));
cancelCommands("Connection closed", drainCommands(), it -> it.completeExceptionally(lazy.get()));
return;
} else if (reliability == Reliability.AT_MOST_ONCE && rejectCommandsWhileDisconnected) {
} else if (rejectCommandsWhileDisconnected) {

Lazy<RedisException> lazy = Lazy.of(() -> new RedisException("Connection disconnected"));
cancelCommands("Connection disconnected", queuedCommands.drainQueue(), it -> it.completeExceptionally(lazy.get()));
Expand Down

0 comments on commit 21f6f5c

Please sign in to comment.