client behaviour during dragonflydb switchover

[jurim76]

Dragonflydb setup

• kubernetes v1.33
• dragonfly operator v1.3.1
• two pod dragonflydb cluster, created via dragonfly CR
• one pod redis-0 is master, the second pod redis-1 is replica
• "redis" clusterIP service pointed to master pod

1. Client connects to "redis" service and keeps master pod IP in own pool.
2. Master pod fails.
3. Operator initiates switchover.
4. The replica pod became master, "redis" service endpoint updated with a new pod's IP
5. Client keeps old pod's IP (the pod is replica now) in own pool and tries to write to read-only replica.
6. Client get the error "Error: READONLY You can't write against a read only replica".
7. Client permanently tries to write to read-only replica again and again.

I did some research how to fix this issue.

• Use redis sentinel

1. Dragonfly doesn't support sentinel model. You need to install a legacy redis as a sentinel.
   This solution lead to over-engineering and complexity - you need to manage dragonfly and redis both (in this case it's better to use native redis operator with a sentinel support, like spotahome operator and drop dragonflydb)

• Use external proxy like haproxy

1. The haproxy multiply redis backends couldn't use the same port 6379, every redis backend should use own unique port - 6379, 6380, 6381 etc
2. Redis service should be configured as a headless service to reduce overhead (optional)
3. Client should configure these ports on own side for every redis instance (devs are not happy with this)

I don't like both solutions, really...
Perhaps it could be worth to add some push logic to df operator to kill client's session after the switchover, to force the client to reconnect?

[romange]

Belongs in https://github.com/dragonflydb/dragonfly-operator
also see dragonflydb/dragonfly-operator#436