Route randomly between master and replica nodes

[ayush-sarda]

I have currently configured my cluster with 3 master nodes, each having a replica. My current usecase is read heavy traffic and I face an issue where my replica nodes receive all of the traffic, if I set SendToReplicas as true. If I set them as false, my master nodes receive all the traffic. I have currently used a random function within SendToReplicas to decide between true or false. This does solve my current problem but I am planning to migrate to 3 replicas per master node. Since random function will decide if the traffic should be routed to replicas or master, load will still be uneven since there are 3 nodes when SendToReplicas return true and only one node when it returns false.
Can we add a functionality similar to go-redis' RouteRandomly config, to allow even distribution of traffic across primary and replica nodes?

[ayush-sarda]

If this can be incorporated, I can work on implementing this feature.

[rueian]

Hi @ayush-sarda, this could be a big change since we currently only keep one replica connection for a shard in memory.

Would you consider using client-side caching?

[ayush-sarda]

@rueian I have multiple workflows associated with redis and have already migrated much of them to use client side caching. There are certain flows which cannot be migrated, hence they contribute heavily towards the redis traffic.
I think alongside client side caching, route randomly could be a part of the library, when someone's workflow does not suppport local caching.
I understand the impact would be keeping all the connections on every replica, but that could be the tradeoff we have to work with, if we enable this flag. WDYT?

[rueian]

Ok, maybe you can try replacing rslots []conn with rslots [][]ReplicaInfo and adding a new private conn field to ReplicaInfo. This way we can be compatible with the existing ReplicaSelector function.

[rueian]

Hi @ayush-sarda, please let me know if you need some help on this.

[ayush-sarda]

Got busy with some different work. I'll make the changes and raise the PR.

[ayush-sarda]

@rueian I have noticed one more thing. For a given slot, we are fixing the connection which will be used for processing any operation for that slot. But if the traffic is heavily skewed towards certain slots, we would be seeing lot of pressure on certain nodes. right?

[rueian]

Hi @ayush-sarda, are you talking about connections or nodes?

if the traffic is heavily skewed towards certain slots, we would be seeing lot of pressure on certain nodes. right?

For nodes, this is unavoidable true for writes. In terms of reads, we currently can send reads to one replica of the slot picked by each topology refresh.

But we can have multiple connections to a node.

[ayush-sarda]

@rueian Please take a look at this PR. Yet to add test cases, but let me know if the approach looks good.

[ayush-sarda]

@rueian Meant nodes when I mentioned connection. Yes, we might have multiple connections to a node, but for a slot, traffic will eventually by routed to a single node, every time an operation (read) is called for that slot. Hence more pressure on few nodes, when the usecase heavily rely on a few slots.
For writes, it's understandable, but for reads, the traffic might not be routed evenly to all the replicas.