Do not remove ejected servers from the hash ring

[ethaniel]

Is your feature request related to a problem? Please describe.

I have 3 redis servers. When one of them gets ejected, then twemproxy saves the data on other two remaining servers as per: Note that an ejected server will not be included in the hash ring for any requests until the retry timeout passes. This will lead to data partitioning as keys originally on the ejected server will now be written to a server still in the pool.

This can lead to data inconsistency in a case where the server returns to the ring for a short period and then gets ejected again, making the client read stale cached data from the second server.

Here is an example:

1. Client issues "SET key value" - set on server 1.
2. Server 1 gets ejected.
3. Client issues "GET key value" - since server 1 is ejected, it would be read from server 2.
4. Client issues "SET key value" - since server 1 is ejected, it would be set on server 2
5. Server 1 returns.
6. Client issues "GET key value" - read from server 1.
7. Client issues "SET key value" - set on server 1.
8. Server 1 gets ejected.
9. Client issues "GET key value" - stale data is read from server 2.

Describe the solution you'd like

Please add an option for twemproxy to immediately reply with NULL (or something similar) when there are operations on an ejected server instead of removing it from the hashring.

Describe alternatives you've considered
Not possible.

[TysonAndre]

See #608 - support for redis sentinel is planned in an upcoming 0.6 release which may improve uptime.

Also, by setting auto_eject_hosts: false, you can avoid changing the ring, though twemproxy will attempt to reconnect - the "heartbeat" patches planned for 0.6.0 would reduce the impact of this on the clients of the application