memcache deployment with dalli?

[josh-m-sharpe]

I'm working on building out a memcache cluster (in AWS) to be used by dalli clients. I'd like to make it highly available and resilient to server faults, restarts and scaling the cluster in size as necessary. All of which means the list of IPs is somewhat dynamic - but as best as I can tell Dalli requires a static set of IPs when it is initialized - which requires clients to restart when these IPs change.

First a general question: Does Dalli shard keys across its nodes? Does the same key usually get sent to the same cache server under general circumstances? Asking this because I would stick a load balancer in front a bunch of servers, but I suspect Dalli would think of that as a single server and then I wouldn't get the benefit of sharding. Maybe??

So if the answer to the above question is "No - load balancer Bad ™️" - then, how else should I deploy this HA cluster?

Thanks!

[josh-m-sharpe]

As an aside, I see that memcache's bleeding edge 1.6.14 has a proxy built in, but this problem must've been solved some way before that - and I suppose I'd like to avoid the bleeding edge for now

[petergoldstein]

So the short answer is:

1. Yes, Dalli requires a static set of hostnames/IPs when it starts
2. It shard across this set of hostnames/IPs
3. Should one or more of the hostnames/IPs become unavailable, it will reshuffle ~1/N of the sharded keys - see the description here
4. Hostnames/IPs cannot be added to existing clients

As you note, this means Dalli as it is today isn't a good fit for the HA situation you envision - long-lived Dalli clients, frequent scaling changes, etc. That's something I'm looking to fix at some point - allowing a more dynamic source of hosts - but it doesn't exist today.

Frankly, one of the reasons it doesn't exist today is that it doesn't tend to come up all that often in real systems. What is far more common is something like:

1. Memcached clusters scale infrequently - with servers dropping out occasionally, and even more occasionally seeing servers added to the cluster
2. Dalli clients are created frequently, as production app servers are restarted (because of deployment, etc.)

Happy to look at any PR or strategy you'd like to discuss for extending Dalli to handle this case.

[josh-m-sharpe]

Thank you for this explanation!