- Overview
- Installation
- Module: memcached-utility
- Module: memcached-cubby
- Spring Integration
- Usage Recipes
- Operational Notes
- Relationship to
smallmind-persistence - Comparison With Other Java Memcached Clients
The smallmind-memcached module is a self-contained memcached stack for Java
services that want a narrow, predictable cache abstraction and/or a complete, production grade memcached client. It is organized as two publishable
artifacts:
-
memcached-utility— a tiny, dependency-light abstraction layer. Five classes: theProxyMemcachedClientSPI every caller in this project programs against, theProxyCASResponsepairing that carries a value and its CAS token, aMemcachedServervalue object that describes a single endpoint, anInMemoryMemcachedClientfor unit and integration tests, and a SpringFactoryBeanthat expands a server pattern into aMemcachedServer[]. -
memcached-cubby— a full, production-grade memcached client (the "Cubby" client) that implementsProxyMemcachedClientover a non-blocking NIO transport and the memcached meta protocol. Cubby owns everything from byte-level framing up to consistent-hash routing: pluggable codecs (with transparent GZIP for large payloads), pluggable key translators (with automatic hashing for oversized keys), two key-locator strategies (simple modulo and full Maglev consistent hashing), per-host connection pooling, SASL authentication, keep-alive probing, and a reconnect daemon that brings hosts back online without application restart.
Together the two modules give a caller a single, testable cache API and a real NIO client behind it. Swapping the in-memory implementation for a live cluster is a one-bean change.
Memcached is a simple key/value store, but clients must contend with a layer of scaffolding around that simplicity, resolving many of the same concerns:
-
A cross-client abstraction so the application code does not depend on a specific driver (
spymemcached,XMemcached,Folsom, …). That dependency creeps into method signatures, exception types, and mocking setups, and rotating it later is painful. -
A test double that behaves like memcached in-process so unit tests do not require a live server.
-
CAS (compare-and-swap) support that survives through the abstraction. Many drivers expose CAS as a first-class feature but the abstraction layers that wrap them typically do not.
-
A real client that actually respects the constraints of memcached in production: 250-byte key limit, 1MB value ceiling (mitigated by compression), hash-based routing that does not re-shard the entire cluster when one node goes away, automatic reconnection, and connection pooling for throughput.
-
A Spring wiring surface that turns
cache-1.internal,cache-2.internal,cache-3.internalinto a ready-to-use client with a single pattern string, not three parallel bean definitions.
smallmind-memcached solves (1) through (3) in memcached-utility and (4)
through (5) in memcached-cubby. Application code is written against
ProxyMemcachedClient; tests inject InMemoryMemcachedClient; production
injects CubbyMemcachedClient; nothing else changes.
-
Defines the one cache API —
ProxyMemcachedClient— that every memcached-dependent component in this codebase programs against.org.smallmind.persistence.cache.memcached.MemcachedCacheis the most notable consumer. -
Supplies an in-memory test double —
InMemoryMemcachedClient— that implements every operation on aHashMap, including TTL expiry and a monotonically issued CAS counter. Good for unit tests and local development where no memcached server is available. -
Speaks the memcached meta protocol. The Cubby client talks exclusively in
mg/ms/md/ma/mncommands rather than the legacy text protocol. This is a deliberate choice: the meta protocol folds get-and-touch, CAS reads, conditional writes, and arithmetic into a small set of flag-driven opcodes that are easier to compose and parse than the legacy surface. -
Runs on NIO selectors, one thread per host. Each open connection has its own selector loop, its own request queue, and its own response queue. Client threads enqueue requests and park on a callback; the selector thread drains the queue, writes the bytes, parses the response stream, and wakes the caller. There is no request/response multiplexing across connections because memcached answers in FIFO order per connection.
-
Routes keys with a consistent-hashing strategy — Maglev, by default in
OPTIMALmode. Adding or removing one host re-routes only the keys that host owned rather than reshuffling the entire keyspace. A simpler modulo-hash locator is available for deployments that want minimum complexity. -
Normalises keys through a
KeyTranslator. The default translator Base64-encodes the key so it cannot contain whitespace or control characters; a decorator translator falls back to a SHA3-512 hash when the encoded key would exceed memcached’s 250-character limit. -
Compresses large values through a decorator codec. Payloads above a configurable threshold (16 KiB by default) are GZIP-compressed and wrapped in a marker envelope so the decoding side knows to decompress; smaller payloads are passed through untouched.
-
Heals itself. When a host stops answering, the connection marks it inactive and the
ServerDefibrillatorthread probes it once per "resuscitation" interval (10s by default). As soon as the probe succeeds the connection is rebuilt, the host is marked active, and the routing table is refreshed. DNS changes (e.g. a load balancer rotating a backend IP) are picked up at probe time. -
Integrates with Spring.
MemcachedServerMapFactoryBeanexpandscache#.internal:11211+1-3into a map of three named servers.CubbyConfigurationFactoryBeanassembles a configuration from a named preset (DEFAULTorOPTIMAL) with per-property overrides.CubbyMemcachedClientFactoryBeanbuilds, starts, and stops the client with the Spring context.
-
It does not implement
flushAll/clearon the real client.CubbyMemcachedClient.clear()throwsUnsupportedOperationException. Clearing a memcached cluster is an operator action, not an application action; expressing it through a cache API tends to produce accidents.InMemoryMemcachedClient.clear()does clear, because the in-memory store is scoped to one JVM and is intended for tests. -
It does not implement a multi-get over many connections in parallel.
CubbyMemcachedClient.get(Collection<String>)iterates the keys and issues onegetper key. Memcached keys typically land on different hosts, so the caller gets correct routing automatically, but there is no pipelining or fan-out at this layer. -
It does not run a general-purpose worker pool. The only scheduled background work Cubby starts is the per-coordinator
ServerDefibrillator, which runs on a single-thread scheduled executor, and each connection has its own dedicated selector thread. There is no general worker pool to tune. -
It does not wrap
ProxyMemcachedClientin a retry or circuit-breaker layer. The abstraction is a thin pass-through. Policy behaviour belongs in a higher layer (e.g. the persistence cache). -
It does not negotiate a serialization format with the server. The codec chosen by the client is opaque to memcached — values go in and come out as
byte[]. Two clients sharing a cache must use compatible codecs, and the project’s default codec is JavaObjectOutputStream. A service that needs cross-language interop should plug in its ownCubbyCodec.
Depend on the module that matches your need.
<dependency>
<groupId>org.smallmind</groupId>
<artifactId>memcached-utility</artifactId>
<version>${smallmind.version}</version>
</dependency>
<dependency>
<groupId>org.smallmind</groupId>
<artifactId>memcached-cubby</artifactId>
<version>${smallmind.version}</version>
</dependency>| Artifact | Module name |
|---|---|
|
|
|
|
Library code that only needs the abstraction (ProxyMemcachedClient,
MemcachedServer, InMemoryMemcachedClient) can depend on
memcached-utility alone. Bringing in memcached-cubby also transitively
brings the utility jar.
memcached-utility ships an explicit module descriptor that exports
org.smallmind.memcached.utility and org.smallmind.memcached.utility.spring and
re-exports org.smallmind.nutsnbolts (requires transitive). Spring is optional:
requires static spring.beans, used only by the FactoryBean classes in the spring
package, which is opened to spring.core and spring.beans.
memcached-cubby ships an explicit module descriptor that exports all eight of its
packages and re-exports org.smallmind.memcached.utility and org.smallmind.nutsnbolts
(requires transitive). It also requires org.smallmind.scribe.pen and spring.beans. It
opens org.smallmind.memcached.cubby unconditionally, so
classpath:org/smallmind/memcached/cubby/memcached.xml stays loadable from other modules,
and opens org.smallmind.memcached.cubby.spring to spring.core and spring.beans. SipHash
is requires static io.whitfin.siphash (an automatic module whose name the jar declares).
See Module Path And Classpath for the repository-wide rules.
Cubby has one optional external dependency. If the Maglev key locator is
used, io.whitfin:siphash must be on the classpath:
<dependency>
<groupId>io.whitfin</groupId>
<artifactId>siphash</artifactId>
</dependency>The siphash dependency is declared <optional>true</optional> in Cubby’s
POM, so pulling it in is an explicit choice. The DefaultKeyLocator
(simple modulo hash) does not require it.
Five classes, no network code. This module exists so the rest of the
codebase (especially org.smallmind.persistence.cache.memcached) can be
compiled and unit-tested without dragging in a real memcached client.
The single SPI every cache consumer in this project programs against. Its shape was chosen to cover exactly the memcached operations the rest of the codebase actually uses: straight get/set/delete, CAS variants of the same, bulk get, touch, and get-and-touch.
public interface ProxyMemcachedClient {
long getDefaultTimeout();
<T> ProxyCASResponse<T> createCASResponse(long cas, T value);
<T> T get(String key) throws Exception;
<T> Map<String, T> get(Collection<String> keys) throws Exception;
<T> ProxyCASResponse<T> casGet(String key) throws Exception;
<T> boolean set(String key, int expiration, T value) throws Exception;
<T> boolean casSet(String key, int expiration, T value, long cas) throws Exception;
boolean delete(String key) throws Exception;
boolean casDelete(String key, long cas) throws Exception;
boolean touch(String key, int expiration) throws Exception;
<T> T getAndTouch(String key, int expiration) throws Exception;
void clear() throws Exception;
void shutdown() throws Exception;
}Every operation declares throws Exception. This is deliberate: real
implementations (notably Cubby) throw IOException,
CubbyOperationException, InterruptedException, and
ClassNotFoundException; the interface cannot import them all without
binding itself to a specific client. Callers either wrap and rethrow, or
propagate.
createCASResponse is a factory method on the client itself, not a
public constructor on a value class, because the CAS response type is
implementation-specific. A caller that wants to assemble a CAS response
(e.g. to stub out a mock) asks the client to build one.
The expiration parameter is a memcached TTL in seconds. 0 means the
entry has no expiry.
A value-and-token pairing.
public interface ProxyCASResponse<T> {
T getValue();
long getCas();
}CAS tokens are opaque long values assigned by the server (or by the
in-memory counter). Pass a token back to casSet or casDelete to turn
an unconditional write into an optimistic-lock write: the mutation
applies only if the stored token still matches, i.e. no other writer has
modified the entry since you read it.
Mutable value object with a no-arg constructor (for Spring property
injection) and a (host, port) constructor (for programmatic use).
MemcachedServer s = new MemcachedServer("cache-1.internal", 11211);
s.getHost(); // "cache-1.internal"
s.getPort(); // 11211Consumed by the factory beans in both modules to open connections.
An in-process implementation of ProxyMemcachedClient backed by a
HashMap<String, Holder<?>>. Every mutating method is synchronized so
the double is safe to use from concurrent tests.
Highlights:
-
CAS counter. A client-wide
AtomicLongissues a unique token on every store, mirroring the opaque token issued by a real server. -
Lazy expiry. An entry with a TTL is not evicted at the deadline; it is simply treated as absent on the next access. This matches real memcached behaviour closely enough for testing.
-
clear()actually clears the internal map, unlikeCubbyMemcachedClientwhich refuses the call. This is the pragmatic test convention: tests often want a clean slate between methods; real code should not issue a flush-all.
Typical test use:
@Test
public void cacheStoresAndReturns() throws Exception {
ProxyMemcachedClient client = new InMemoryMemcachedClient();
client.set("greeting", 0, "hello");
assertThat(client.<String>get("greeting")).isEqualTo("hello");
ProxyCASResponse<String> cas = client.casGet("greeting");
assertThat(client.casSet("greeting", 0, "world", cas.getCas())).isTrue();
assertThat(client.casSet("greeting", 0, "again", cas.getCas())).isFalse();
}The double is drop-in: any class that accepts ProxyMemcachedClient
works with it.
Immutable ProxyCASResponse produced by InMemoryMemcachedClient. No
surprises: it holds a value and a cas, exposed by getters.
Expands a pattern and a spread into a MemcachedServer[], for
applications that pass server arrays rather than maps. The pattern uses
# as a placeholder; the spread string is handed to
org.smallmind.nutsnbolts.util.Spread.calculate, so 1-3 produces
1, 2, 3; alpha|beta|gamma produces alpha, beta, gamma.
A pattern with no # is treated as a single server. Port is 11211
when the pattern does not include :port.
<bean id="memcachedServers"
class="org.smallmind.memcached.utility.spring.MemcachedServerFactoryBean">
<property name="serverPattern" value="cache-#.internal:11211"/>
<property name="serverSpread" value="1-3"/>
</bean>MemcachedServerMapFactoryBean (in the utility module, see below)
produces a Map<String, MemcachedServer> from the same syntax, which is
what Cubby’s client factory consumes.
The full client. Everything inside memcached-cubby exists to implement
ProxyMemcachedClient for a live cluster over NIO. The top-level
entry point is CubbyMemcachedClient; everything else is a supporting
role.
CubbyMemcachedClient
│
▼
ConnectionMultiplexer (1 instance)
│
│ distributes across N = connectionsPerHost
▼
ConnectionCoordinator × N
│
│ owns: ServerPool, KeyLocator, ServerDefibrillator
│ one CubbyConnection per host
▼
CubbyConnection (NIOCubbyConnection) (1 per host per coordinator)
│
│ selector thread + request queue + response queue
▼
memcached server
-
Client. Thin facade. Serializes values through the codec, builds a
Command, hands it to the multiplexer, unwraps theResult, returns. -
Multiplexer. Holds N coordinators, where N is
connectionsPerHost. Dispatches each command to a coordinator chosen uniformly at random viaThreadLocalRandom. The multiplexer is the shard that adds concurrency: one connection per host serialises requests on that host, so a single connection can bottleneck under load; with N coordinators there are N connections to each host and that many requests can be in flight simultaneously. -
Coordinator. Owns one logical set of connections to the cluster plus the machinery that keeps them healthy: a
ServerPoolofHostControlrecords, aKeyLocatorthat maintains a routing table, aServerDefibrillatorthat probes inactive hosts, and oneCubbyConnectionper host. -
Connection. Does the wire work: opens the socket, runs a selector loop, serialises commands, parses responses, dispatches responses to waiting callbacks, and notifies the coordinator if it dies so the routing table can drop the host.
Every per-host connection has its own dedicated selector thread. A command’s route through the system is:
-
Caller thread calls
client.set(key, expiration, value). -
Client serializes the value, wraps it in a
SetCommand, hands it to the multiplexer. -
Multiplexer picks a coordinator (random if N > 1) and forwards.
-
Coordinator asks the key locator for the target host, grabs the corresponding
CubbyConnection, forwards the command. -
Connection builds a
MissingLink(command + callback), puts it on the request queue, wakes the selector, then blocks on the callback. -
Selector thread drains the request queue, moves each
MissingLinkto the response queue, writes the command bytes. -
Server responds; selector thread reads bytes, parses a
Response, pops the oldestMissingLinkfrom the response queue, callssetResult(response)on its callback. -
Caller thread wakes up with the response, the command’s
processmethod turns it into aResult, the client deserializes the value out of the result.
A few ideas recur and are worth naming up front.
- Memcached meta protocol
-
The second-generation memcached text protocol, introduced to replace the legacy
get/gets/set/add/replace/cas/incr/decr/deletesurface. Commands are two letters:mg(meta get),ms(meta set),md(meta delete),ma(meta arithmetic),mn(meta noop). Behaviour is selected with single-character flags (v=return value,c=return CAS,C=match CAS,T=set TTL,M=mode,O=opaque token, …). Cubby speaks exclusively in this dialect. - Logical host name
-
The
namefield ofMemcachedHost, a stable identifier used throughout the routing layer. Different from the DNS hostname — Cubby uses the logical name in the hash ring so that DNS changes (load-balancer backend rotations) do not move keys. Host names must be unique within a pool. - CAS (compare-and-swap)
-
An opaque
longtoken returned by the server with every successful read and store. Pass it to a conditional write (casSet,casDelete) and the mutation applies only if the server’s current token for that key still matches; if someone else wrote in the meantime the mutation is rejected with anEXresponse code. CAS is the basis of every correct read-modify-write in memcached — there is no server-side locking. - Virtual host count
-
The per-host multiplier used by
MaglevKeyLocatorto size the lookup table. WithvirtualHostCount = 100(the default) a 3-host cluster builds a table ofnextPrime(300) = 307slots. A larger multiplier produces a more uniform distribution at the cost of more memory and a slowerinstallRouting. - Resuscitation interval
-
The number of seconds the
ServerDefibrillatorwaits between rounds of probing inactive hosts. Default 10s. Shorter intervals recover faster at the cost of more speculative TCP traffic to a flapping host. - Keep-alive interval
-
The number of idle seconds after which the selector thread injects an
mn(noop) command to keep the connection warm and prove it is still usable. Default 30s. - Opaque token
-
An arbitrary caller-supplied string attached to a command via
setOpaqueToken. The server echoes it back in the response’s token field. Useful for correlating pipelined responses on diagnostic paths; Cubby’s normal request/response loop does not need it because per-connection FIFO ordering already correlates them.
The public entry point. Implements ProxyMemcachedClient and adds a few
extra methods (send, casGetAndTouch) that expose Cubby-specific
capabilities without breaking the abstraction.
CubbyMemcachedClient client = new CubbyMemcachedClient(
CubbyConfiguration.OPTIMAL,
new MemcachedHost("0", "cache-1.internal", 11211),
new MemcachedHost("1", "cache-2.internal", 11211),
new MemcachedHost("2", "cache-3.internal", 11211));
client.start();
try {
client.set("user:42", 600, userRecord);
UserRecord loaded = client.get("user:42");
} finally {
client.stop();
}Lifecycle is explicit: the constructor only stores state, start()
opens connections and launches the defibrillator, stop() tears
everything down. Do not use the client between construction and
start(), or after stop() — both will fail with
CubbyOperationException("The connection has been stopped").
Operational methods:
| Method | Purpose |
|---|---|
|
Open or close every connection, manage the defibrillator daemon. |
|
Fetch one value, or iterate fetches for a set of keys. The bulk form is a convenience, not a pipelined multi-get. |
|
Fetch with CAS token, optionally refreshing the TTL in the same
round trip. Returns |
|
Unconditional store, replaces any existing value. |
|
Optimistic-lock store. |
|
Remove a key, optionally guarded by CAS. |
|
Refresh TTL without fetching the value. |
|
Fetch the value and refresh its TTL atomically. |
|
Low-level escape hatch: send an already-built |
|
Delegates to |
|
Throws |
The expiration argument is memcached’s native "expiration time" value.
0 means the entry never expires.
A mutable bean that bundles every tunable knob. Setters are fluent
(return this). Two pre-built instances are exposed as constants:
| Constant | Contents |
|---|---|
|
|
|
|
In most real deployments you want OPTIMAL:
CubbyConfiguration configuration = CubbyConfiguration.OPTIMAL
.setAuthentication(new Authentication("cacheuser", "cachepass"))
.setConnectionsPerHost(4)
.setDefaultRequestTimeoutMilliseconds(500);A note on the two constants: they are static singletons, so mutating
them with a setter mutates the shared instance. If you need two clients
with different configurations, build your own new CubbyConfiguration()
per client.
Properties:
| Property | Default | Meaning |
|---|---|---|
|
|
Serializes / deserializes values. See Codecs. |
|
|
Strategy for choosing which host owns a key. See Key locators. |
|
|
Applied to keys before hashing and transmission. See Key translators. |
|
|
SASL credentials; |
|
|
Maximum time a |
|
|
TCP connect-establishment timeout. |
|
|
Socket read timeout used by the defibrillator probe. |
|
|
Idle seconds before a keep-alive NOOP is injected. |
|
|
Interval between defibrillator rounds. |
|
|
Number of parallel connections to each host. Increase for throughput under contention. |
|
|
Seconds |
A host descriptor: a logical name, a DNS hostname, and a port.
new MemcachedHost("shard-a", "cache-1.internal", 11211);The logical name is what the key locator and routing table use as the
stable identifier. Adding DNS rotation behind a single logical name
keeps keys pinned to that host even as the IP changes: the defibrillator
re-resolves on every probe and calls regenerate(InetSocketAddress)
before reconnecting.
Two logical names on the same IP are treated as two distinct hosts by the locator, which can be useful if you want oversubscribed shards.
Immutable (username, password) pair.
configuration.setAuthentication(new Authentication("user", "secret"));When set, the connection layer dispatches an AuthenticationCommand as
its first act on every new socket. Credentials are transmitted in plain
text, reusing the ms opcode with a placeholder key; the server is
expected to terminate the connection if they are wrong. This matches
the reference memcached SASL PLAIN behaviour.
CASValue<T> is Cubby’s implementation of ProxyCASResponse<T>:
immutable pairing of a decoded value and its CAS token. Returned by
casGet / casGetAndTouch or constructed via
client.createCASResponse(cas, value) when you need to build one by
hand.
All Cubby-level exceptions extend formatted-exception base classes from
smallmind-nutsnbolts, so they accept printf-style message arguments.
| Exception | Raised when |
|---|---|
|
A generic recoverable error: command cannot be routed, connection is stopped, authentication fails, etc. Parent of several specific subtypes. |
|
Opening a TCP connection to a host did not complete within
|
|
A response did not arrive within the configured per-request timeout. |
|
The NIO selector reported the socket key as invalid three times in a row; the connection is dead. |
|
The key locator has no active host to route to, i.e. every server in the pool is marked inactive. |
|
The peer closed the TCP connection. |
|
The server returned |
|
The response parser encountered bytes that do not match any known meta-protocol code. |
|
A |
The CubbyCodec interface has two methods:
public interface CubbyCodec {
byte[] serialize(Object obj) throws IOException;
Object deserialize(byte[] bytes) throws IOException, ClassNotFoundException;
}The client converts every set argument to bytes with serialize and
every get payload back with deserialize. Two implementations ship:
Default. Uses ObjectOutputStream / ObjectInputStream to encode
Serializable values. Deserialization falls back to the thread context
class loader when the default resolver cannot find a class; this makes
the codec work in OSGi, servlet containers, and other environments with
split classloader hierarchies.
Trade-offs:
-
Cross-JVM compatible as long as both sides agree on the class’s
serialVersionUID. -
Not interoperable with non-Java callers.
-
Refuses to serialize
null:serialize(null)throwsNullPointerException. If a caller needs to cache anullresult, store a sentinel value or represent absence as key absence.
Decorator. Serializes through the delegate codec, and if the result is
at least compressionThreshold bytes (16 KiB by default), GZIP-compresses
the bytes and re-serializes them inside a GzipObjectWrapper. The
deserialize path looks for the wrapper and decompresses when present.
CubbyCodec codec = new LargeValueCompressingCodec(
new ObjectStreamCubbyCodec(),
8192); // custom threshold
configuration.setCodec(codec);The threshold exists so small values are not paying a GZIP overhead round-trip for no benefit. 16 KiB is an empirically reasonable default; tune down if you know every value is large, tune up if most values are tiny.
Compression is the main mitigation for memcached’s 1 MB value-size limit. A 900 KB JSON document that compresses to 120 KB fits comfortably; an incompressible 2 MB binary will not.
Implementing CubbyCodec takes roughly ten lines. A JSON codec on top
of Jackson:
public class JsonCubbyCodec implements CubbyCodec {
private final ObjectMapper mapper = new ObjectMapper();
@Override
public byte[] serialize(Object obj) throws IOException {
return mapper.writeValueAsBytes(obj);
}
@Override
public Object deserialize(byte[] bytes) throws IOException {
return mapper.readValue(bytes, Object.class);
}
}Wrap it in LargeValueCompressingCodec if you want compression for
free.
Memcached keys must be printable ASCII without whitespace or control
characters, and must not exceed 250 bytes. The KeyTranslator
interface hides all that from the application layer.
public interface KeyTranslator {
String encode(String key) throws IOException, CubbyOperationException;
}Base64-encodes every key. Base64 expands the input by about a third, so a key of up to 187 bytes encodes to within the 250-byte limit. Usable on its own only if you control key lengths.
Decorator. Runs the delegate; if the result is over 250 characters, falls back to a SHA3-512 hash of the original UTF-8 bytes, Base64-encoded (88 characters). Values above 250 bytes therefore become 88-character keys regardless of how large they started.
configuration.setKeyTranslator(
new LargeKeyHashingTranslator(new DefaultKeyTranslator()));The hash is applied to the original key, not the encoded one, so two callers that supply the same logical key always compute the same hashed memcached key. Routing still uses the hashed form, so long and short versions of the same key end up on different hosts — the collision you care about (two different long keys mapping to the same short key) is statistically negligible with SHA3-512.
A KeyLocator owns the mapping from cache key to MemcachedHost. The
lifecycle is:
public interface KeyLocator {
void installRouting(ServerPool serverPool) throws CubbyOperationException;
void updateRouting(ServerPool serverPool);
MemcachedHost find(ServerPool serverPool, String key)
throws IOException, CubbyOperationException;
}-
installRoutingis called once on client startup. -
updateRoutingis called whenever a host joins or leaves (connects, disconnects, or is resuscitated). Implementations should detect no-op changes viaServerPool.representsHosts(List)and skip the rebuild in that case. -
findis on every request’s hot path.
Both shipped implementations guard their routing state with a
ReentrantReadWriteLock: find takes the read lock, rebuilds take the
write lock.
Sorts active host names alphabetically and indexes them as a string
array. A key is routed to activeNames[key.hashCode() % length].
Upside: trivial to reason about.
Downside: it is not a consistent hash. Adding or removing one host in a three-host cluster re-routes approximately 2/3 of all keys, so the cache hit rate drops sharply during a topology change. Pick this locator only when that is acceptable (e.g. a small warm-cache where re-fill is cheap).
Google’s Maglev consistent hashing. Builds a fixed-size lookup table
(the first prime greater than hosts * virtualHostCount, so
3 * 100 = 300 becomes 307 slots) and fills it by round-robining
through each host’s pre-computed permutation. The result: every host
owns approximately the same number of slots, and removing one host
re-routes only that host’s slots — the other ~2/3 of the keyspace
stays pinned to its existing home.
Cost: installRouting computes two hashes per host (SHA-256 for the
permutation offset, SHA3-256 for the skip) plus builds a
permutationSize integer array per host. For a 10-host cluster with
the default virtual count of 100 that’s a 1009-slot table (rounded to
the next prime), which is cheap; a 100-host cluster is a 10007-slot
table, still cheap but no longer free.
Per-request routing uses SipHash-1-3 (from io.whitfin:siphash) on
the key, which is fast and has good avalanche properties. The hash is
not cryptographic, and it does not need to be — the property that
matters is uniform distribution.
// default: 100 virtual entries per real host
configuration.setKeyLocator(new MaglevKeyLocator());
// more virtual entries => more balanced, larger table, slower install
configuration.setKeyLocator(new MaglevKeyLocator(250));-
Development or a single-node deployment:
DefaultKeyLocator. Simpler, no siphash dependency needed. -
Any production cluster with more than one host that might occasionally lose a node:
MaglevKeyLocator.
CubbyConfiguration.OPTIMAL chooses MaglevKeyLocator.
Every protocol operation has a Command subclass. The client layer is
a thin wrapper over constructing these and handing them to the
connection layer.
Base class:
public abstract class Command {
public abstract String getKey() throws CubbyOperationException;
public abstract byte[] construct(KeyTranslator keyTranslator)
throws IOException, CubbyOperationException;
public abstract Result process(Response response)
throws UnexpectedResponseException;
}| Command | Wire opcode & role |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
An |
Supporting types:
| Type | Role |
|---|---|
|
Enum over |
|
Enum over |
|
Immutable |
Commands are designed to be built fluently and sent directly via
client.send(command, timeoutSeconds) when the caller wants a feature
the client’s top-level API does not expose. For example, append:
Response r = client.send(
new SetCommand()
.setKey("log")
.setValue(" more\n".getBytes(StandardCharsets.UTF_8))
.setMode(SetMode.APPEND),
null);The connection’s read path parses every reply into a Response:
ResponseCode code = response.getCode();
byte[] value = response.getValue();
long cas = response.getCas();
boolean stale = response.isStale();
boolean won = response.isWon(); // miss-leader lease held
boolean alsoWon = response.isAlsoWon(); // concurrent lease holder
String token = response.getToken(); // echoed opaqueResponseCode is the enum over the two-letter codes defined by the
meta protocol:
-
MN— noop acknowledgement -
HD— hit / stored (success) -
VA— value follows -
EN— miss (get) -
EX— exists / CAS mismatch -
NF— not found (for CAS store / delete) -
NS— not stored (non-error failure)
Command.process(Response) is where these codes are collapsed into a
boolean-ish outcome. Callers of the top-level client API never see
raw codes.
Internals, documented because they are the source of most runtime behaviour and failure modes.
CubbyConnection is the interface — just start(), stop(),
send(Command, Long), and the Runnable.run() I/O loop.
NIOCubbyConnection is the only production implementation.
The selector loop:
-
selector.select(1000)— wait up to one second for readiness. -
If nothing is ready, increment the idle counter. When the counter exceeds
keepAliveSeconds, enqueue anmnnoop to prove the socket still works. If the selection key has gone invalid three times in a row, throwInvalidSelectionKeyException. -
If a key is ready, process reads first: call
responseReader.read(), and for every completeResponseextracted, pop aMissingLinkfrom the response queue and deliver the response to its callback. -
Then process writes: drain the request queue into the request writer, moving each
MissingLinkto the response queue as it is enqueued for transmission. Flush the writer to the socket. -
Any
IOExceptionorCubbyOperationExceptioninside the loop terminates it, closes the socket, and callsConnectionCoordinator.disconnect(host)so the host is removed from rotation until the defibrillator revives it.
RequestWriter, ResponseReader, JoinedBuffer,
ExposedByteArrayOutputStream, CommandBuffer, and MissingLink are
internal plumbing — they batch writes, parse response headers, glue
together fragmented reads, and carry (callback, bytes) pairs between
the request and response queues.
Request and response queues stay synchronized because memcached answers
in FIFO order per connection, so the oldest MissingLink on the
response queue always matches the oldest response on the wire. If that
invariant breaks (e.g. the server misbehaves), the read path raises
CubbyOperationException("Desynchronized connection state") and the
connection is torn down.
A periodic task run on a single-thread scheduled executor, one per
ConnectionCoordinator. It runs every resuscitationSeconds seconds
(scheduled with a fixed delay between passes), iterates
serverPool.values(), and for each inactive host opens a short-lived TCP
socket to the host’s current address. If the connection succeeds:
-
The newly resolved address is stored back into the
MemcachedHostviaregenerate(InetSocketAddress), so any DNS change is captured. -
After all inactive hosts have been probed the coordinator’s
reconnect(host)is called for each one that answered — this rebuilds theCubbyConnection, marks theHostControlactive, and refreshes the routing table.
Failure on the probe (any IOException) is silent; the host stays
inactive and will be retried on the next cycle.
Shutdown is owned by the coordinator: stop() shuts the scheduled
executor down and waits up to terminationStint (10s by default) for an
in-flight pass to finish via awaitTermination. If the pass does not
complete within that window the coordinator logs a warning and proceeds
to close the connections anyway. Probing the executor before tearing down
connections this way avoids a teardown race between closing connections
and the defibrillator trying to reopen them.
A 32-byte monotonic counter over a 90-character printable ASCII
alphabet, emitting strings of growing length as the counter increments.
90^32 tokens before wraparound is enough for any realistic workload.
The class is synchronized so it is safe to share across threads.
It is primarily useful when a caller wants to attach opaque tokens to
pipelined commands; Cubby’s normal request/response matching does not
use it.
Every module is wired by Spring factory beans. The canonical arrangement
is four beans: a server map, a configuration, a client, and whatever
downstream component consumes ProxyMemcachedClient.
Produces Map<String, MemcachedServer> from a pattern + spread.
CubbyMemcachedClientFactoryBean consumes this map directly.
Pattern rules:
-
host— single server, default port 11211. -
host:port— single server, explicit port. -
cache#.internal— multiple servers;#is replaced by each spread designator; each designator becomes the map key. -
cache#.internal:11211— same, with explicit port.
Spread examples (handed to
org.smallmind.nutsnbolts.util.Spread.calculate):
-
1-3→1,2,3 -
a,b,c→a,b,c -
01-04→01,02,03,04
<bean id="memcachedServerMap"
class="org.smallmind.memcached.utility.spring.MemcachedServerMapFactoryBean">
<property name="serverPattern" value="cache-#.internal:11211"/>
<property name="serverSpread" value="1-3"/>
</bean>Result: a three-entry map keyed { "1" → cache-1.internal:11211,
"2" → cache-2.internal:11211, "3" → cache-3.internal:11211 }. The map
keys are the logical host names carried into the routing table, so
they double as stable identifiers for consistent-hash placement.
A pattern with no # produces a single entry keyed "memcached".
Starts from a named preset (CubbyConfigurations.DEFAULT or
CubbyConfigurations.OPTIMAL) and applies any non-null properties you
set. Unset properties fall through to the preset’s values.
<bean id="memcachedConfiguration"
class="org.smallmind.memcached.cubby.spring.CubbyConfigurationFactoryBean">
<property name="initial" value="OPTIMAL"/>
<property name="defaultRequestTimeoutMilliseconds" value="500"/>
<property name="connectionsPerHost" value="4"/>
<property name="authentication">
<bean class="org.smallmind.memcached.cubby.Authentication">
<constructor-arg value="cacheuser"/>
<constructor-arg value="cachepass"/>
</bean>
</property>
</bean>The preset’s setters are called on the preset’s singleton constant —
see the note in [CubbyConfiguration]. If you need two clients with
different overrides, give them two different
CubbyConfigurationFactoryBean beans.
Builds, starts, and stops the client with the Spring context.
Implements InitializingBean (start on refresh) and DisposableBean
(stop on close).
<bean id="memcachedClient"
class="org.smallmind.memcached.cubby.spring.CubbyMemcachedClientFactoryBean">
<property name="enabled" value="true"/>
<property name="configuration" ref="memcachedConfiguration"/>
<property name="servers" ref="memcachedServerMap"/>
</bean>-
enabled=falsesuppresses client creation entirely;getObject()returnsnull. Useful for environments with no cache available — combined with anInMemoryMemcachedClientbean gated by profile, the same application code runs locally and in production. -
serversis the map produced byMemcachedServerMapFactoryBean. Map key is used as the logical host name. -
The factory logs
Memcached servers(…) initializedandMemcached client startedon success so you can confirm the right hosts were resolved on startup.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="memcachedServerMap"
class="org.smallmind.memcached.utility.spring.MemcachedServerMapFactoryBean">
<property name="serverPattern" value="cache-#.internal"/>
<property name="serverSpread" value="1-3"/>
</bean>
<bean id="memcachedConfiguration"
class="org.smallmind.memcached.cubby.spring.CubbyConfigurationFactoryBean">
<property name="initial" value="OPTIMAL"/>
<property name="defaultRequestTimeoutMilliseconds" value="500"/>
<property name="connectionsPerHost" value="4"/>
</bean>
<bean id="memcachedClient"
class="org.smallmind.memcached.cubby.spring.CubbyMemcachedClientFactoryBean">
<property name="configuration" ref="memcachedConfiguration"/>
<property name="servers" ref="memcachedServerMap"/>
</bean>
<!-- any downstream component that accepts ProxyMemcachedClient -->
<bean id="userCache"
class="org.smallmind.persistence.cache.memcached.MemcachedCache">
<constructor-arg ref="memcachedClient"/>
<constructor-arg value="users"/>
<constructor-arg value="com.example.UserRecord"/>
<constructor-arg value="600"/>
</bean>
</beans>client.set("user:42", 600, userRecord); // TTL 10 minutes
UserRecord u = client.get("user:42"); // null if absent or expired
client.delete("user:42");CASValue<UserRecord> snapshot = client.casGet("user:42");
if (snapshot == null) {
// key missing; insert with add-only semantics:
client.casSet("user:42", 600, newRecord, 0L);
} else {
UserRecord updated = mutate(snapshot.getValue());
boolean stored = client.casSet("user:42", 600, updated, snapshot.getCas());
if (!stored) {
// another writer got there first; retry or give up
}
}A casSet with cas = 0 is Cubby’s idiom for insert-if-absent:
SetCommand promotes itself to add-only mode and asks the server to
return the resulting CAS token in the response.
Refresh the TTL of an entry each time it is read so hot items stay in the cache indefinitely and cold items expire on their own:
UserRecord u = client.getAndTouch("user:42", 600);Or, when you need the CAS token alongside:
CASValue<UserRecord> v = client.casGetAndTouch("user:42", 600);Use the low-level ArithmeticCommand through client.send:
Response r = client.send(
new ArithmeticCommand()
.setKey("hits:2026-04-21")
.setDelta(1)
.setInitial(0) // create with 0 if absent
.setExpiration(86400), // 24-hour TTL on creation
null);
long hits = Long.parseLong(new String(r.getValue(), StandardCharsets.UTF_8));Negative deltas auto-flip to decrement: setDelta(-5) subtracts 5.
Memcached’s append and prepend are byte-level operations on existing
values. If you also want the key to be created when missing, turn on
vivify:
client.send(
new SetCommand()
.setKey("log:today")
.setValue(logLine.getBytes(StandardCharsets.UTF_8))
.setMode(SetMode.APPEND)
.setVivify(true)
.setExpiration(86400),
null);Remember that Cubby’s default codec wraps written values in an
ObjectOutputStream envelope, which is not something you can
meaningfully append to. For byte-level append/prepend workflows,
switch the codec to a pass-through one (or set raw bytes yourself via
the low-level send API).
Map<String, UserRecord> users =
client.get(List.of("user:1", "user:2", "user:3"));Absent keys appear in the map with a null value (Cubby does not
omit them).
public class UserCacheTest {
private ProxyMemcachedClient client;
private MemcachedCache<UserRecord> cache;
@BeforeMethod
public void setUp() {
client = new InMemoryMemcachedClient();
cache = new MemcachedCache<>(client, "users", UserRecord.class, 600);
}
@Test
public void roundTrip() {
UserRecord u = new UserRecord(42L, "Alice");
cache.put("42", u, 600);
assertThat(cache.get("42")).isEqualTo(u);
}
}Because MemcachedCache accepts any ProxyMemcachedClient, the
test is a straightforward substitution.
The quickest way to push JSON through the cache:
CubbyConfiguration configuration = CubbyConfiguration.OPTIMAL
.setCodec(new LargeValueCompressingCodec(new JsonCubbyCodec()));LargeValueCompressingCodec still works on top of a JSON codec — it
looks at serialized byte length, not content. Be aware that
LargeValueCompressingCodec uses GzipObjectWrapper, a
Serializable envelope, so the wrapping step uses Java
ObjectOutputStream internally even if your codec is JSON. If the
compression decorator is undesirable for your codec pipeline, leave it
off.
MemcachedHost[] hosts = {
new MemcachedHost("a", "cache-1.internal", 11211),
new MemcachedHost("b", "cache-2.internal", 11211),
new MemcachedHost("c", "cache-3.internal", 11211),
};
CubbyMemcachedClient client = new CubbyMemcachedClient(
CubbyConfiguration.OPTIMAL.setConnectionsPerHost(4),
hosts);
client.start();
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
try {
client.stop();
} catch (Exception ignored) {
}
}));-
connectionTimeoutMillisecondsgoverns TCP connection establishment. A host that does not accept the connection within this window is declared unreachable. Typical value: 1–5 seconds. -
defaultRequestTimeoutMillisecondsgoverns per-operation waits.0means "block forever," which is appropriate for batch jobs but catastrophic for a request-path service: one flaky server can park every client thread indefinitely. Set this to a concrete value (100–1000 ms is typical) in production. -
readTimeoutMillisecondsapplies to the defibrillator’s probe socket, not to the main NIO channel.
connectionsPerHost is the one throughput knob you can turn without
touching anything else. With the default of 1, each host has one
connection; all commands addressed to that host are serialised through
that connection’s selector thread. Under heavy concurrent load this
becomes a bottleneck. With connectionsPerHost = 4 the multiplexer
fans out to four coordinators, which means four connections per host
and four selector threads handling them.
Each coordinator has its own ServerPool, its own routing table, and
its own defibrillator thread. So with C connections per host and H
hosts you get C × H selector threads and C defibrillator threads. For
a typical 3-host cluster at connectionsPerHost = 4, that is 12
selector threads and 4 defibrillators — modest.
-
A host’s connection dies. The connection’s selector loop catches the exception, calls
shutdown(unexpected=true), which callsConnectionCoordinator.disconnect(host). The coordinator marks theHostControlinactive and asks the key locator to rebuild the routing table. Keys previously routed to that host are now routed to the locator’s fallback choice (Maglev re-fills the table; Default reshuffles mod-length). -
Inactive host stays inactive until the defibrillator probes it (within
resuscitationSeconds). -
Successful probe → coordinator rebuilds the connection, marks active, refreshes routing.
Consequences worth knowing:
-
A short outage of one host with
DefaultKeyLocatorinvalidates ~2/3 of the cache (every key is re-routed); withMaglevKeyLocatoronly the downed host’s slice is disrupted. -
A noisy host that flaps in and out will produce churn in the routing table. The defibrillator’s interval is the cadence of that churn; tune
resuscitationSecondsup if you would rather wait longer before readmitting a flapping host.
Each defibrillator probe constructs a fresh InetSocketAddress, which
re-resolves the hostname. If your memcached endpoints sit behind a
load balancer (or a K8s service) whose backends rotate, the client
picks up the new backend automatically on the next probe. The stored
address on MemcachedHost is replaced via regenerate() before the
reconnect.
-
Key size. 250 bytes on the wire.
LargeKeyHashingTranslatorcollapses anything longer to an 88-byte SHA3-512 hash, so application code can use arbitrarily long logical keys. Memcached sees only the hashed form. -
Value size. 1 MB on the wire.
LargeValueCompressingCodecmitigates by GZIP-compressing anything over the threshold; for bulk binary payloads that do not compress, the limit is firm and you should store out-of-band (S3/GCS) and cache a reference. -
Null. Memcached has no null; key absence is the null.
ObjectStreamCubbyCodecrefuses to serialize a Javanullbecause the distinction between "stored null" and "absent" cannot survive the round trip.
Cubby logs via scribe-pen (org.smallmind.scribe.pen.LoggerManager),
which bridges to whatever logging backend the host application has
configured. Connection lifecycle events (Connecting to memcached
host(…), Disconnected memcached host(…),
Reconnected memcached host(…)) are logged at info. Unhandled
exceptions on the selector loop and the defibrillator are logged at
error with stack traces. A production deployment should ensure this
logger is wired through to the central log system.
The primary consumer of memcached-utility in this repository is
org.smallmind.persistence.cache.memcached.MemcachedCache, a
CASSupportingPersistenceCache<String, V> implementation that sits one
layer above ProxyMemcachedClient. It applies a discriminator prefix
to every key (so multiple named caches can share one cluster without
collision), exposes a uniform exception type
(CacheOperationException), and implements putIfAbsent via a CAS
loop on top of the SPI. Any cache-consuming component elsewhere in the
codebase receives that MemcachedCache — never CubbyMemcachedClient
directly — which is why ProxyMemcachedClient exists and why swapping
between the in-memory and the Cubby backend is a one-bean change.
Three other Java memcached clients are in common use. Cubby is not trying to replace any of them — it exists because this project needs a specific set of properties (a narrow SPI, an in-memory double, Maglev routing, automatic large-key and large-value handling, Spring wiring in the idiom the rest of the project uses) that none of the others supply as a single package. A quick side-by-side helps to set expectations about when Cubby is the right choice and when it is not.
The most widely deployed Java memcached client. Mature, battle-tested, broad feature set.
-
Protocol: supports both the legacy text protocol and the binary protocol; defaults to binary. Cubby speaks only the meta protocol, which is a newer text-protocol dialect that postdates spymemcached’s binary support. The two are not wire-compatible with each other on a given request, but they target the same memcached server (memcached understands all three).
-
Concurrency model: asynchronous. Every operation returns a
Futureand the caller opts in to blocking via.get(). Cubby is synchronous at the SPI level —client.get(…)blocks the caller thread on a callback until the response arrives or the timeout elapses. This is a deliberate trade: most of this project’s code wants a value, not a future, and the cost of parking the calling thread is acceptable when the request path is already doing I/O. -
Routing: Ketama consistent hashing (and a few legacy options). Cubby offers Maglev or simple modulo, not Ketama. Maglev gives more even distribution than Ketama at the cost of a larger lookup table; the two are functionally equivalent for the property that matters — only the downed host’s keys get re-routed on a topology change.
-
Connection model: one connection per host, with an internal worker queue. Cubby supports N connections per host via the multiplexer, which can relieve lock contention under high concurrent load.
-
Serialization: pluggable
Transcoder<T>per operation. Cubby’sCubbyCodecis a single cluster-wide strategy — simpler, but less flexible if different keys should be serialized differently. -
Large values / large keys: no automatic compression or key hashing; the caller has to solve both. Cubby handles both out of the box (
LargeValueCompressingCodec,LargeKeyHashingTranslator). -
License: MIT.
Prefer spymemcached if you need an async API, Ketama compatibility with non-Java callers, or a large community and long track record. Prefer Cubby if you want a synchronous SPI with an in-memory double for tests, automatic handling of oversized keys and values, and Spring wiring in this project’s idiom.
A high-throughput NIO client with a similar architectural footprint to spymemcached.
-
Protocol: text and binary; Cubby is meta-protocol only.
-
Concurrency model: offers both synchronous and asynchronous APIs on the same client. Cubby is synchronous only.
-
Routing: Ketama by default; several other strategies are configurable. No Maglev.
-
Connection model: configurable connection pool per host (similar to Cubby’s
connectionsPerHost). Uses a custom NIO framework (yanf4j), which is one more transitive dependency; Cubby’s NIO is hand-rolled onjava.nio.channels.Selectorwith no extra transport jar. -
Serialization: pluggable
Transcoder, similar to spymemcached. -
Large values / large keys: built-in compression for values over a configurable threshold (similar in spirit to Cubby’s
LargeValueCompressingCodec). No automatic key hashing. -
License: Apache 2.0.
Prefer XMemcached if you want async + sync on the same client and are already aligned with Ketama. Prefer Cubby if the value add is the SPI boundary plus the in-memory double plus Maglev, not raw throughput.
The original Java memcached client. Long-standing, blocking-I/O based, much older than the other two.
-
Protocol: text protocol only; no binary, no meta.
-
Concurrency model: synchronous, like Cubby. Pools blocking sockets rather than running an NIO selector. Under heavy concurrent load the pool itself becomes the throughput ceiling.
-
Routing: consistent hashing (original Ketama variant) or simple modulo.
-
Connection model: blocking sockets, one-per-pool-entry. No event loop.
-
Serialization: Java object streams by default, with hooks for other types. Similar in scope to Cubby’s
ObjectStreamCubbyCodec(the default for this project). -
Large values / large keys: compression support for values; no automatic key hashing.
-
License: Apache 2.0.
Prefer Whalin if the use case is conservatively low-concurrency and you value a minimal, blocking-socket model. Prefer Cubby (or spymemcached, or XMemcached) for anything throughput-sensitive — the NIO model is the main reason the three newer clients exist at all.
-
A stable SPI (
ProxyMemcachedClient) with a drop-in in-memory implementation that satisfies the same contract. The other clients all offer their ownMemcachedClientinterface, but substituting a test double means writing one yourself or pulling in a heavyweight mocking library. Cubby’s double is six files and ships with the module. -
Automatic oversize-key hashing.
LargeKeyHashingTranslatortransparently collapses any key over 250 characters to a Base64- encoded SHA3-512 hash, so application code never has to reason about the 250-byte wire limit. None of the three clients above do this by default. -
Maglev consistent hashing (rather than Ketama), which produces more even slot distribution and has independent, well-characterised re-mapping properties.
-
SmallMind Spring wiring out of the box — `FactoryBean`s for the server map, the configuration, and the client, plus pattern + spread expansion so a three-node cluster is a one-line XML edit.
-
Integration with the rest of the SmallMind stack —
scribe-penlogging,nutsnboltsutilities (Base64, spread parser, component status, hash utilities), and the same formatted-exception pattern used elsewhere in the codebase. This is valuable inside this project and neutral-to-negative outside it.
| Feature | Cubby | spymemcached | XMemcached | Whalin |
|---|---|---|---|---|
Wire protocol |
meta |
text + binary |
text + binary |
text |
I/O model |
NIO, sync SPI |
NIO, async |
NIO, sync + async |
blocking pool, sync |
Default routing |
Maglev |
Ketama |
Ketama |
Ketama / modulo |
Conn. pool / host |
configurable |
one (+ queue) |
configurable |
socket pool |
Auto large-value compression |
yes |
no |
yes |
yes |
Auto large-key hashing |
yes |
no |
no |
no |
In-memory test double |
yes |
no |
no |
no |
SPI abstraction |
yes ( |
no |
no |
no |
SASL auth |
yes (PLAIN) |
yes |
yes |
no |
External dependency weight |
nutsnbolts + scribe-pen (+ optional siphash) |
minimal |
yanf4j |
minimal |
Cubby is a good fit when the abstraction boundary matters more than the last few percent of raw throughput. If you are already on spymemcached or XMemcached and happy with them, there is no particular reason to switch; if you are starting fresh inside this project, the integration benefits make Cubby the default.