Scylla is a distributed system, with multiple Scylla nodes communicating with each other and with client nodes making requests. All these different nodes use many different protocols and TCP ports (currently, all use TCP) to communicate with each other. Some of these protocols have an external specification - for example Scylla supports Cassandra's CQL protocol and Amazon's DynamoDB protocol for client requests. Other protocols, both client-facing and internal (between Scylla nodes), are Scylla-defined.
The goal of this document is to survey all these different protocols - what goes over which protocol, which port each protocol uses (and how this can be configured), how to disable or enable some of the protocols, and what part of the Scylla source code handles which protocol.
Unfortunately, the long evolution of some of these protocols resulted in some of them getting confusing names, and often different names in Scylla's code, documentation and configuration. This document aims to clear up some of this confusion.
The Wireshark tool can be used to inspect and understand the messages sent over many of the following protocols. This includes even Scylla's internal inter-node protocol - for instructions and examples check out the blog post: https://www.scylladb.com/2020/05/21/dissecting-scylla-packets-with-wireshark/
All inter-node communiction - communication between the Scylla nodes themselves - happen over one protocol and one TCP port - by default port 7000 (unencrypted) or 7001 (encrypted). These are the same TCP ports used by Cassandra for this purpose, but Scylla's internal protocol is completely different from Cassandra, and incompatible with it (one cannot mix Scylla and Cassandra nodes in a single cluster).
Scylla's internal protocol is built on top of Seastar's "RPC" messaging
mechanism, itself built on top of TCP. This protocol includes messages of
different types ("verbs") - which are all listed in the source file
message/messaging_service.cc.
Here is a non-exhaustive list of things that Scylla does with these messages:
-
"Gossip" between the nodes to maintain knowledge of the cluster topology: Which nodes belong to the cluster, their state and token ownership.
-
Read and write (mutation) messages sent from a node receiving a client's request (a so-called coordinator) to a replica which holds relevant data.
-
In particular, mutations to special system tables are used to maintain cluster-wide agreement on schemas (table definitions).
-
Streaming data to new nodes and repairing data between nodes.
-
Messages used to implement LWT (lightweight transactions).
Although all these different types of messages use the same RPC protocol and
the same destination port on the receiver, we don't want to multiplex all of
these messages over a single TCP connection. If we do, this would allow
messages of one type to severely delay messages of other types (head-of-line
blocking). So Scylla opens several sockets to the same destination port
and uses a different socket for different message types. The function
do_get_rpc_client_idx determines which types of messages get bunched
together over one socket. As of this writing, the different messages are
split into four sockets.
As everything in Scylla, the messaging service (providing this internal communication) is a sharded service, so each shard keeps its own sockets to each remote node. The remote node is also sharded. We cannot know for sure which shard on the remote node will handle the messages, but we make an effort (which is not a guarantee!) that if the two nodes have the same number of shards, messages from shard N in the source node arrive to shard N in the destination node.
Port 7000 is the default port for Scylla's internal communication. This choice
can be overriden by the storage_port configuration option. This awkward
name, storage_port, was kept for backward compatibility with Cassandra's
YAML configuration file. In very early Cassandra versions, this port was
used only for "storage" messages (read and write), and other messages such
as gossip were sent to a different port. So today we are still stuck with
this outdated name of this configuration option.
There is also a listen_address configuration option to set the IP address
(and therefore network interface) on which Scylla should listen for the
internal protocol. This address defaults to localhost, but in any
setup except a one-node test, should be overriden.
TODO: there is also listen_interface option... Which wins? What's the default?
TODO: mention SSL, how it is configured, and ssl_storage_port (default 7001).
The CQL binary protocol is Cassandra's and Scylla's main client-facing protocol. Scylla supports several Scylla-only extensions to this protocol, described in protocol-extensions.md.
By default, Scylla listens to the CQL protocol on port 9042, which can be
configured via the native_transport_port configuration option. Scylla also
supports the CQL protocol via TLS/SSL encryption, which can be enabled via the
native_transport_port_ssl configuration option (default port is 9142). Users
that want to only support the encrypted CQL protocol can disable the
unencrypted default CQL protocol by setting the native_transport_port to 0.
The CQL protocol support can be disabled altogether by setting the
start_native_transport option to false. These option names were chosen for
backward-compatibility with Cassandra configuration files: They refers to CQL
as the "native transport", to contrast with the older Thrift protocol
(described below) which wasn't native to Cassandra.
There is also a rpc_address configuration option to set the IP address
(and therefore network interface) on which Scylla should listen for the
CQL protocol. This address defaults to localhost, but in any setup except
a one-node test, should be overriden. Note that the same option rpc_address
applies to both CQL and Thrift protocols.
TODO: there is also rpc_interface option... Which wins? What's the default?
TODO: again mention SSL, and native_transport_port_ssl (default 9142).
The Apache Thrift protocol was early Cassandra's client protocol, until it was superceded in Cassandra 1.2 with the binary CQL protocol. Thrift was still nominally supported by both Cassandra and Scylla for many years, but was recently dropped in Cassandra (version 4.0) and is likely to be dropped by Scylla in the future as well, so it is not recommended for new applications.
By default scylla listens to the Thirft protocol on port 9160, which can be
configured via the rpc_port configuration option. Again, this confusing name
was used for backward-compatibility with Cassandra's configuration files.
Cassandra used the term "rpc" because Apache Thrift is a remote procedure
call (RPC) framework. In Scylla, this name is especially confusing, because
as mentioned above, Scylla's internal communication protocol is based on
Seastar's RPC, which has nothing to do with the "rpc_port" described here.
There is also a rpc_address configuration option to set the IP address
(and therefore network interface) on which Scylla should listen for the
Thrift protocol. This address defaults to localhost, but in any
setup except a one-node test, should be overriden. Note that the same
option rpc_address applies to both CQL and Thrift protocols.
TODO: there is also rpc_interface option... Which wins? What's the default?
TODO: is there an SSL version of Thrift?
Scylla also supports, as an experimental feature, Amazon's DynamoDB API.
The DynamoDB API is a JSON over HTTP (unencrypted) or HTTPS (encrypted)
protocol. Because Scylla's support for this protocol is experimental,
it is not turned on by default, and must be turned on manually by setting
the alternator_port and/or alternator_https_port configuration option.
"Alternator" is the codename of Scylla's DynamoDB API support, and is
documented in more detail in alternator.md.
The standard ports that DynamoDB uses are the standard HTTP and HTTPS ports (80 and 443, respectively), but in tests we usually use the unprivileged port numbers 8000 and 8043 instead.
There is also an alternator_address configuration option to set the IP
address (and therefore network interface) on which Scylla should listen
for the DynamoDB protocol. This address defaults to 0.0.0.0.
Scylla also has partial and experimental support for the Redis API.
Again, because this support is experimental, it is not turned on by
default, and must be turned on manually by setting the redis_port
and/or redis_ssl_port configuration option.
The traditional port used for Redis is 6379. Regular Redis does not support SSL, so there is no traditional choice of port for it.
See redis.md for more information about Scylla's support for the Redis protocol.
Scylla provides an HTTP-based protocol to fetch performance and activity metrics from Scylla, which is described in detail in metrics.md.
Scylla listens by default on port 9180 for metric requests. This port number
can be configured with the prometheus_port configuration option, named
after the Prometheus protocol - and the Prometheus server which is usually
used to collect these metrics.
There is also a prometheus_address configuration option to set the IP
address (and therefore network interface) on which Scylla should listen
for the metrics protocol. This address defaults to 0.0.0.0.
The CQL client protocol mentioned above is useful mostly for data and table requests, but does not offer commands for administrative operations such as repair, compact, adding or removing nodes, and so on. Cassandra uses nodetool and JMX for these (see below), but Scylla's native approach is a RESTful API (HTTP requests).
Scylla listens for this REST API by default on port 10000, which can be
configured with the api_port configuration option.
The REST API has no notion of autentication of authorization, and allows
anyone connecting to it to perform destructive operations. Therefore, it
only listens for connection on the localhost (127.0.0.1) interface. This
default can be overridden by the api_address option - but shouldn't.
The available REST API commands are listed in JSON files in the api/api-doc/
directory of the Scylla source. A user can explore this API interactively
by pointing a browser to http://localhost:10000/ui/.
There is an ongoing, but incomplete, effort to replace this REST API by a newer, "v2", API. See api_v2.md. When complete, this v2 API will be available on the same port. You can explore the little it offers now in the aforementioned UI by replacing the URL in the box with "http://localhost:10000/v2".
The "nodetool" management command connects to Scylla using Java's JMX protocol, not the REST API described above. This protocol was kept for backward compatibility with Cassandra and its unmodified "nodetool" command. To implement the JMX protocol, we have a separate project, scylla-jmx, which runs a Java program which accepts the JMX requests supported by Cassandra, and translates them to requests to our own REST API. These REST API requests are sent to Scylla's REST API port over the loopback (localhost) interface.
The port on which scylla-jmx listens is by default port 7199. This port,
and the listen address, can be overridden with the -jp and -ja options
(respectively) of the scylla-jmx script.