CRC removal during diskless full sync with TLS enabled. #1479

talxsha · 2024-12-23T15:37:43Z

Implemented a mechanism to eliminate CRC64 checksumming during full sync when not writing to disk (using a connection that has data integrity checks such as TLS), as it adds overhead with minimal benefit.

Nodes can skip CRC calculations when these conditions are met:

Running diskless sync on primary.
Running diskless load on the replica.
Primary-replica connection is integrity checked.

Closes #1129

ranshid · 2024-12-23T19:04:51Z

@talxsha before I look into this, lets put some details in the top comment. linking the issue is not what we susually do.
Please state shortly what is the problem we are solving and what this solution includes.

ranshid · 2024-12-23T19:14:46Z

src/replication.c

@@ -1244,11 +1244,12 @@ void syncCommand(client *c) {
 * the primary can accurately lists replicas and their listening ports in the
 * INFO output.
 *
- * - capa <eof|psync2|dual-channel>
+ * - capa <eof|psync2|dual-channel|disable_sync_crc>


I am not super fond of the disable_sync_crc capability name. maybe a better name woulod be bypass_crc?

ranshid · 2024-12-23T19:16:10Z

src/config.c

@@ -3156,6 +3156,7 @@ static int applyClientMaxMemoryUsage(const char **err) {
 standardConfig static_configs[] = {
    /* Bool configs */
    createBoolConfig("rdbchecksum", NULL, IMMUTABLE_CONFIG, server.rdb_checksum, 1, NULL, NULL),
+    createBoolConfig("disable-sync-crc", NULL, MODIFIABLE_CONFIG, server.disable_sync_crc, 0, NULL, NULL),


we normally do not like to introduce new configurations. In this case the feature is controlled via capability so no issues with compatibility. Is there a way this would still be required in some cases>

Yes, please don't introduce a config for this.

ranshid · 2024-12-23T19:26:20Z

src/server.c

@@ -2218,6 +2218,7 @@ void initServerConfig(void) {
    server.fsynced_reploff_pending = 0;
    server.rdb_client_id = -1;
    server.loading_process_events_interval_ms = LOADING_PROCESS_EVENTS_INTERVAL_DEFAULT;
+    server.repl_meet_disable_crc_cond = 0;


I did not follow on why we need to keep this flag on the server? aren't all the checks in readSyncBulkPayload valid at the point of decision to flag the rdb?

ranshid · 2024-12-23T19:44:22Z

src/server.h

@@ -1838,6 +1842,7 @@ struct valkeyServer {
    double stat_fork_rate;                         /* Fork rate in GB/sec. */
    long long stat_total_forks;                    /* Total count of fork. */
    long long stat_rejected_conn;                  /* Clients rejected because of maxclients */
+    size_t stat_total_crc_disabled_syncs_stated;   /* Total number of full syncs stated with CRC checksum disabled */   // AMZN


Please remove the AMZN, we are not in Kansas anymore. Also the stat name is weird, maybe we can use another stat eg, sync_bypass_crc? .
Also note that In general I would not find any reason to have this statistic unless it is used for writing tests right? Maybe such stats are better be placed under the debug section of the info, but I guess we already have so many stats so i would let it pass.

ranshid · 2024-12-23T19:45:02Z

src/rdb.c

@@ -3601,6 +3612,12 @@ int rdbSaveToReplicasSockets(int req, rdbSaveInfo *rsi) {
        }
        serverSetCpuAffinity(server.bgsave_cpulist);

+        if (disable_sync_crc_capa == 1) {


Suggested change

if (disable_sync_crc_capa == 1) {

if (disable_sync_crc_capa) {

ranshid · 2024-12-23T19:46:22Z

src/rdb.c

@@ -3354,7 +3355,7 @@ int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadin
        if (rioRead(rdb, &cksum, 8) == 0) goto eoferr;
        if (server.rdb_checksum && !server.skip_checksum_validation) {
            memrev64ifbe(&cksum);
-            if (cksum == 0) {
+            if (cksum == 0 || (rdb->flags & RIO_FLAG_DISABLE_CRC) != 0) {


Suggested change

if (cksum == 0 || (rdb->flags & RIO_FLAG_DISABLE_CRC) != 0) {

if (cksum == 0 || (rdb->flags & RIO_FLAG_DISABLE_CRC)) {

madolson · 2024-12-24T04:08:42Z

Added the functionality to disable CRC calculations during diskless full sync with TLS enabled.

Also add justification for why we should do this only when TLS is enabled. Given that the network has built in checksumming, I'm still not convinced about the tradeoff we are making given that the steady state replication is not checksummed.

ranshid · 2024-12-24T05:23:52Z

@madolson should I tag it as a major-decision ? I think it worth discussion.

ranshid · 2024-12-24T05:27:07Z

src/replication.c

+        // Set a flag to determin later whether or not the replica will skip CRC calculations for this sync -
+        // Disable CRC on replica if: (1) TLS is enabled; (2) replica disable_sync_crc is enabled; (3) diskelss sync enabled on both replica and primary.
+        // Otherwise, CRC should be enabled/disabled as per server.rdb_checksum
+        if (connIsTLS(conn) && server.disable_sync_crc && use_diskless_load && usemark)


a. I think we should encapsulate this whole condition in some function to make the code more readable.
b. The connIsTLS part of the decision is somewhat too intrusive IMO. I think maybe we can add an API in the connection abstraction like connIntegrityChecked or something like this. maybe there will be non-TLS connections (eg QUIC) which will provide some integrity mechanism which will not be defined as "TLS"

madolson · 2024-12-24T06:03:26Z

@madolson should I tag it as a major-decision ? I think it worth discussion.

For now it's not. It's just an internal one. I would probably just ping PingXie directly and core team if anyone else is interested.

Signed-off-by: Tal Shachar <[email protected]>

…o bypass_crc, encapsulated condition checks for skipping CRC, and chenged connIsTLS condition to connIntegrityChecked in ConnectionType. Some changes in the test as well Signed-off-by: Tal Shachar <[email protected]>

Signed-off-by: talxsha <[email protected]>

ranshid · 2025-01-02T09:42:41Z

src/connection.h

@@ -121,6 +121,9 @@ typedef struct ConnectionType {

    /* TLS specified methods */
    sds (*get_peer_cert)(struct connection *conn);
+
+    /* Miselenious */
+    int (*connIntegrityChecked)(void);  // return 1 iff connection type has built-in integrity checks


Suggested change

int (*connIntegrityChecked)(void); // return 1 iff connection type has built-in integrity checks

int (*connIntegrityChecked)(void); // return 1 if connection type has built-in integrity checks

ranshid · 2025-01-02T09:51:59Z

src/server.h

@@ -1988,6 +1993,7 @@ struct valkeyServer {
    char *rdb_filename;                   /* Name of RDB file */
    int rdb_compression;                  /* Use compression in RDB? */
    int rdb_checksum;                     /* Use RDB checksum? */
+    int bypass_crc;                       /* Skip RDB checksum? Applicable only for TLS enabled diskless full sync */


I am not sure why do we need to keep this server flag?

ranshid · 2025-01-02T09:53:22Z

src/replication.c

@@ -1972,6 +1974,11 @@ static int useDisklessLoad(void) {
    return enabled;
 }

+/* Returns 1 if the replica can skip CRC calculations during full sync */
+int replicationBypassCRC(connection *conn, int is_replica_diskless_load, int is_primary_diskless_sync) {


can we change it to be more explicit name? eg replicationSupportBypassCRC

ranshid · 2025-01-02T09:55:00Z

src/replication.c

@@ -2251,6 +2259,7 @@ void readSyncBulkPayload(connection *conn) {

        serverLog(LL_NOTICE, "PRIMARY <-> REPLICA sync: Loading DB in memory");
        startLoading(server.repl_transfer_size, RDBFLAGS_REPLICATION, asyncLoading);
+        if (replicationBypassCRC(conn, use_diskless_load, usemark)) rdb.flags |= RIO_FLAG_BYPASS_CRC;


lets document some explaining why we think it is O.K to take the decision based on the usemark and connection support.

Signed-off-by: Tal Shachar <[email protected]>

src/rdb.c

Signed-off-by: ranshid <[email protected]>

ranshid

@madolson the code seems fine now. would you like to take a quick look as well?

Signed-off-by: Ran Shidlansik <[email protected]>

madolson

The code looks okay, mostly some nits to improve clarity.

madolson · 2025-01-02T19:12:29Z

src/tls.c

@@ -1186,6 +1190,9 @@ static ConnectionType CT_TLS = {

    /* TLS specified methods */
    .get_peer_cert = connTLSGetPeerCert,
+
+    /* Miselenious */


Suggested change

/* Miselenious */

/* Miscellaneous */

madolson · 2025-01-02T19:12:38Z

src/unix.c

@@ -207,6 +207,9 @@ static ConnectionType CT_Unix = {
    .process_pending_data = NULL,
    .postpone_update_state = NULL,
    .update_state = NULL,
+
+    /* Miselenious */


Suggested change

/* Miselenious */

/* Miscellaneous */

madolson · 2025-01-02T19:13:04Z

src/server.c

@@ -5879,6 +5880,7 @@ sds genValkeyInfoString(dict *section_dict, int all_sections, int everything) {
                "instantaneous_input_repl_kbps:%.2f\r\n", (float)getInstantaneousMetric(STATS_METRIC_NET_INPUT_REPLICATION) / 1024,
                "instantaneous_output_repl_kbps:%.2f\r\n", (float)getInstantaneousMetric(STATS_METRIC_NET_OUTPUT_REPLICATION) / 1024,
                "rejected_connections:%lld\r\n", server.stat_rejected_conn,
+                "total_sync_bypass_crc:%ld\r\n", server.stat_total_sync_bypass_crc,


Suggested change

"total_sync_bypass_crc:%ld\r\n", server.stat_total_sync_bypass_crc,

"total_sync_bypass_crc:%ld\r\n", server.stat_total_sync_bypass_crc,

What is the use of this metric? What are end users supposed to do with this information?

This metric is merely for testing purposes

madolson · 2025-01-02T19:17:35Z

src/socket.c

@@ -447,6 +447,9 @@ static ConnectionType CT_Socket = {
    .process_pending_data = NULL,
    .postpone_update_state = NULL,
    .update_state = NULL,
+
+    /* Miselenious */


Suggested change

/* Miselenious */

/* Miscellaneous */

madolson · 2025-01-02T19:22:19Z

src/replication.c

@@ -3511,11 +3527,19 @@ void syncWithPrimary(connection *conn) {
         *
         * EOF: supports EOF-style RDB transfer for diskless replication.
         * PSYNC2: supports PSYNC v2, so understands +CONTINUE <new repl ID>.
+         * BYPASS-CRC: supports skipping CRC calculations during full sync.


BYPASS isn't the right word, since it implies we are doing something else instead. I think SKIP-CRC is what you want.

madolson · 2025-01-02T19:34:22Z

tests/integration/bypass-crc.tcl

+                    set replica_bypassing_crc_count [string match {*total_sync_bypass_crc:1*} [$replica info stats]]
+                    set stats [regexp -inline {total_sync_bypass_crc:(\d+)} [$primary info stats]]
+                    set primary_bypass_crc_count [lindex $stats 1]


Suggested change

set replica_bypassing_crc_count [string match {*total_sync_bypass_crc:1*} [$replica info stats]]

set stats [regexp -inline {total_sync_bypass_crc:(\d+)} [$primary info stats]]

set primary_bypass_crc_count [lindex $stats 1]

set replica_bypassing_crc_count [s 0 total_sync_bypass_crc]

set primary_bypass_crc_count [s 1 total_sync_bypass_crc]

madolson · 2025-01-02T19:36:46Z

tests/integration/bypass-crc.tcl

@@ -0,0 +1,37 @@
+start_server {tags {"repl tls"} overrides {save {}}} {


We only need to run these tests with TLS enabled, otherwise this condition is always false.

Correct. Shouldn't we have tests to check that without TLS we do not skip CRC?

madolson · 2025-01-02T19:46:06Z

src/replication.c

+            /* We can bypass CRC checks when data is transmitted through a verified stream.
+             * The usemark flag indicates that the primary is streaming the data directly without
+             * writing it to storage.
+             * Similarly, the use_diskless_load flag indicates that the
+             * replica will load the payload directly into memory without first writing it to disk. */


This comment is documenting the internal behavior of the function, which means if the function changes this comment is unlikely to get updated. I would move the comment to replicationSupportBypassCRC.

madolson · 2025-01-02T19:46:54Z

src/rdb.c

@@ -3614,6 +3624,12 @@ int rdbSaveToReplicasSockets(int req, rdbSaveInfo *rsi) {
        }
        serverSetCpuAffinity(server.bgsave_cpulist);

+        if (bypass_crc) {
+            serverLog(LL_NOTICE, "CRC checksum is disabled for this RDB transfer");
+            // mark rdb object to skip CRC checksum calculations


Suggested change

// mark rdb object to skip CRC checksum calculations

This comment is just describing the next line, which seems very self explanatory to me.

madolson · 2025-01-02T19:51:56Z

src/replication.c

@@ -2494,6 +2509,7 @@ char *sendCommand(connection *conn, ...) {
    while (1) {
        arg = va_arg(ap, char *);
        if (arg == NULL) break;
+        if (strcmp(arg, "") == 0) continue;


A few things things:

Empty strings can be valid arguments, this might break some obscure functionality like if someone set a password like an empty string. (I don't think that is allowed today, but would like to be more resistant to weird edge cases. You can introduce an arbitrary static string that points to nothing to have the same behavior.

If you continue with this path, you should use strncmp() where possible. You can actually just do if (arg[0] == '\0') as well.

talxsha force-pushed the unstable branch from b13cec9 to 468729f Compare December 23, 2024 15:44

ranshid reviewed Dec 23, 2024

View reviewed changes

ranshid reviewed Dec 24, 2024

View reviewed changes

talxsha marked this pull request as ready for review December 29, 2024 16:24

talxsha added 2 commits December 29, 2024 16:25

CRC removal during diskless full sync with TLS enabled

67fa361

Signed-off-by: Tal Shachar <[email protected]>

talxsha force-pushed the unstable branch from cb3acfd to 2e5314e Compare December 29, 2024 16:25

Merge branch 'unstable' into unstable

f33f78f

Signed-off-by: talxsha <[email protected]>

ranshid reviewed Jan 2, 2025

View reviewed changes

talxsha and others added 2 commits January 2, 2025 14:47

Merge branch 'valkey-io:unstable' into unstable

c605a83

Removed bypass_crc flag from server, renaming and comments changes

dd180dd

Signed-off-by: Tal Shachar <[email protected]>

ranshid reviewed Jan 2, 2025

View reviewed changes

src/rdb.c Outdated Show resolved Hide resolved

src/rdb.c Outdated Show resolved Hide resolved

src/rdb.c Outdated Show resolved Hide resolved

src/rdb.c Outdated Show resolved Hide resolved

ranshid added 4 commits January 2, 2025 18:23

Update src/rdb.c

6126177

Signed-off-by: ranshid <[email protected]>

Update src/rdb.c

5c34391

Signed-off-by: ranshid <[email protected]>

Update src/rdb.c

e481f73

Signed-off-by: ranshid <[email protected]>

Update src/rdb.c

14eeb6f

Signed-off-by: ranshid <[email protected]>

ranshid approved these changes Jan 2, 2025

View reviewed changes

fix format issues

266cd65

Signed-off-by: Ran Shidlansik <[email protected]>

madolson reviewed Jan 2, 2025

View reviewed changes

Merge branch 'valkey-io:unstable' into unstable

be9bb4d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CRC removal during diskless full sync with TLS enabled. #1479

CRC removal during diskless full sync with TLS enabled. #1479

talxsha commented Dec 23, 2024 •

edited

Loading

ranshid commented Dec 23, 2024

ranshid Dec 23, 2024

ranshid Dec 23, 2024

madolson Dec 24, 2024

ranshid Dec 23, 2024

ranshid Dec 23, 2024

ranshid Dec 23, 2024

ranshid Dec 23, 2024

madolson commented Dec 24, 2024

ranshid commented Dec 24, 2024

ranshid Dec 24, 2024

madolson commented Dec 24, 2024

ranshid Jan 2, 2025

ranshid Jan 2, 2025

ranshid Jan 2, 2025

ranshid Jan 2, 2025

ranshid left a comment

madolson left a comment

madolson Jan 2, 2025

madolson Jan 2, 2025

madolson Jan 2, 2025

talxsha Jan 5, 2025

madolson Jan 2, 2025

madolson Jan 2, 2025

madolson Jan 2, 2025

madolson Jan 2, 2025

talxsha Jan 5, 2025

madolson Jan 2, 2025

madolson Jan 2, 2025

madolson Jan 2, 2025

	if (disable_sync_crc_capa == 1) {
	if (disable_sync_crc_capa) {

	if (cksum == 0 \|\| (rdb->flags & RIO_FLAG_DISABLE_CRC) != 0) {
	if (cksum == 0 \|\| (rdb->flags & RIO_FLAG_DISABLE_CRC)) {

	int (*connIntegrityChecked)(void); // return 1 iff connection type has built-in integrity checks
	int (*connIntegrityChecked)(void); // return 1 if connection type has built-in integrity checks

	"total_sync_bypass_crc:%ld\r\n", server.stat_total_sync_bypass_crc,
	"total_sync_bypass_crc:%ld\r\n", server.stat_total_sync_bypass_crc,

		@@ -0,0 +1,37 @@
		start_server {tags {"repl tls"} overrides {save {}}} {

CRC removal during diskless full sync with TLS enabled. #1479

Are you sure you want to change the base?

CRC removal during diskless full sync with TLS enabled. #1479

Conversation

talxsha commented Dec 23, 2024 • edited Loading

ranshid commented Dec 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

madolson commented Dec 24, 2024

ranshid commented Dec 24, 2024

Choose a reason for hiding this comment

madolson commented Dec 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ranshid left a comment

Choose a reason for hiding this comment

madolson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

talxsha commented Dec 23, 2024 •

edited

Loading