Multithreaded BGSave

[madolson]

One of the bottlenecks that we often observe in AWS is the time it takes to do a full sync either when replacing a node or bootstrapping a new nodes into a cluster. We can partially solve this bottleneck by supporting the usage of additional threads for a multi-threaded save and restore. The save can be trivially parallelized by either having each thread own a subset of the slots (for cluster mode) or a subset of the hash table buckets (for standalone). The restore is a bit less trivial, but by properly setting the initial state of the dictionary, we should be able to safely load data into it from multiple threads.

Each of these threads can use its own TCP connection for transmitting the full-sync data. The primary and replica will need to negotiate the number of RDB full sync connections to use.

[ranshid]

Each of these threads can use its own TCP connection for transmitting the full-sync data. The primary and replica will need to negotiate the number of RDB full sync connections to use.

Does your suggestion here implies that the feature will only work for rd-channel enabled primary/replica?

I think we can also look at some ways we can use a single multiplexing connection to pass all traffic on a single stream. For example QUIC can help logically separate the data sent by each thread OR we can also logically multiplex the slot/bucket data using a dedicated headers which will proceed each data chunk (like crc is).

[madolson]

I think we can also look at some ways we can use a single multiplexing connection to pass all traffic on a single stream. For example QUIC can help logically separate the data sent by each thread OR we can also logically multiplex the slot/bucket data using a dedicated headers which will proceed each data chunk (like crc is).

I suppose this is related, #1300.

[PingXie]

I like this idea!

By the way, what you described in this issue is the diskless full sync. I think there's also value in speeding up the RDB snapshot (to disk). We'll likely need to use a "scatter-gather" solution that writes different parts of the keyspace in parallel to separate files and then concatenates them to form the final RDB file.

[xbasel]

Currently, BGSAVE is not the bottleneck. The limiting factor is on the replica side during RDB processing. There's ongoing work to improve replica-side RDB handling (I'll open an issue and update here). Until then, there's little benefit in optimizing BGSAVE further. (there's still benefit when saving to file though)

[madolson]

Currently, BGSAVE is not the bottleneck. The limiting factor is on the replica side during RDB processing. There's ongoing work to improve replica-side RDB handling (I'll open an issue and update here). Until then, there's little benefit in optimizing BGSAVE further. (there's still benefit when saving to file though)

My intention with this issue is that it covers both the primary and replica side. I assumed we would send it across multiple links, so the replica would also need multiple threads handling and parsing the input.

[Nicky-2000]

Hi! I am interested in working on this issue. @PingXie could you assign this to me please? Thanks!

[Nicky-2000]

Hi everyone!

I have made a PR #2470 that introduces Multi Threaded RDB Save/BGSave. The performance testing results were really positive with multi-threading giving us a 3x+ speedup on many workloads! Please see the comment I left on the PR for the full performance results and design overview.

Right now for Full Sync we do not open up multiple connections to the replica. Instead we parallelized the computation and stream all data to a single "target" Rio. That being said, this design makes transitioning to multiple connections between Primary and Replica feasible so we can push towards that in the future.

One thing I want to emphasize is that with a large number of threads, multi-threading essentially offsets the computational cost of compression, making it negligible. So it could be worth it to turn compression on or use a heavier compression algorithm when multi-thread RDB Save is on. Doing this could result in RDB Save running faster with compression enabled compared to when it is disabled due to the smaller file size.

If anyone has some time to review the code I would really appreciate it!

Also, just for visibility, I am currently working on multi-threading the RDB Load functionality to give us an end-to-end multi-threaded full sync.

[madolson]

I have made a PR #2470 that introduces Multi Threaded RDB Save/BGSave. The performance testing results were really positive with multi-threading giving us a 3x+ speedup on many workloads! Please see the comment I left on the PR for the full performance results and design overview.

Have you done any perf testing to see where the bottleneck is? When we tested it at AWS, we were able to get about a 60x with multiple connections, so maybe it's worth doing now?

One thing I want to emphasize is that with a large number of threads, multi-threading essentially offsets the computational cost of compression, making it negligible. So it could be worth it to turn compression on or use a heavier compression algorithm when multi-thread RDB Save is on. Doing this could result in RDB Save running faster with compression enabled compared to when it is disabled due to the smaller file size.

@sarthakaggarwal97 Would you take a look? I know you were looking at updating the compression.

[sarthakaggarwal97]

@sarthakaggarwal97 Would you take a look? I know you were looking at updating the compression.

Yes, I am planning to take a look. I took an initial glance, and the tests have been conducted with valkey-benchmark, where none of the values aren't compressed using LZF or LZ4 irrespective of value size. I would be interested in those scenarios as well where the workload is kind of real-world (with legible words, JSON structure maybe?) and compressible. Also, I am curious about at how many threads do we see this behaviour.