[SPARK-48742][SS] Virtual Column Family for RocksDB #47107
+591
−191
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jingz-db
changed the title
jingz-db
commented
Jun 29, 2024
| @@ -126,6 +126,7 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with | |||
| valueSchema: StructType, | |||
| keyStateEncoderSpec: KeyStateEncoderSpec, | |||
| useMultipleValuesPerKey: Boolean = false, | |||
There was a problem hiding this comment.
TODO: as discussed with Anish today, will made this a ENUM type and passed into RocksDB
jingz-db
commented
Jun 29, 2024
| @@ -2168,6 +2168,15 @@ object SQLConf { | |||
| .checkValue(v => Set(1, 2).contains(v), "Valid versions are 1 and 2") | |||
| .createWithDefault(2) | |||
|
|
|||
| val STREAMING_ROCKSDB_VIRTUAL_COL_FAMILY_ENABLED = | |||
There was a problem hiding this comment.
TODO: remove this config, and pass virtual col family as an argument similar as useColumnFamilies per operator based
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Introducing virtual column family to RocksDB. We attach an 8-byte-Id prefix as column family identifier for each of the key row that is put into RocksDB. The encoding and decoding of the virtual column family prefix happens at the
RocksDBKeyEncoderlayer as we can pre-allocate extra 8 bytes and avoid additional memcpy.Why are the changes needed?
Currently within the scope of the arbitrary stateful API v2 (transformWithState) project, each state variable is stored inside one physical column family within the RocksDB state store instance. Column families are also used to implement secondary indexes for various features. Each physical column family has its own memtables, creates its own SST files, and handles compaction independently on those independent SST files.
When the number of operations to RocksDB is relatively small and the number of column families is relatively large (what is considered to be “small” and “large” will be discussed in the benchmark results in the following sections), the overhead of handling small SST files becomes high, especially since all of these have to be uploaded in the snapshot dir and referenced in the metadata file for the uploaded RocksDB snapshot. Using prefix to manage different key spaces / virtual column family could reduce such overheads.
Does this PR introduce any user-facing change?
TODO: decide how we are exposing the Virtual Col Family Config to users
How was this patch tested?
Unit tests in
RocksDBStateStoreSuite, and integration tests inTransformWithStateSuiteWas this patch authored or co-authored using generative AI tooling?
No.