DSP-23657: Introduce sstable encryption by szymon-miezal · Pull Request #1669 · datastax/cassandra

szymon-miezal · 2025-04-02T09:28:34Z

What is the issue

The sstable components like *Data.db, *Rows.db, *Partitions.db, *Statistics.db contain sensitive data that the customer may want to protect with encryption.

What does this PR fix and why was it fixed

This patch adds support for encrypting SSTable data, indexes (partition and row), and metadata.

Data encryption is integrated via the Cassandra compression framework.

Index and metadata encryption is implemented by updating their respective readers and writers.

It is achieved by implementing a special EncryptedSequentialWriter that
encrypts each chunk before writing, and an EncryptedChunkReader that
decrypts the chunks when reading. To enable this, the usable space in
each chunk is reduced by the size of the encryption metadata plus 8
bytes used to store the CRC and actual chunk length.

This design avoids the need to write and keep in memory a compressed
offsets map for the indices, since each chunk's offset remains the same
before and after compression.

This works well for trie data, but index chunks may also include other
data such as keys, offsets, and deletion times, which may not fit
entirely in a chunk. To handle this, the writer supports data that
spans chunk boundaries, splitting it as needed.

The random access reader is updated to skip encryption metadata during
reads using a new RebuffererFactory.adjustPosition() method.

Encryption is enabled by altering the table to specify an encryptable compressor.

Encryption keys are read from the local file system, under the directory specified by the JVM flag cassandra.system_key_directory (default: /etc/cassandra/conf).

Note: Support for generating encryption keys will be introduced in a future patch.

blambov · 2025-04-25T10:47:09Z

+                {
+                    compressionMetadata = CompressionMetadata.read(channelCopy.getFile(), sliceDescriptor, encryptionOnly);
+                    if (!encryptionOnly)
+                        overrideLength = compressionMetadata.compressedFileLength;


If the caller passes a length override, it should have priority over the one in the metadata; this is likely to make EarlyOpenCachingTest fail.

This is still valid, if a user has passed in a length override, it should take priority over this.
Perhaps make it if (overrideLength < 0) overrideLength = ...?

Note: I believe this also means that there should be no need for the !encryptionOnly check.

Fixed, running CI to double-check.

I had to keep the !encryptionOnly as otherwise sstables loaded from DSE were unreadable - I made it !encryptionOnly && overrideLength < 0.

This commit adds encryption for partition and row index data. It is achieved by implementing a special EncryptedSequentialWriter that encrypts each chunk before writing, and an EncryptedChunkReader that decrypts the chunks when reading. To enable this, the usable space in each chunk is reduced by the size of the encryption metadata plus 8 bytes used to store the CRC and actual chunk length. This design avoids the need to write and keep in memory a compressed offsets map for the indices, since each chunk's offset remains the same before and after compression. This works well for trie data, but index chunks may also include other data such as keys, offsets, and deletion times, which may not fit entirely in a chunk. To handle this, the writer supports data that spans chunk boundaries, splitting it as needed. The random access reader is updated to skip encryption metadata during reads using a new RebuffererFactory.adjustPosition() method. No additional configuration is required. The encryptor is retrieved from the data file compression settings and applied to the indices.

- Add a test that verifies the encrypted sstable are queryable - Add a test that verifies reading data without the key doesn't work

Adds a preliminary test that reads encrypted data written by DSE. There were two tables added: - one with simple PRIMARY KEY, - one with clustering columns.

That patch is inspired by DB-3845, it encrypts metadata components to ensure sensitive data is not leaked

… end Stop writing an empty compressed or encrypted chunk at the end when the file size is a multiple of the chunk size. This patch is inspired by DB-2931.

Fix the version check for metadata

…y implementations in subclasses

### What is the issue The sstable components like `*Data.db`, `*Rows.db`, `*Partitions.db`, `*Statistics.db` contain sensitive data that the customer may want to protect with encryption. ### What does this PR fix and why was it fixed This patch adds support for encrypting SSTable data, indexes (partition and row), and metadata. Data encryption is integrated via the Cassandra compression framework. Index and metadata encryption is implemented by updating their respective readers and writers. It is achieved by implementing a special EncryptedSequentialWriter that encrypts each chunk before writing, and an EncryptedChunkReader that decrypts the chunks when reading. To enable this, the usable space in each chunk is reduced by the size of the encryption metadata plus 8 bytes used to store the CRC and actual chunk length. This design avoids the need to write and keep in memory a compressed offsets map for the indices, since each chunk's offset remains the same before and after compression. This works well for trie data, but index chunks may also include other data such as keys, offsets, and deletion times, which may not fit entirely in a chunk. To handle this, the writer supports data that spans chunk boundaries, splitting it as needed. The random access reader is updated to skip encryption metadata during reads using a new RebuffererFactory.adjustPosition() method. Encryption is enabled by altering the table to specify an encryptable compressor. Encryption keys are read from the local file system, under the directory specified by the JVM flag `cassandra.system_key_directory` (default: `/etc/cassandra/conf`). Note: Support for generating encryption keys will be introduced in a future patch. --------- Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com>

This reverts commit 229854b.

The sstable components like `*Data.db`, `*Rows.db`, `*Partitions.db`, `*Statistics.db` contain sensitive data that the customer may want to protect with encryption. This patch adds support for encrypting SSTable data, indexes (partition and row), and metadata. Data encryption is integrated via the Cassandra compression framework. Index and metadata encryption is implemented by updating their respective readers and writers. It is achieved by implementing a special EncryptedSequentialWriter that encrypts each chunk before writing, and an EncryptedChunkReader that decrypts the chunks when reading. To enable this, the usable space in each chunk is reduced by the size of the encryption metadata plus 8 bytes used to store the CRC and actual chunk length. This design avoids the need to write and keep in memory a compressed offsets map for the indices, since each chunk's offset remains the same before and after compression. This works well for trie data, but index chunks may also include other data such as keys, offsets, and deletion times, which may not fit entirely in a chunk. To handle this, the writer supports data that spans chunk boundaries, splitting it as needed. The random access reader is updated to skip encryption metadata during reads using a new RebuffererFactory.adjustPosition() method. Encryption is enabled by altering the table to specify an encryptable compressor. Encryption keys are read from the local file system, under the directory specified by the JVM flag `cassandra.system_key_directory` (default: `/etc/cassandra/conf`). Note: Support for generating encryption keys will be introduced in a future patch. --------- Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com>

### What is the issue The sstable components like `*Data.db`, `*Rows.db`, `*Partitions.db`, `*Statistics.db` contain sensitive data that the customer may want to protect with encryption. ### What does this PR fix and why was it fixed This patch adds support for encrypting SSTable data, indexes (partition and row), and metadata. Data encryption is integrated via the Cassandra compression framework. Index and metadata encryption is implemented by updating their respective readers and writers. It is achieved by implementing a special EncryptedSequentialWriter that encrypts each chunk before writing, and an EncryptedChunkReader that decrypts the chunks when reading. To enable this, the usable space in each chunk is reduced by the size of the encryption metadata plus 8 bytes used to store the CRC and actual chunk length. This design avoids the need to write and keep in memory a compressed offsets map for the indices, since each chunk's offset remains the same before and after compression. This works well for trie data, but index chunks may also include other data such as keys, offsets, and deletion times, which may not fit entirely in a chunk. To handle this, the writer supports data that spans chunk boundaries, splitting it as needed. The random access reader is updated to skip encryption metadata during reads using a new RebuffererFactory.adjustPosition() method. Encryption is enabled by altering the table to specify an encryptable compressor. Encryption keys are read from the local file system, under the directory specified by the JVM flag `cassandra.system_key_directory` (default: `/etc/cassandra/conf`). Note: Support for generating encryption keys will be introduced in a future patch. Previously reviewed and merged under #1669. As a follow-up which addresses earlier CNDB regression this patch also uses a different file handle depending on write/read time. It is required by CNDB as the file writer has to access the file on local disk (the file wans't uploaded yet) and the reader need to access it via remote storage (as it does not have it at hand locally). Note: This patch does not introduce encryption for SAI indexes. --------- Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com>

### What is the issue The sstable components like `*Data.db`, `*Rows.db`, `*Partitions.db`, `*Statistics.db` contain sensitive data that the customer may want to protect with encryption. ### What does this PR fix and why was it fixed This patch adds support for encrypting SSTable data, indexes (partition and row), and metadata. Data encryption is integrated via the Cassandra compression framework. Index and metadata encryption is implemented by updating their respective readers and writers. It is achieved by implementing a special EncryptedSequentialWriter that encrypts each chunk before writing, and an EncryptedChunkReader that decrypts the chunks when reading. To enable this, the usable space in each chunk is reduced by the size of the encryption metadata plus 8 bytes used to store the CRC and actual chunk length. This design avoids the need to write and keep in memory a compressed offsets map for the indices, since each chunk's offset remains the same before and after compression. This works well for trie data, but index chunks may also include other data such as keys, offsets, and deletion times, which may not fit entirely in a chunk. To handle this, the writer supports data that spans chunk boundaries, splitting it as needed. The random access reader is updated to skip encryption metadata during reads using a new RebuffererFactory.adjustPosition() method. Encryption is enabled by altering the table to specify an encryptable compressor. Encryption keys are read from the local file system, under the directory specified by the JVM flag `cassandra.system_key_directory` (default: `/etc/cassandra/conf`). Note: Support for generating encryption keys will be introduced in a future patch. --------- Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com>

The sstable components like `*Data.db`, `*Rows.db`, `*Partitions.db`, `*Statistics.db` contain sensitive data that the customer may want to protect with encryption. This patch adds support for encrypting SSTable data, indexes (partition and row), and metadata. Data encryption is integrated via the Cassandra compression framework. Index and metadata encryption is implemented by updating their respective readers and writers. It is achieved by implementing a special EncryptedSequentialWriter that encrypts each chunk before writing, and an EncryptedChunkReader that decrypts the chunks when reading. To enable this, the usable space in each chunk is reduced by the size of the encryption metadata plus 8 bytes used to store the CRC and actual chunk length. This design avoids the need to write and keep in memory a compressed offsets map for the indices, since each chunk's offset remains the same before and after compression. This works well for trie data, but index chunks may also include other data such as keys, offsets, and deletion times, which may not fit entirely in a chunk. To handle this, the writer supports data that spans chunk boundaries, splitting it as needed. The random access reader is updated to skip encryption metadata during reads using a new RebuffererFactory.adjustPosition() method. Encryption is enabled by altering the table to specify an encryptable compressor. Encryption keys are read from the local file system, under the directory specified by the JVM flag `cassandra.system_key_directory` (default: `/etc/cassandra/conf`). Note: Support for generating encryption keys will be introduced in a future patch. Previously reviewed and merged under #1669. As a follow-up which addresses earlier CNDB regression this patch also uses a different file handle depending on write/read time. It is required by CNDB as the file writer has to access the file on local disk (the file wans't uploaded yet) and the reader need to access it via remote storage (as it does not have it at hand locally). Note: This patch does not introduce encryption for SAI indexes. --------- Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com>

### What is the issue The sstable components like `*Data.db`, `*Rows.db`, `*Partitions.db`, `*Statistics.db` contain sensitive data that the customer may want to protect with encryption. ### What does this PR fix and why was it fixed This patch adds support for encrypting SSTable data, indexes (partition and row), and metadata. Data encryption is integrated via the Cassandra compression framework. Index and metadata encryption is implemented by updating their respective readers and writers. It is achieved by implementing a special EncryptedSequentialWriter that encrypts each chunk before writing, and an EncryptedChunkReader that decrypts the chunks when reading. To enable this, the usable space in each chunk is reduced by the size of the encryption metadata plus 8 bytes used to store the CRC and actual chunk length. This design avoids the need to write and keep in memory a compressed offsets map for the indices, since each chunk's offset remains the same before and after compression. This works well for trie data, but index chunks may also include other data such as keys, offsets, and deletion times, which may not fit entirely in a chunk. To handle this, the writer supports data that spans chunk boundaries, splitting it as needed. The random access reader is updated to skip encryption metadata during reads using a new RebuffererFactory.adjustPosition() method. Encryption is enabled by altering the table to specify an encryptable compressor. Encryption keys are read from the local file system, under the directory specified by the JVM flag `cassandra.system_key_directory` (default: `/etc/cassandra/conf`). Note: Support for generating encryption keys will be introduced in a future patch. --------- Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com>

The sstable components like `*Data.db`, `*Rows.db`, `*Partitions.db`, `*Statistics.db` contain sensitive data that the customer may want to protect with encryption. This patch adds support for encrypting SSTable data, indexes (partition and row), and metadata. Data encryption is integrated via the Cassandra compression framework. Index and metadata encryption is implemented by updating their respective readers and writers. It is achieved by implementing a special EncryptedSequentialWriter that encrypts each chunk before writing, and an EncryptedChunkReader that decrypts the chunks when reading. To enable this, the usable space in each chunk is reduced by the size of the encryption metadata plus 8 bytes used to store the CRC and actual chunk length. This design avoids the need to write and keep in memory a compressed offsets map for the indices, since each chunk's offset remains the same before and after compression. This works well for trie data, but index chunks may also include other data such as keys, offsets, and deletion times, which may not fit entirely in a chunk. To handle this, the writer supports data that spans chunk boundaries, splitting it as needed. The random access reader is updated to skip encryption metadata during reads using a new RebuffererFactory.adjustPosition() method. Encryption is enabled by altering the table to specify an encryptable compressor. Encryption keys are read from the local file system, under the directory specified by the JVM flag `cassandra.system_key_directory` (default: `/etc/cassandra/conf`). Note: Support for generating encryption keys will be introduced in a future patch. Previously reviewed and merged under #1669. As a follow-up which addresses earlier CNDB regression this patch also uses a different file handle depending on write/read time. It is required by CNDB as the file writer has to access the file on local disk (the file wans't uploaded yet) and the reader need to access it via remote storage (as it does not have it at hand locally). Note: This patch does not introduce encryption for SAI indexes. --------- Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com>

The sstable components like `*Data.db`, `*Rows.db`, `*Partitions.db`, `*Statistics.db` contain sensitive data that the customer may want to protect with encryption. This patch adds support for encrypting SSTable data, indexes (partition and row), and metadata. Data encryption is integrated via the Cassandra compression framework. Index and metadata encryption is implemented by updating their respective readers and writers. It is achieved by implementing a special EncryptedSequentialWriter that encrypts each chunk before writing, and an EncryptedChunkReader that decrypts the chunks when reading. To enable this, the usable space in each chunk is reduced by the size of the encryption metadata plus 8 bytes used to store the CRC and actual chunk length. This design avoids the need to write and keep in memory a compressed offsets map for the indices, since each chunk's offset remains the same before and after compression. This works well for trie data, but index chunks may also include other data such as keys, offsets, and deletion times, which may not fit entirely in a chunk. To handle this, the writer supports data that spans chunk boundaries, splitting it as needed. The random access reader is updated to skip encryption metadata during reads using a new RebuffererFactory.adjustPosition() method. Encryption is enabled by altering the table to specify an encryptable compressor. Encryption keys are read from the local file system, under the directory specified by the JVM flag `cassandra.system_key_directory` (default: `/etc/cassandra/conf`). Note: Support for generating encryption keys will be introduced in a future patch. Previously reviewed and merged under #1669. As a follow-up which addresses earlier CNDB regression this patch also uses a different file handle depending on write/read time. It is required by CNDB as the file writer has to access the file on local disk (the file wans't uploaded yet) and the reader need to access it via remote storage (as it does not have it at hand locally). Note: This patch does not introduce encryption for SAI indexes. --------- Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com> (Rebase of commit ab9fb4c)

szymon-miezal force-pushed the DSP-23657 branch 2 times, most recently from cc55e61 to a908d2d Compare April 7, 2025 14:57

szymon-miezal force-pushed the DSP-23657 branch from a908d2d to 91511e4 Compare April 9, 2025 16:04

szymon-miezal force-pushed the DSP-23657 branch 4 times, most recently from 489642a to 1f82b43 Compare April 22, 2025 11:43

szymon-miezal changed the title ~~[WIP] DSP-23657: TDE port POC~~ [WIP] DSP-23657: Introduce data, row and partition index encryption Apr 22, 2025

szymon-miezal changed the title ~~[WIP] DSP-23657: Introduce data, row and partition index encryption~~ [WIP] DSP-23657: Introduce sstable encryption Apr 22, 2025

szymon-miezal changed the title ~~[WIP] DSP-23657: Introduce sstable encryption~~ DSP-23657: Introduce sstable encryption Apr 22, 2025

blambov self-requested a review April 23, 2025 12:54

blambov reviewed Apr 25, 2025

View reviewed changes

szymon-miezal force-pushed the DSP-23657 branch from 7a8bcdd to ebcbdc3 Compare May 5, 2025 13:54

szymon-miezal and others added 17 commits May 16, 2025 13:13

DSP-23657: Resolve TODOs

fd062a1

DSP-23657: Add E2E tests for data and index encryption

8e36292

- Add a test that verifies the encrypted sstable are queryable - Add a test that verifies reading data without the key doesn't work

DSP-23657: Test reading encrypted sstables created by DSE

2295cae

Adds a preliminary test that reads encrypted data written by DSE. There were two tables added: - one with simple PRIMARY KEY, - one with clustering columns.

DSP-23657: Encrypt metadata components

5effa37

That patch is inspired by DB-3845, it encrypts metadata components to ensure sensitive data is not leaked

DSP-23657: Stop writing an empty compressed or encrypted chunk at the…

031e748

… end Stop writing an empty compressed or encrypted chunk at the end when the file size is a multiple of the chunk size. This patch is inspired by DB-2931.

DSP-23657: Use checksums as a encryption verifier in a test

004ffd2

DSP-23657: Add a test that checks reading fails with a different key

e8bf074

Drop unused EncryptionKeyBackup

a6e2688

Avoid calling thread local twice

25ae98e

Avoid calling for IV length twice

e00ed7e

Add comments explaining versions used in {indices|metadata}AreEncrypted

2d62a90

Fix the version check for metadata

Revert trie version to cc

63b1010

Resolve position vs chunkOffset TODO

360caaa

Remove dev encryption/decryption logs

9f1aca5

Resolve pool type TODO

278ce95

Default RebuffererFactory.invalidateIfCached to empty and remove empt…

5b1d150

…y implementations in subclasses

szymon-miezal deleted the DSP-23657 branch May 23, 2025 07:36

szymon-miezal added a commit that referenced this pull request May 23, 2025

Revert "DSP-23657: Introduce sstable encryption (#1669)"

425d83f

This reverts commit 229854b.

szymon-miezal mentioned this pull request Jun 13, 2025

DSP-23657: Introduce sstable encryption (v2) #1795

Merged

driftx mentioned this pull request Jul 28, 2025

DSP-23657: Introduce sstable encryption (v2) (#1795) #1926

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DSP-23657: Introduce sstable encryption#1669

DSP-23657: Introduce sstable encryption#1669
szymon-miezal merged 30 commits intomainfrom
DSP-23657

szymon-miezal commented Apr 2, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blambov Apr 25, 2025

Uh oh!

blambov May 16, 2025 •

edited

Loading

Uh oh!

szymon-miezal May 19, 2025

Uh oh!

szymon-miezal May 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

szymon-miezal commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the issue

What does this PR fix and why was it fixed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blambov Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

blambov May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szymon-miezal May 19, 2025

Choose a reason for hiding this comment

Uh oh!

szymon-miezal May 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

szymon-miezal commented Apr 2, 2025 •

edited

Loading

blambov May 16, 2025 •

edited

Loading