Skip to content

CASSANDRA-20918 Add cursor-based low allocation optimized compaction implementation #4402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: trunk
Choose a base branch
from
+13,118 −459
Open

CASSANDRA-20918 Add cursor-based low allocation optimized compaction implementation #4402

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
9368327
Add cursor based optimized compaction path (WIP)
Jun 22, 2025
759dc2c
Merge branch 'trunk' into compaction-work-pr-prep
nitsanw Sep 29, 2025
d75d59c
Fix MEGABYTE constant
Oct 9, 2025
35ea514
Fix 0x11/0x10 should be 0b11/0b10
Oct 14, 2025
33b9e47
Introduce ENABLE_CURSOR_COMPACTION controls via CassandraRelevantProp…
Oct 14, 2025
d46511d
Revert `matched` comments left to track stats tracking
Oct 14, 2025
b417274
Revert change to RandomAccessReader and match row skipping logic from…
Oct 14, 2025
85379f1
Improve CompactionCursor javadoc
Oct 14, 2025
097b995
Extract `isSupported` from constructor
Oct 14, 2025
1af0f0e
Fix Trasnactions/Transformations
Oct 15, 2025
7576864
Add comment and rename `sortForPartitionMerge` -> `prepareForPartitio…
Oct 15, 2025
2205e48
Typo: `preturbed` -> `perturbed`
Oct 15, 2025
dea15c9
Javadoc for bubbleInsertToPreSorted and minor refactor
Oct 15, 2025
5d200ef
Typo: passed -> past
Oct 15, 2025
1a1c761
Remove redundant TODOs
Oct 15, 2025
b9dc6db
Revert 'unused' params
Oct 20, 2025
5accdd5
Rename `ElementDescriptor` -> `UnfilteredDescriptor` (and fallout)
Oct 20, 2025
17be5f4
Remove unused parameter
Oct 20, 2025
3b0f0a9
Remove unused method
Oct 20, 2025
ae65b7e
Fix indentation
Oct 20, 2025
62837f9
Move SSTableCursorPipeUtil to benchmarks
Oct 20, 2025
4064d32
Rename `partitionLength` back to `finishResult` and clarify comment
Oct 20, 2025
797ace1
Remove unused methods
Oct 20, 2025
855a740
Improve bubbleInsertElementToPreSorted, delay element insert
Oct 21, 2025
beef9ad
Remove redundant cursor status check
Oct 21, 2025
b19b959
Simplify deletion merging loop, clarify partitionDeletion variable names
Oct 21, 2025
f5718b3
Neaten up SSTableCursorReader
Oct 22, 2025
87a1c1c
Dead code removal
Oct 22, 2025
ff5eff1
Revert making classes public
Oct 22, 2025
ed48fe5
Transform LivenessInfo an interface
Oct 22, 2025
a8bc083
Fix javadoc
Oct 22, 2025
be8b4e3
Fix intellij warnings
Oct 22, 2025
011213a
Explicitly split DeletionTime implementations
Oct 29, 2025
4d74fce
Rely on nextElementEquality in findMergeLimit
Oct 30, 2025
b9f5802
Refactor prepareAndSortForMerge code
Oct 30, 2025
308a3b3
Move merge limit == 0 out of mergeRows
Oct 31, 2025
562e437
Add TODO for clustering read/skip
Oct 31, 2025
37104f6
Simplify ClusteringComparator code and remove redundant code
Nov 4, 2025
d7abcb6
Refactor ClusteringDescriptor for clarity and fix MetadataCollector i…
Nov 4, 2025
8e24365
Add comment regarding boundary
Nov 4, 2025
bc27a3d
Split checkNextFlags by context and tidy-up
Nov 4, 2025
16f2eee
Use RuntimeException instead of ISE
Nov 5, 2025
80007eb
Improve CompactionCursor javadoc
Nov 5, 2025
cf77915
Remove "system" table check
Nov 5, 2025
ff8ab58
Check output format for `isSupported`, improve unsupported reason log…
Nov 5, 2025
647f3b6
Move partition start out of maybePurgedOutputDeletion
Nov 5, 2025
35af776
Improve prepareAndSortForPartitionMerge javadoc, avoid repeated reset…
Nov 5, 2025
2e8dff9
Change option name enable_cursor_compaction -> cursor_compaction_enabled
Nov 6, 2025
d2b23d6
Add option cursor_compaction_enabled to testlist-oa
Nov 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/java/org/apache/cassandra/config/CassandraRelevantProperties.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,7 @@ public enum CassandraRelevantProperties
DTEST_IS_IN_JVM_DTEST("org.apache.cassandra.dtest.is_in_jvm_dtest"),
/** In_JVM dtest property indicating that the test should use "latest" configuration */
DTEST_JVM_DTESTS_USE_LATEST("jvm_dtests.latest"),
ENABLE_CURSOR_COMPACTION("cassandra.enable_cursor_compaction", "true"),
ENABLE_DC_LOCAL_COMMIT("cassandra.enable_dc_local_commit", "true"),
/**
* Whether {@link org.apache.cassandra.db.ConsistencyLevel#NODE_LOCAL} should be allowed.
Expand Down
3 changes: 3 additions & 0 deletions src/java/org/apache/cassandra/config/Config.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@

import static org.apache.cassandra.config.CassandraRelevantProperties.AUTOCOMPACTION_ON_STARTUP_ENABLED;
import static org.apache.cassandra.config.CassandraRelevantProperties.CASSANDRA_AVAILABLE_PROCESSORS;
import static org.apache.cassandra.config.CassandraRelevantProperties.ENABLE_CURSOR_COMPACTION;
import static org.apache.cassandra.config.CassandraRelevantProperties.FILE_CACHE_ENABLED;
import static org.apache.cassandra.config.CassandraRelevantProperties.SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE;
import static org.apache.cassandra.config.CassandraRelevantProperties.SKIP_PAXOS_REPAIR_ON_TOPOLOGY_CHANGE_KEYSPACES;
Expand Down Expand Up @@ -644,6 +645,8 @@ public static class SSTableConfig
@Replaces(oldName = "enable_drop_compact_storage", converter = Converters.IDENTITY, deprecated = true)
public volatile boolean drop_compact_storage_enabled = false;

public boolean enable_cursor_compaction = ENABLE_CURSOR_COMPACTION.getBoolean();

public volatile boolean use_statements_enabled = true;

/**
Expand Down
11 changes: 11 additions & 0 deletions src/java/org/apache/cassandra/config/DatabaseDescriptor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4642,6 +4642,17 @@ public static void setTransientReplicationEnabledUnsafe(boolean enabled)
conf.transient_replication_enabled = enabled;
}

public static boolean enableCursorCompaction()
{
return conf.enable_cursor_compaction;
}

@VisibleForTesting
public static void setEnableCursorCompaction(boolean enable_cursor_compaction)
{
conf.enable_cursor_compaction = enable_cursor_compaction;
}

public static boolean enableDropCompactStorage()
{
return conf.drop_compact_storage_enabled;
Expand Down
3 changes: 2 additions & 1 deletion src/java/org/apache/cassandra/db/compaction/AbstractCompactionPipeline.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

package org.apache.cassandra.db.compaction;

import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.AbstractCompactionController;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.Directories;
Expand All @@ -31,7 +32,7 @@

abstract class AbstractCompactionPipeline extends CompactionInfo.Holder implements AutoCloseable {
static AbstractCompactionPipeline create(CompactionTask task, OperationType type, AbstractCompactionStrategy.ScannerList scanners, AbstractCompactionController controller, long nowInSec, TimeUUID compactionId) {
if (CompactionTask.CURSOR_COMPACTION_ENABLED) {
if (DatabaseDescriptor.enableCursorCompaction()) {
try {
return new CursorCompactionPipeline(task, type, scanners, controller, nowInSec, compactionId);
} catch (IllegalArgumentException e) {
Expand Down
2 changes: 0 additions & 2 deletions src/java/org/apache/cassandra/db/compaction/CompactionTask.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@
import org.apache.cassandra.dht.Range;
import org.apache.cassandra.dht.Token;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.SystemProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down Expand Up @@ -74,7 +73,6 @@ public class CompactionTask extends AbstractCompactionTask
{
private static final int MEGABYTE = 1024 * 1024;
protected static final Logger logger = LoggerFactory.getLogger(CompactionTask.class);
public static final boolean CURSOR_COMPACTION_ENABLED = SystemProperties.getBoolean("cassandra.enable_cursor_compaction", () -> true);

protected final long gcBefore;
protected final boolean keepOriginals;
Expand Down