Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ducktape: Increase partition density to 3k per shard in the MPT #24710

Merged
merged 2 commits into from
Jan 17, 2025
Merged

ducktape: Increase partition density to 3k per shard in the MPT #24710

merged 2 commits into from
Jan 17, 2025

Conversation

StephanDollberg
Copy link
Member

@StephanDollberg StephanDollberg commented Jan 7, 2025 •
edited
Loading

Increase the partition density in the MPT to 3k per shard and use the new memory group aware config.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

@travisdowns
Copy link
Member

Partial fix for https://redpandadata.atlassian.net/browse/CORE-8732.

travisdowns
travisdowns reviewed Jan 7, 2025
View reviewed changes
tests/rptest/scale_tests/many_partitions_test.py
@@ -43,12 +43,16 @@
BIG_FETCH = 104857600

# How much memory to assign to redpanda per partition. Redpanda will be started
# with MIB_PER_PARTITION * PARTITIONS_PER_SHARD * CORE_COUNT memory
DEFAULT_MIB_PER_PARTITION = 4
# with (MIB_PER_PARTITION * PARTITIONS_PER_SHARD * CORE_COUNT) / (PARTITIONS_MEMORY_ALLOCATION_PERCENT / 100) memory
Copy link
Member

@travisdowns travisdowns Jan 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be / (PARTITIONS_MEMORY_ALLOCATION_PERCENT be * / (PARTITIONS_MEMORY_ALLOCATION_PERCENT)?

Copy link
Member Author

@StephanDollberg StephanDollberg Jan 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

* / (PARTITIONS_MEMORY_ALLOCATION_PERCENT)

Do you mean * 100 / PARTITIONS_MEMORY_ALLOCATION_PERCENT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I meant * (PARTITIONS_MEMORY_ALLOCATION_PERCENT)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was just wrong though, it should be / as I see now this is calculating from the partition memory alloc % to the amount RP should be started with.

travisdowns
travisdowns reviewed Jan 7, 2025
View reviewed changes
tests/rptest/scale_tests/many_partitions_test.py Outdated

# How many partitions we will create per shard: this is the primary scaling
# factor that controls how many partitions a given cluster will get.
DEFAULT_PARTITIONS_PER_SHARD = 1000
DEFAULT_PARTITIONS_PER_SHARD = 2000
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost feels like we should read it from config, but anyway LGTM.

@StephanDollberg StephanDollberg changed the title ducktape: Increase partition density to 2k per shard in the MPT ducktape: Increase partition density to 3k per shard in the MPT Jan 10, 2025
@StephanDollberg
ducktape: Increase partition density in the MPT
279adf7 
Increase partition density to 3k partitions per shard and adapt the test
to use the new memory group aware config.
@StephanDollberg
ducktape: Increase kgo-repeater status timeout
49e4fb0 
kgo-repeaters start to get pretty loaded at higher partition densities
so the status query can take a bit longer.
@StephanDollberg
Copy link
Member Author

Pushed the change to 3k partitions now given a fix is up.

@vbotbuildovich
Copy link
Collaborator

CI test results

test results on build#60606
test_id test_kind job_url test_status passed
rptest.tests.partition_reassignments_test.PartitionReassignmentsTest.test_reassignments_kafka_cli ducktape https://buildkite.com/redpanda/redpanda/builds/60606#0194517f-732e-4c69-8145-3610850c59c7 FLAKY 3/6

@StephanDollberg StephanDollberg merged commit 564fbb5 into dev Jan 17, 2025
16 checks passed
@StephanDollberg StephanDollberg deleted the stephan/mpt-new-limits branch January 17, 2025 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@travisdowns travisdowns travisdowns approved these changes

@ballard26 ballard26 Awaiting requested review from ballard26

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@StephanDollberg @travisdowns @vbotbuildovich