[TPC-H] Adjust default cluster sizes #1370
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I think the argument for scale 10 000 was that this is one of the features of distributed frameworks that you can scale out |
That makes sense, but then we should, at the very least, highlight that it's not a fair comparison. |
| @@ -158,15 +158,15 @@ def cluster_spec(scale): | |||
| } | |||
| elif scale == 1000: | |||
| return { | |||
| "worker_vm_types": ["m6i.large"], | |||
| "worker_vm_types": ["m6i.xlarge"], | |||
There was a problem hiding this comment.
I noted that pyspark query 21 would always fail with this setup, but will succeed with m6i.2xlarge.
Maybe that's okay though? If we succeed with this and pyspark mostly succeeds w/ other queries except one, then too bad for them(?).
milesgranger
approved these changes
Feb 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adjusts default cluster sizes following changes suggested in #1166.
Changes:
Scale 10
m6i.8xlarge(32 CPUs, 128 GiB RAM) tom6i.4xlarge(16 CPUs, 64 GiB RAM) to match.Scale 100
No changes
Scale 1000
m6i.large(2 CPUs, 8 GiB RAM) tom6i.xlarge(4 CPUs, 16 GiB RAM).During our latest runs, we've noticed that the configuration for scale 1000 is not large enough and needs to be doubled.
m6i.16xlarge(64 CPUs, 256 GiB RAM) tom6i.32xlarge(128 CPUs, 256 GiB RAM) to match.Scale 10000
Note: We have not run this yet but this is the only way to keep CPU count constant as there's no larger
m6iVM available for the single VM types.