Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add r5.8xlarge instance runs for concurrent search baseline #4584

Merged
merged 2 commits into from
Mar 27, 2024
Merged

Add r5.8xlarge instance runs for concurrent search baseline #4584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4b7ab97
Add r5.8xlarge instance runs for concurrent search baseline
rishabh6788 Mar 27, 2024
12911e4
Merge branch 'opensearch-project:main' into main
rishabh6788 Mar 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions jenkins/opensearch/benchmark-test.jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,6 @@ pipeline {
H 14 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-concurrent-search-0-slice-1-shard-0-replica-4-clients-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:0;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard","search_clients":4};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 14 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.8xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.8xlarge,major-version:2x,cluster-config:x64-r5.8xlarge-concurrent-search-0-slice-1-shard-0-replica-2-clients-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:0;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard","search_clients":2};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 14 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.8xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.8xlarge,major-version:2x,cluster-config:x64-r5.8xlarge-concurrent-search-0-slice-1-shard-0-replica-4-clients-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:0;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard","search_clients":4};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 14 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-concurrent-search-0-slice-1-shard-0-replica-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:0;WORKLOAD_PARAMS={"number_of_replicas":"0","number_of_shards":"1"};CAPTURE_NODE_STAT=true;TEST_PROCEDURE=aggs

H 15 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=http_logs;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-concurrent-search-4-slice-1-shard-0-replica-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:4;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"http_logs_1_shard"};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 15 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=big5;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-concurrent-search-4-slice-1-shard-0-replica-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:4;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"big5_1_shard"};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
Expand All @@ -131,7 +130,6 @@ pipeline {
H 15 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-concurrent-search-4-slice-1-shard-0-replica-4-clients-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:4;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard","search_clients":4};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 15 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.8xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.8xlarge,major-version:2x,cluster-config:x64-r5.8xlarge-concurrent-search-4-slice-1-shard-0-replica-2-clients-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:4;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard","search_clients":2};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 15 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.8xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.8xlarge,major-version:2x,cluster-config:x64-r5.8xlarge-concurrent-search-4-slice-1-shard-0-replica-4-clients-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:4;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard","search_clients":4};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 15 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-concurrent-search-4-slice-1-shard-0-replica-snapshot;ADDITIONAL_CONFIG=search.concurrent_segment_search.enabled:true search.concurrent.max_slice_count:4;WORKLOAD_PARAMS={"number_of_replicas":"0","number_of_shards":"1"};CAPTURE_NODE_STAT=true;TEST_PROCEDURE=aggs

H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=http_logs;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-1-shard-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"http_logs_1_shard"};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=big5;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-1-shard-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"big5_1_shard"};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
Expand All @@ -140,7 +138,10 @@ pipeline {
H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-1-shard-0-replica-4-clients-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard","search_clients":4};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-2-shard-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard"};TEST_PROCEDURE=restore-from-snapshot-2-shards;CAPTURE_NODE_STAT=true
H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-5-shard-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard"};TEST_PROCEDURE=restore-from-snapshot-5-shards;DATA_NODE_STORAGE=300;CAPTURE_NODE_STAT=true
H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.2xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.2xlarge,major-version:2x,cluster-config:x64-r5.2xlarge-1-shard-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_replicas":"0","number_of_shards":"1"};CAPTURE_NODE_STAT=true;TEST_PROCEDURE=aggs
H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.8xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.8xlarge,major-version:2x,cluster-config:x64-r5.8xlarge-1-shard-0-replica-2-clients-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard","search_clients":2};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.8xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.8xlarge,major-version:2x,cluster-config:x64-r5.8xlarge-1-shard-0-replica-4-clients-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard","search_clients":4};TEST_PROCEDURE=restore-from-snapshot;CAPTURE_NODE_STAT=true
H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.8xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.8xlarge,major-version:2x,cluster-config:x64-r5.8xlarge-2-shard-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard"};TEST_PROCEDURE=restore-from-snapshot-2-shards;CAPTURE_NODE_STAT=true
H 16 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.13.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=true;DATA_INSTANCE_TYPE=r5.8xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.8xlarge,major-version:2x,cluster-config:x64-r5.8xlarge-5-shard-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_1_shard"};TEST_PROCEDURE=restore-from-snapshot-5-shards;DATA_NODE_STORAGE=300;CAPTURE_NODE_STAT=true
'''
)
}
Expand Down
Loading
Loading