Setup a nightly runs for semantic search at noaa workload

[martin-gaievski]

Is your feature request related to a problem? Please describe

Similar to https://opensearch.org/benchmarks/ , we would like to set up nightly runs that runs semantic search workloads every night, publish performance metrics, and make it available for public.

Describe the solution you'd like

For nightly run we need to get results for following configuration:

Cluster configuration:

• 3 data nodes of type r5.4xlarge or similar, with 16 vCPU cores, 64Gb+ RAM
• 1 to 3 leader nodes

2 different run for cases when concurrent_segment_search is enabled and disabled.

Workload 1: Concurrent_segment_search disabled:
Workload test procedure:

• hybrid-query-aggs-full

Workload parameters:

• number_of_shards:6
• max_num_segments:8
• concurrent_segment_search_enabled:'false'

Workload 2: Concurrent_segment_search enabled:
Workload test procedure:

• hybrid-query-aggs-full

Workload parameters:

• number_of_shards:6
• max_num_segments:8
• concurrent_segment_search_enabled:'true'

Describe alternatives you've considered

No response

Additional context

Workload is part of the opensearch-benchmarks-workloads repo, thus results should be easily reproducible by users.

[martin-gaievski]

For reference we can use following command to run workloads:

workload 1:

opensearch-benchmark execute-test --workload="noaa_semantic_search" --test-procedure=hybrid-query-aggs-full --pipeline=benchmark-only --target-host=http://myserver:80 --kill-running-processes --workload-params="number_of_shards:6,max_num_segments:8,concurrent_segment_search_enabled:'false'"

workload 2:

opensearch-benchmark execute-test --workload="noaa_semantic_search" --test-procedure=hybrid-query-aggs-full --pipeline=benchmark-only --target-host=http://myserver:80 --kill-running-processes --workload-params="number_of_shards:6,max_num_segments:8,concurrent_segment_search_enabled:'true'"

[rishabh6788]

The nightly runs have been scheduled. Will wait for this week to generate enough data to create public dashboards.

[martin-gaievski]

@rishabh6788 Can we please make few changes to this setup:

1. change ec2 instance type of data node to r5.2xlarge

2. change command for running workload:

opensearch-benchmark execute-test --workload="noaa_semantic_search" --test-procedure=hybrid-query-aggs-full --pipeline=benchmark-only --target-host=http://myserver:80 --kill-running-processes --workload-params="/noaa_semantic_search/params/one_replica_no_concurrent_segment_search.json"

only different comparing to what we have today is reference to params from the file vs defining params as part of the command

3. add one more workload run on the same cluster configuration with following command:

opensearch-benchmark execute-test --workload="noaa_semantic_search" --test-procedure=hybrid-query-aggs-full --pipeline=benchmark-only --target-host=http://myserver:80 --kill-running-processes --workload-params="/noaa_semantic_search/params/one_replica_with_concurrent_segment_search.json"

Related changes are merged to workloads repo: opensearch-project/opensearch-benchmark-workloads#319

[rishabh6788]

Closing this issue. Please re-open if you have any comments or concerns.