Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RELEASE] Release version 2.4.0 #2649

Closed
42 of 52 tasks
github-actions bot opened this issue Sep 20, 2022 · 17 comments
Closed
42 of 52 tasks

[RELEASE] Release version 2.4.0 #2649

github-actions bot opened this issue Sep 20, 2022 · 17 comments
Assignees
Labels
release v2.4.0 'Issues and PRs related to version v2.4.0'

Comments

@github-actions
Copy link
Contributor

github-actions bot commented Sep 20, 2022

Release OpenSearch and OpenSearch Dashboards 2.4.0

I noticed that a manifest was automatically created in manifests/2.4.0. Please follow the following checklist to make a release.

How to use this issue

This Release Issue

This issue captures the state of the OpenSearch release, its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. More details are included in the Maintainers Release owner section.

Release Steps

There are several steps to the release process, these steps are completed as the whole release and components that are behind present risk to the release. The release owner completes the tasks in this ticket, whereas component owners resolve tasks on their ticket in their repositories.

Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release.

Component List

To aid in understanding the state of the release there is a table with status indicating each component state. This is updated based on the status of the component issues.

Preparation

  • Assign this issue to a release owner.
  • Declare a pencils down date for new features to be merged.
  • 2022/11/04 is pencils down date for code freeze.
  • Update the Campaigns section to include monitoring campaigns during this release.
  • Update this issue so all __REPLACE_RELEASE-__ placeholders have actual dates.
  • Document any new quality requirements or changes.
  • Finalize scope and feature set and update the Public Roadmap.
  • Create a release issue in every component repo that links back to this issue, update Components section with these links.
  • Ensure the label is created in each component repo for this new version, and the next minor release. Create a version label
  • Ensure that all release issues created above are assigned to an owner in the component team.
  • Increase the build frequency for the this release from once a day (H 1 * * *) to once every hour (H/60 * * * *) in jenkinsFile

Campaigns

Release Readiness

GitHub issue custom search

Repo Untriaged issues Issues tagged to 2.4 version
OpenSearch
OpenSearch-Dashboards
alerting
anomaly-detection
asynchronous-search
common-utils
cross-cluster-replication
dashboards-reports
geospatial
index-management
job-scheduler
k-NN
ml-commons
notifications
observability
performance-analyzer
performance-analyzer-rca
security
sql
alerting-dashboards-plugin
anomaly-detection-dashboards-plugin
dashboards-maps
index-management-dashboards-plugin
dashboards-visualizations
opensearch-dashboards-functional-test
security-dashboards-plugin
security-analytics
security-analytics-dashboards-plugin
neural-search
dashboards-search-relevance

Release branch readiness - Ends 2022/10/31

  • Component versions are auto-incremented to 2.4.0 version
  • Plugins team to ensure working CI in component repositories and merge the version increment PR's.
  • OpenSearch and OpenSearch-Dashboards core create 2.4 release branch
  • All components create 2.4 branch for the release.

CI/CD Freeze - Ends 2022/11/02

  • OpenSearch Engineering Effectiveness team will update the input manifests to point to 2.4 branch and configure the CI to run against 2.4 branch.
  • Create Jenkins workflows that run daily snapshot builds for OpenSearch and OpenSearch Dashboards.

Code Complete - Ends 2022/11/03 2022/11/04

Release testing - 2022/11/07 - 2022/11/08

  • Declare a release candidate build, and provide the instructions with the release candidates for teams on testing (2022/11/08).
  • Stop builds for this version of OpenSearch and/or OpenSearch Dashboards in order to avoid accidental commits going in unknowingly. Restart only if necessary else manually run the build workflow and declare new release candidate.
  • After the initial release candidate creation, send PR to lock input manifest refs of both OS and OSD builds to use HEAD commit ids of the components on branch 2.4.
  • Sanity Testing (2022/11/07 - 2022/11/08): Sanity testing and fixing of critical issues found by teams. Teams test their components within the distribution, ensuring integration, backwards compatibility, and perf tests pass.
  • Publish all test results in the comments of this issue.

Performance testing validation - 2022/11/10 - 2022/11/11

  • Performance tests do not show a regression
  • Longevity tests do not show any issues

Release - ** 2022/11/10 2022/11/15**

Post Release

Components

Component On track Notes
opensearch-project/OpenSearch#4755 🟢 Owner Assigned Now
opensearch-project/OpenSearch-Dashboards#2556 🟢 Owner Assigned Now
opensearch-project/alerting#600 🟢 Owner Assigned Now
opensearch-project/anomaly-detection#689 🟢 Owner Assigned Now
opensearch-project/asynchronous-search#191 🟢 Owner Assigned Now
opensearch-project/reporting#505 🟢 Owner Assigned Now
opensearch-project/dashboards-visualizations#118 🟢 Owner Assigned Now
opensearch-project/index-management#570 🟢 Owner Assigned Now
opensearch-project/job-scheduler#252 🟢 Owner Assigned Now
opensearch-project/k-NN#576 🟢 Owner Assigned Now
opensearch-project/performance-analyzer#299 🟢 Owner Assigned Now
opensearch-project/security-dashboards-plugin#1140 🟢 Owner Assigned Now
opensearch-project/security#2160 🟢 Owner Assigned Now
opensearch-project/sql#905 🟢 Owner Assigned Now
opensearch-project/observability#1114 🟢 Owner Assigned Now
opensearch-project/alerting-dashboards-plugin#347 🟢 Owner Assigned Now
opensearch-project/notifications#548 🟢 Owner Assigned Now
opensearch-project/index-management-dashboards-plugin#281 🟢 Owner Assigned Now
opensearch-project/anomaly-detection-dashboards-plugin#327 🟢 Owner Assigned Now
opensearch-project/cross-cluster-replication#588 🟢 Owner Assigned Now
opensearch-project/common-utils#280 🟢 Owner Assigned Now
opensearch-project/ml-commons#466 🟢 Owner Assigned Now
opensearch-project/geospatial#166 🟢 Owner Assigned Now
opensearch-project/dashboards-maps#60 🟢 Owner Assigned Now
opensearch-project/security-analytics#63 🟢 Owner Assigned Now
opensearch-project/security-analytics-dashboards-plugin#16 🟢 Owner Assigned Now
opensearch-project/neural-search#21 🟢 Owner Assigned Now
opensearch-project/dashboards-search-relevance#7 🟢 Owner Assigned Now
Legend

Symbol Meaning
🟢 On track with overall release
🟡 Missed last milestone
🔴 Missed multiple milestones

PRs:

@github-actions github-actions bot added release untriaged Issues that have not yet been triaged v2.4.0 'Issues and PRs related to version v2.4.0' labels Sep 20, 2022
@peterzhuamazon peterzhuamazon self-assigned this Sep 26, 2022
@bbarani bbarani removed the untriaged Issues that have not yet been triaged label Sep 27, 2022
@peterzhuamazon peterzhuamazon pinned this issue Sep 29, 2022
This was referenced Oct 12, 2022
@peterzhuamazon
Copy link
Member

peterzhuamazon commented Nov 8, 2022

New Date:

2022/11/08 EOD: New Freeze Date
2022/11/09 EOD: New RC Builds

New Release Candidate OS 6476: (2022/11/14 21:00PM PST)

#2896 to resolve the performance issues in previous builds.

New Release Candidate OS 6469 6469 / OSD 4304: (2022/11/14 17:37PM PST)

OpenSearch - Build 6469
OpenSearch - Build 6469
OpenSearch Dashboards - Build 4304

  • Use Docker-Compose to setup a cluster

    • Download this gist as docker-compose.yml on your machine (Updated to 6469/4304)
    • Get latest image versions docker-compose pull
    • Start the cluster docker-compose up
  • OpenSearch docker 2.4.0.6469

    • Start without security
      • Docker command docker pull opensearchstaging/opensearch:2.4.0.6469 && docker run -it -p 9200:9200 -e “discovery.type=single-node” -e “DISABLE_SECURITY_PLUGIN=true” opensearchstaging/opensearch:2.4.0.6469
      • Connect command curl https://localhost:9200/
    • Start with security
      • Docker command docker pull opensearchstaging/opensearch:2.4.0.6469 && docker run -it -p 9200:9200 -e “discovery.type=single-node” opensearchstaging/opensearch:2.4.0.6469
      • Connect command curl --insecure https://admin:admin@localhost:9200/
  • OpenSearch-Dashboards docker 2.4.0.4304

    • Start without security
      • Docker command docker pull opensearchstaging/opensearch-dashboards:2.4.0.4304 && docker run -it --network=“host” -e “DISABLE_SECURITY_DASHBOARDS_PLUGIN=true” opensearchstaging/opensearch-dashboards:2.4.0.4304
      • URL https://localhost:5601/
    • Start without security
      • Docker command docker pull opensearchstaging/opensearch-dashboards:2.4.0.4304 && docker run -it --network=“host” opensearchstaging/opensearch-dashboards:2.4.0.4304
      • URL https://localhost:5601/
  • Use TARs to deploy OpenSearch Manually

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Nov 9, 2022

OS 6476 IntegTest (2022/11/15 10:45AM PST)

Adding #5250:

Old IntegTest Results (2022/11/14 10:27AM PST)

  Running:  core-opensearch-dashboards/opensearch-dashboards/datasource-manageme           (9 of 57)
            nt-plugin/1_create_datasource.spec.js                                                   

  0 passing (1ms)

Warning: We failed processing this video.

This error will not alter the exit code.

TimeoutError: operation timed out
    at afterTimeout (C:\Users\Administrator\AppData\Local\Cypress\Cache\9.5.4\Cypress\resources\app\packages\server\node_modules\bluebird\js\release\timers.js:46:19)
    at Timeout.timeoutTimeout [as _onTimeout] (C:\Users\Administrator\AppData\Local\Cypress\Cache\9.5.4\Cypress\resources\app\packages\server\node_modules\bluebird\js\release\timers.js:76:13)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)


Below Outdated now (2022/11/09)

IntegTest Results:

Linux OS integ test results (OUTDATED)
x64: https://build.ci.opensearch.org/job/integ-test/3234/
2022-11-06 00:54:30 INFO     | cross-cluster-replication | with-security        | PASS  |    0 |
2022-11-06 00:54:30 INFO     | cross-cluster-replication | without-security     | PASS  |    0 |
2022-11-06 00:54:44 INFO     | security             | with-security        | PASS  |    0 |
2022-11-06 00:55:13 INFO     | neural-search        | with-security        | PASS  |    0 |
2022-11-06 00:55:13 INFO     | neural-search        | without-security     | PASS  |    0 |
2022-11-06 00:55:18 ERROR    | ml-commons           | with-security        | FAIL  |    1 |
2022-11-06 00:55:18 INFO     | ml-commons           | without-security     | PASS  |    0 |
2022-11-06 00:57:00 INFO     | dashboards-reports   | with-security        | PASS  |    0 |
2022-11-06 00:57:00 INFO     | dashboards-reports   | without-security     | PASS  |    0 |
2022-11-06 00:57:06 INFO     | opensearch-observability | with-security        | PASS  |    0 |
2022-11-06 00:57:06 INFO     | opensearch-observability | without-security     | PASS  |    0 |
2022-11-06 00:57:19 INFO     | geospatial           | with-security        | PASS  |    0 |
2022-11-06 00:57:19 INFO     | geospatial           | without-security     | PASS  |    0 |
2022-11-06 00:58:21 INFO     | asynchronous-search  | with-security        | PASS  |    0 |
2022-11-06 00:58:21 INFO     | asynchronous-search  | without-security     | PASS  |    0 |
2022-11-06 01:00:54 ERROR    | sql                  | with-security        | FAIL  |    1 |
2022-11-06 01:00:54 ERROR    | sql                  | without-security     | FAIL  |    1 |
2022-11-06 01:02:53 INFO     | notifications        | with-security        | PASS  |    0 |
2022-11-06 01:02:53 INFO     | notifications        | without-security     | PASS  |    0 |
2022-11-06 01:06:11 INFO     | k-NN                 | with-security        | PASS  |    0 |
2022-11-06 01:06:11 INFO     | k-NN                 | without-security     | PASS  |    0 |
2022-11-06 01:08:06 ERROR    | security-analytics   | with-security        | FAIL  |    1 |
2022-11-06 01:08:06 ERROR    | security-analytics   | without-security     | FAIL  |    1 |
2022-11-06 01:08:08 INFO     | alerting             | with-security        | PASS  |    0 |
2022-11-06 01:08:08 INFO     | alerting             | without-security     | PASS  |    0 |
2022-11-06 01:16:53 INFO     | anomaly-detection    | with-security        | PASS  |    0 |
2022-11-06 01:16:53 INFO     | anomaly-detection    | without-security     | PASS  |    0 |
2022-11-06 02:03:56 ERROR    | index-management     | with-security        | FAIL  |    1 |
2022-11-06 02:03:56 ERROR    | index-management     | without-security     | FAIL  |    1 |

arm64 https://build.ci.opensearch.org/job/integ-test/3235/
2022-11-06 00:54:10 ERROR    | ml-commons           | with-security        | FAIL  |    1 |
2022-11-06 00:54:10 ERROR    | ml-commons           | without-security     | FAIL  |    1 |
2022-11-06 00:55:39 ERROR    | neural-search        | with-security        | FAIL  |    1 |
2022-11-06 00:55:39 ERROR    | neural-search        | without-security     | FAIL  |    1 |
2022-11-06 00:55:39 INFO     | security             | with-security        | PASS  |    0 |
2022-11-06 00:55:44 ERROR    | sql                  | with-security        | FAIL  |    1 |
2022-11-06 00:55:44 ERROR    | sql                  | without-security     | FAIL  |    1 |
2022-11-06 00:55:48 INFO     | cross-cluster-replication | with-security        | PASS  |    0 |
2022-11-06 00:55:48 INFO     | cross-cluster-replication | without-security     | PASS  |    0 |
2022-11-06 00:58:21 INFO     | opensearch-observability | with-security        | PASS  |    0 |
2022-11-06 00:58:21 INFO     | opensearch-observability | without-security     | PASS  |    0 |
2022-11-06 00:58:36 INFO     | geospatial           | with-security        | PASS  |    0 |
2022-11-06 00:58:36 INFO     | geospatial           | without-security     | PASS  |    0 |
2022-11-06 00:59:07 INFO     | dashboards-reports   | with-security        | PASS  |    0 |
2022-11-06 00:59:07 INFO     | dashboards-reports   | without-security     | PASS  |    0 |
2022-11-06 01:00:47 INFO     | asynchronous-search  | with-security        | PASS  |    0 |
2022-11-06 01:00:47 INFO     | asynchronous-search  | without-security     | PASS  |    0 |
2022-11-06 01:05:21 INFO     | notifications        | with-security        | PASS  |    0 |
2022-11-06 01:05:21 INFO     | notifications        | without-security     | PASS  |    0 |
2022-11-06 01:09:18 INFO     | k-NN                 | with-security        | PASS  |    0 |
2022-11-06 01:09:18 INFO     | k-NN                 | without-security     | PASS  |    0 |
2022-11-06 01:09:39 ERROR    | security-analytics   | with-security        | FAIL  |    1 |
2022-11-06 01:09:39 ERROR    | security-analytics   | without-security     | FAIL  |    1 |
2022-11-06 01:10:18 INFO     | alerting             | with-security        | PASS  |    0 |
2022-11-06 01:10:18 INFO     | alerting             | without-security     | PASS  |    0 |
2022-11-06 01:20:20 INFO     | anomaly-detection    | with-security        | PASS  |    0 |
2022-11-06 01:20:20 INFO     | anomaly-detection    | without-security     | PASS  |    0 |
2022-11-06 02:06:59 ERROR    | index-management     | with-security        | FAIL  |    1 |
2022-11-06 02:06:59 INFO     | index-management     | without-security     | PASS  |    0 |

WINDOWS: 240-dashboards-test-4270.txt

Failures:

windows failed OSD component list (OUTDATED)
core-opensearch-dashboards/opensearch-dashboards/apps/vis_builder/basic.spec.js
core-opensearch-dashboards/opensearch-dashboards/apps/vis_builder/vis_types/bar.spec.js
core-opensearch-dashboards/opensearch-dashboards/apps/vis_builder/vis_types/line.spec.js
plugins/gantt-chart-dashboards/gantt_ui.spec.js
plugins/index-management-dashboards-plugin/indices_spec.js
plugins/index-management-dashboards-plugin/managed_indices_spec.js
plugins/observability-dashboards/1_trace_analytics_dashboard.spec.js
plugins/observability-dashboards/3_trace_analytics_traces.spec.js
plugins/observability-dashboards/4_panels.spec.js
plugins/observability-dashboards/5_event_analytics.spec.js
plugins/observability-dashboards/7_app_analytics.spec.js
plugins/query-workbench-dashboards/ui.spec.js
plugins/search-relevance-dashboards/1_query_compare.spec.js
plugins/security/permissions_spec.js

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Nov 9, 2022

Performance Test:

Version Architecture Security Enabled Start Time(PDT) Duration Instance Type Workload Details Index Latency(ms) p50 Index Latency(ms) p90 Index Latency(ms) p99 Index Latency(ms) p100 Index Throughput(docs/s) p0 Index Throughput(docs/s) p50 Index Throughput(docs/s) p100 Index Throughput (docs/s) Ops Count Index Operation Counts Op Error Count Index Operation Counts Op Error Rate Query Latency (ms) p50 Query Latency (ms) p90 Query Latency (ms) p99 Query Latency (ms) p100 Query Throughput (docs/s) p0 Query Throughput (docs/s) p50 Query Throughput (docs/s) p100 Query Operation Counts Ops Count Query Operation Counts Op Error Count Query Operation Counts Op Error Rate CPU (%) p50 CPU (%) p90 CPU (%) p99 CPU (%) p100 Memory (%) p50 Memory (%) p90 Memory (%) p99 Memory (%) p100 GC (ms) Old GC (ms) Young
2.2.1 arm64 with_security 9/11/22, 3:01 AM 6h 35m 33s m6g.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 1851.8598505373673 2456.7932834712446 4549.877352294918 6659.5146484375 39204.41796875 40484.896675314696 43940.48046875 46459 0 0 360.88779627482097 373.73701316833495 384.26831719970704 390.29581985473635 1.670055627822876 1.7078201830387116 1.7455639243125916 1510 0 0 96 97 97 97 49.2 64 76.96000000000004 91 0 985508
2.2.1 arm64 without_security 9/12/22, 3:01 AM 6h 10m 30s m6g.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 1646.453753 2210.699582 4342.278781 6122.276367 44029.94922 45598.62776 49929.39063 46013 0 0 362.3607367 374.8823477 378.7677886 391.089801 1.704830134 1.731174356 1.754105937 1510 0 0 96 97 97 97 48 62 83.8 92 0 829901
2.2.1 x64 with_security 9/11/22, 2:58 AM 8h 10m 40s m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2288.2764957074537 3038.3064399577747 5879.324521689244 9268.974609375 32105.65234375 33927.90692669141 38235.58203125 46894 0 0 286.41736221313477 293.68860984802245 301.12121810913084 318.229997253418 1.7262567162513733 1.740007507801056 1.75261892080307 1510 0 0 97 97 97 97 52 66 87.76999999999992 93 0 1231253
2.2.1 x64 without_security 9/11/22, 2:58 AM 7h 49m 37s m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2192.6901234475677 2914.758178724321 5700.923922209283 11106.1728515625 33684.34375 35614.496027572684 39991.8125 46757 0 0 295.2953796386719 302.1095615386963 309.9044530029297 322.45588912963865 1.7269416570663452 1.7409352322419487 1.7553181290626525 1510 0 0 97 97 98 98 49 63 84 94 0 1054789
2.3.0 arm64 with_security 9/9/22, 10:41 AM 6h 42m 35s m6g.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 1884.939796 2498.486831 4672.044931 8184.402832 38684.21484 40013.98915 43568.23438 46488 0 0 351.4363827 359.4697859 368.6659151 437.3339294 1.699884045 1.722742015 1.75187211 1510 0 0 96 97 97 97 52 65 88.98 94 0 979894
2.3.0 arm64 without_security 9/9/22, 10:43 AM 6h 12m 31s m6g.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 1625.392231 2180.028306 4150.245827 6011.796387 44516.28516 46462.11049 51156.74219 45950 0 0 364.6436938 373.184549 379.6433586 386.4868347 1.685038006 1.709957743 1.753778672 1510 0 0 96 97 97 97 48 64 83.16 91 0 853041
2.3.0 x64 with_security 9/9/22, 10:41 AM 8h 5m 38s m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2274.482999 3001.790512 5625.219549 9248.959961 32337.31836 33916.93773 37967.18359 46902 0 0 290.8293233 300.3195797 309.3330738 314.5757118 1.725921583 1.739219183 1.752308607 1510 0 0 97 97 98 98 51 65 81.66 87 0 1238934
2.3.0 x64 without_security 9/9/22, 10:41 AM 7h 41m 37s m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2174.641495 2896.447136 5494.624087 9741.860352 33844.23438 35826.79863 40342.28125 46705 0 0 287.6040042 297.3744589 310.0044941 328.9224983 1.731490076 1.743171066 1.757029057 1510 0 0 97 98 98 98 49 64 86.56 96 0 1033209
2.4.0 arm64 linux 6443 with_security 11/8/2022 10:22:23 PM 6h 53m m6g.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 1,947.2 2,553.9 4,524 7,408 37,780.5 39,306.1 42,731.4 46,547 0 0% 360.4 369.2 375.5 381.8 1.679 1.703 1.745 1510 0 0 96 99.3 100 100 52.333 66 81 90 0 1,048,597
2.4.0 arm64 linux 6443 without_security 11/8/2022 10:22:09 PM 6h 17m m6g.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 1,731.3 2,315.6 4,148.1 5,856.7 42,176.1 43,538 48,088.2 46,180 0 0% 379.5 394.4 398.6 402.9 1.699 1.726 1.753 1510 0 0 96 100 100 100 47 65 85.38 92 0 890,836
2.4.0 x64 linux 6443 with_security 11/8/2022 10:21:59 PM 8h 38m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,462.3 3,247.1 5,880.8 10,683.6 29,857.4 31,437.4 34,705.7 47,112 1 0% 277 284.1 293.9 318.1 1.725 1.739 1.753 1510 0 0 97 100 100 100 52 66 87.32 94 0 1,288,361
2.4.0 x64 linux 6443 without_security 11/8/2022 10:21:38 PM 8h 32m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,460 3,226.5 5,754.8 9,419.8 30,270.1 31,663.9 35,097.1 47,088 0 0% 302.5 309.1 318.3 322.4 1.73 1.742 1.756 1510 0 0 97 100 100 100 48 64 86 92 0 1,086,947
2.4.0 x64 windows 6443 with_security 11/9/2022 16:17:35 PM 9h 7m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,393.2 3,490 8,011.4 17,680.8 28,401.8 29,648.8 33,415.8 47,249 0 0% 136.8 139.4 151.2 168.6 1.742 1.75 1.764 1510 0 0 95 98 100 100 57 67 84 94 0 1,792,702
2.4.0 x64 windows 6443 without_security 11/9/2022 15:21:47 PM 8h 21m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,214.3 3,190.7 7,772.1 20,755 31,145.8 32,359 36,830.4 46,990 0 0% 142.8 145.8 165 179.1 1.741 1.75 1.764 1510 0 0 95 98 100 100 54 66.2 85 91 0 1,455,204
2.4.0 arm64 linux 6452 with_security 11/10/2022 17:16:07 6h 56m m6g.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 1,920.3 2,542 4,622.5 7,006.1 37,559.4 39,548.6 43,218.9 46,515 0 0% 351.7 363.7 372.7 378.7 1.704 1.727 1.751 1,510 0 0 96 99 100 100 52 65 83.05 94 0 1,054,809
2.4.0 arm64 linux 6452 without_security 11/10/2022 17:16:07 6h 24m m6g.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 1,800.7 2,419.4 4,370.6 7,155.7 40,626 41,986.6 45,865.4 46,328 0 0% 337.3 344.6 354 359.4 1.727 1.741 1.755 1,510 0 0 96 100 100 100 51 64 83.98 94 0 861,058
2.4.0 x64 linux 6452 with_security 11/10/2022 17:16:07 8h 49m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,532.8 3,313.9 6,184.4 9,893.8 29,239.2 30,338.8 33,739.2 47,191 0 0% 293.6 304.9 310.4 373.4 1.727 1.74 1.753 1510 0 0 97 100 100 100 52 66 77.76 91 0 1,273,731
2.4.0 x64 linux 6452 without_security 11/10/2022 17:16:07 8h 16m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,392.8 3,180.7 5,759.4 9,917.9 30,749.1 32,121.4 35,796.6 47,060 0 0% 299.9 309.9 325.5 351.4 1.727 1.741 1.756 1510 0 0 97 99 100 100 49 64 77.1 95 0 1,084,556
2.4.0 x64 windows 6452 with_security 11/10/2022 18:00:32 9h 15m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,334.2 3,473 8,081.5 20,238.5 28,798 30,299.1 34,696.1 47,134 0 0% 146.7 149.5 159.8 166.3 1.743 1.751 1.764 1510 0 0 95 98 100 100 58 68.6 80 92 0 1,847,183
2.4.0 x64 windows 6452 without_security 11/10/2022 18:06:41 9h 15m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,211.4 3,235 8,021.7 20,577.1 30,200.8 32,881.7 38,057 46,950 0 0% 139.6 143.3 153.3 167 1.743 1.75 1.764 1510 0 0 95 98 100 100 54 67.6 92.61 96 0 1,491,492

@bbarani
Copy link
Member

bbarani commented Nov 10, 2022

OpenSearch 2.3.0 vs 2.4.0- Linux (With security) performance stats:

Version Architecture Type Date Time Instane Load data set p50Index Latency(ms) p90Index Latency(ms) p99Index Latency(ms) p100Index Latency(ms) Index Throughput(docs/s) p0 p50Index Throughput(docs/s) Index Throughput(docs/s) p100 Index Throughput (docs/s)Ops Count Index Throughput (docs/s)Op Error Count Index Operation Counts Op Error Rate Query Latency (ms)p50 Query Latency (ms)p90 Query Latency (ms)p99 Query Latency (ms)p100 Query Throughput (docs/s)p0 Query Throughput (docs/s)p50 Query Throughput (docs/s)p100 Query Operation CountsOps Count Query Operation CountsOp Error Count Query Operation CountsError Rate p50CPU (%) p90CPU (%) p99CPU (%) p100CPU (%) p50Memory (%) p90Memory (%) p99Memory (%) p100Memory (%) OldGC (ms) YoungGC (ms)
2.4.0 x64 with_security 11/8/22, 10:21 PM 8h 38m 45s m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2462.33505 3247.12681 5880.81022 10683.56738 29857.36328 31437.41158 34705.74609 47112 1 0.00002 277.02202 284.07408 293.89088 318.08207 1.72519 1.7393 1.75265 1510 0 0 97 100 100 100 52 66 87.32 94 0 1288361
2.3.0 x64 with_security 11/8/22, 5:13 PM 8h 17m 43s m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2342.61928 3089.664 5978.26382 8922.70898 31505.12109 33191.37682 37524.30859 46920 0 0 280.41168 293.73222 309.35961 330.25104 1.7244 1.73838 1.75177 1510 0 0 97 97 97 98 52 65 88.08 93 0 1233696
              -4.86% -4.85% 1.66% -16.48% 5.52% 5.58% 8.12% -0.41% -100% -100% 1.22% 3.40% 5.26% 3.83% -0.05% -0.05% -0.05% 0%     0% -3% -3% -2% 0% -1.52% 0.87% -1.06%   -4.24%

OpenSearch 2.3.0 vs 2.4.0- Linux (Without security) performance stats:

Version Architecture Type Date Time Instane Load data set p50Index Latency(ms) p90Index Latency(ms) p99Index Latency(ms) p100Index Latency(ms) Index Throughput(docs/s) p0 p50Index Throughput(docs/s) Index Throughput(docs/s) p100 Index Throughput (docs/s)Ops Count Index Throughput (docs/s)Op Error Count Index Operation Counts Op Error Rate Query Latency (ms)p50 Query Latency (ms)p90 Query Latency (ms)p99 Query Latency (ms)p100 Query Throughput (docs/s)p0 Query Throughput (docs/s)p50 Query Throughput (docs/s)p100 Query Operation CountsOps Count Query Operation CountsOp Error Count Query Operation CountsError Rate p50CPU (%) p90CPU (%) p99CPU (%) p100CPU (%) p50Memory (%) p90Memory (%) p99Memory (%) p100Memory (%) OldGC (ms) YoungGC (ms)
2.4.0 x64 without_security 11/8/22, 10:21 PM 8h 32m 41s m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2460.03827 3226.48118 5754.77673 9419.75098 30270.07617 31663.94831 35097.07031 47088 0 0 302.54904 309.05994 318.34879 322.37557 1.72988 1.74227 1.75621 1510 0 0 97 100 100 100 48 64 86 92 0 1086947
2.3.0 x64 without_security 11/8/22, 5:13 PM 7h 51m 38s m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2178.79893 2873.20353 5599.81038 9851.05273 33502.58984 35472.34105 40172.14844 46746 0 0 280.42906 288.05516 300.95042 309.34666 1.72794 1.74152 1.75622 1510 0 0 97 98 98 98 49.25 64 82.79 91 0 1026230
              -11.43% -10.95% -2.69% 4.58% 10.68% 12.03% 14.46% -0.73%     -7.31% -6.80% -5.47% -4.04% -0.11% -0.04% 0% 0%     0% -2% -2% -2% 2.60% 0% -3.73% -1.09%   -5.59%

OpenSearch 2.4.0 - Windows vs Linux performance stats:

Type Date Time Instane Load data set p50Index Latency(ms) p90Index Latency(ms) p99Index Latency(ms) p100Index Latency(ms) Index Throughput(docs/s) p0 p50Index Throughput(docs/s) Index Throughput(docs/s) p100 Ops Count Query Latency (ms)p50 Query Latency (ms)p90 Query Latency (ms)p99 Query Latency (ms)p100 Query Throughput (docs/s)p0 Query Throughput (docs/s)p50 Query Throughput (docs/s)p100 Ops Count p50CPU (%) p90CPU (%) p99CPU (%) p100CPU (%) p50Memory (%) p90Memory (%) p99Memory (%) p100Memory (%) Young
with_security 11/8/2022 22:21 8h 38m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,462.3 3,247.1 5,880.8 10,683.6 29,857.4 31,437.4 34,705.7 47,112 277 284.1 293.9 318.1 1.725 1.739 1.753 1510 97 100 100 100 52 66 87.32 94 1,288,361
without_security 11/8/2022 22:21 8h 32m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,460 3,226.5 5,754.8 9,419.8 30,270.1 31,663.9 35,097.1 47,088 302.5 309.1 318.3 322.4 1.73 1.742 1.756 1510 97 100 100 100 48 64 86 92 1,086,947
X64 Windows with_security 11/9/2022 16:17 9h 7m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,393.2 3,490 8,011.4 17,680.8 28,401.8 29,648.8 33,415.8 47,249 136.8 139.4 151.2 168.6 1.742 1.75 1.764 1510 95 98 100 100 57 67 84 94 1,792,702
X64 Windows without_security 11/9/2022 15:21 8h 21m m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2,214.3 3,190.7 7,772.1 20,755 31,145.8 32,359 36,830.4 46,990 142.8 145.8 165 179.1 1.741 1.75 1.764 1510 95 98 100 100 54 66.2 85 91 1,455,204
2.4.0 Linux vs Windows (Without security) 9.99% 1.11% -35.05% -120.33% -2.89% -2.20% -4.94% 0.21% 52.79% 52.83% 48.16% 44.45% -0.64% -0.46% -0.46% 0.00% 2.06% 2.00% 0.00% 0.00% -12.50% -3.44% 1.16% 1.09% -33.88%
2.4.0 Linux vs Windows (With security) 2.81% -7.48% -36.23% -65.49% 4.88% 5.69% 3.72% -0.29% 50.61% 50.93% 48.55% 47.00% -0.99% -0.63% 100.26% 100.00% 2.06% 2.00% 0.00% 0.00% -9.62% -1.52% 3.80% 0.00% -39.15%

@bbarani
Copy link
Member

bbarani commented Nov 14, 2022

Performance stats:

Product/ Version Architecture Description Instance Type Workload Index latency (p50) Index latencyp90 Index latencyp99 Index latencyp100 Index throughputp0 Index throughputp50 Index throughputp100 Ops Count Op Error Count Error Rate Query latencyp50 Query latencyp90 Query latencyp99 Query latencyp100 Query throughputp0 Query throughputp50 Query throughputp100 Ops Count Op Error Count Error Rate CPUp50 CPUp90 CPUp99 CPUp100 Memoryp50 Memoryp90 Memoryp99 Memoryp100 Old Young
OS 2.3.0 Linux X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2274.483 3001.79051 5625.21955 9248.95996 32337.31836 33916.93773 37967.18359 46902 - - 290.82932 300.31958 309.33307 314.57571 1.72592 1.73922 1.75231 1510 - - 97 97 98 98 51 65 81.66 87 - 1238934
OS 2.4.0 Linux 6443 X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,393.2 3,490 8,011.4 17,680.8 28,401.8 29,648.8 33,415.8 47,249 - - 136.8 139.4 151.2 168.6 1.742 1.75 1.764 1510 - - 95 98 100 100 57 67 84 94 - 1,792,702
Comparison         -5.22% -16.26% -42.42% -91.17% -12.17% -12.58% -11.99%       52.96% 53.58% 51.12% 46.40% 0.93% 0.62% 0.67%                          
OS 2.3.0 Linux X64 Without Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2174.64149 2896.44714 5494.62409 9741.86035 33844.23438 35826.79863 40342.28125 46705 - - 287.604 297.37446 310.00449 328.9225 1.73149 1.74317 1.75703 1510 - - 97 98 98 98 49 64 86.56 96 - 1033209
OS 2.4.0 Windows 6443 X64 Without Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,214.3 3,190.7 7,772.1 20,755 31,145.8 32,359 36,830.4 46,990 - - 142.8 145.8 165 179.1 1.741 1.75 1.764 1510 - - 95 98 100 100 54 66.2 85 91 - 1,455,204
Comparison         -1.82% -10.16% -41.45% -113.05% -7.97% -9.68% -8.71%       50.35% 50.97% 46.77% 45.55% 0.55% 0.39% 0.40%                          
OS 2.4.0 Linux 6443 X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,462.3 3,247.1 5,880.8 10,683.6 29,857.4 31,437.4 34,705.7 47,112 - - 277 284.1 293.9 318.1 1.725 1.739 1.753 1510 - - 97 100 100 100 52 66 87.32 94 - 1,288,361
OS 2.4.0 Windows 6443 X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,393.2 3,490 8,011.4 17,680.8 28,401.8 29,648.8 33,415.8 47,249 - - 136.8 139.4 151.2 168.6 1.742 1.75 1.764 1510 - - 95 98 100 100 57 67 84 94 - 1,792,702
Comparison         2.81% -7.48% -36.23% -65.49% -4.88% -5.69% -3.72%       50.61% 50.93% 48.55% 47.00% 0.99% 0.63% 0.63%                          
OS 2.4.0 Linux 6443 X64 Without Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,460 3,226.5 5,754.8 9,419.8 30,270.1 31,663.9 35,097.1 47,088 - - 302.5 309.1 318.3 322.4 1.73 1.742 1.756 1510 - - 97 100 100 100 48 64 86 92 - 1,086,947
OS 2.4.0 Windows 6443 X64 Without Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,214.3 3,190.7 7,772.1 20,755 31,145.8 32,359 36,830.4 46,990 - - 142.8 145.8 165 179.1 1.741 1.75 1.764 1510 - - 95 98 100 100 54 66.2 85 91 - 1,455,204
Comparison         9.99% 1.11% -35.05% -120.33% 2.89% 2.20% 4.94%       52.79% 52.83% 48.16% 44.45% 0.64% 0.46% 0.46%                          
                                                                     
                                                                     
OS 2.3.0 Linux X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2274.483 3001.79051 5625.21955 9248.95996 32337.31836 33916.93773 37967.18359 46902 - - 290.82932 300.31958 309.33307 314.57571 1.72592 1.73922 1.75231 1510 - - 97 97 98 98 51 65 81.66 87 - 1238934
OS 2.4.0 Linux 6452 X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,532.8 3,313.9 6,184.4 9,893.8 29,239.2 30,338.8 33,739.2 47,191 - - 293.6 304.9 310.4 373.4 1.727 1.74 1.753 1510 - - 97 100 100 100 52 66 77.76 91 - 1,273,731
Comparison         -11.36% -10.40% -9.94% -6.97% -9.58% -10.55% -11.14%       -0.95% -1.53% -0.34% -18.70% 0.06% 0.04% 0.04%                          
OS 2.3.0 Linux X64 Without Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2174.64149 2896.44714 5494.62409 9741.86035 33844.23438 35826.79863 40342.28125 46705 - - 287.604 297.37446 310.00449 328.9225 1.73149 1.74317 1.75703 1510 - - 97 98 98 98 49 64 86.56 96 - 1033209
OS 2.4.0 Windows 6452 X64 Without Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,211.4 3,235 8,021.7 20,577.1 30,200.8 32,881.7 38,057 46,950 - - 139.6 143.3 153.3 167 1.743 1.75 1.764 1510 - - 95 98 100 100 54 67.6 92.61 96 - 1,491,492
Comparison         -1.69% -11.69% -45.99% -111.22% -10.77% -8.22% -5.66%       51.46% 51.81% 50.55% 49.23% 0.66% 0.39% 0.40%                          
OS 2.4.0 Linux 6452 X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,532.8 3,313.9 6,184.4 9,893.8 29,239.2 30,338.8 33,739.2 47,191 - - 293.6 304.9 310.4 373.4 1.727 1.74 1.753 1510 - - 97 100 100 100 52 66 77.76 91 - 1,273,731
OS 2.4.0 Windows 6452 X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,334.2 3,473 8,081.5 20,238.5 28,798 30,299.1 34,696.1 47,134 - - 146.7 149.5 159.8 166.3 1.743 1.751 1.751 1510 - - 95 98 100 100 58 68.6 80 92 - 1,847,183
Comparison         7.84% -4.80% -30.68% -104.56% -1.51% -0.13% 2.84%       50.03% 50.97% 48.52% 55.46% 0.93% 0.63% -0.11%                          
OS 2.4.0 Linux 6452 X64 Without Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,392.8 3,180.7 5,759.4 9,917.9 30,749.1 32,121.4 35,796.6 47,060 - - 299.9 309.9 325.5 351.4 1.727 1.741 1.756 1510 - - 97 99 100 100 49 64 77.1 95 - 1,084,556
OS 2.4.0 Windows 6452 X64 Without Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,211.4 3,235 8,021.7 20,577.1 30,200.8 32,881.7 38,057 46,950 - - 139.6 143.3 153.3 167 1.743 1.75 1.764 1510 - - 95 98 100 100 54 67.6 92.61 96 - 1,491,492
Comparison         7.58% -1.71% -39.28% -107.47% -1.78% 2.37% 6.31%       53.45% 53.76% 52.90% 52.48% 0.93% 0.52% 0.46%                          

@bbarani
Copy link
Member

bbarani commented Nov 15, 2022

Updated performance stats after code fix ( i.e. After reverting the code that was causing the degradation view this PR)

Version Architecture Description Instance Type Workload Index latency (p50) Index latencyp90 Index latencyp99 Index latencyp100 Index throughputp0 Index throughputp50 Index throughputp100 Ops Count
2.4.0 with code fix x64 with_security m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2243.21797 2975.90527 5681.49956 10124.73535 32799.05469 34558.59002 38533.34375 46867
2.3.0 x64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2274.483 3001.79051 5625.21955 9248.95996 32337.31836 33916.93773 37967.18359 46902
        % diff -1.38411 -0.86606 0.99551 9.04087 1.41775 1.87411 1.48015 -0.07465
                         
                         
2.4.0 build with code fix x64 with_security m5.xlarge nyc_taxis / 2 warmupIterations / 3 testIterations 2243.21797 2975.90527 5681.49956 10124.73535 32799.05469 34558.59002 38533.34375 46867
2.4.0 without code fix X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,393.2 3,490 8,011.4 17,680.8 28,401.8 29,648.8 33,415.8 47,249
        % diff -6.46974 -15.90171 -34.03078 -54.34936 14.36991 15.29354 14.22545 -0.81176
                         
2.3.0 X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2274.483 3001.79051 5625.21955 9248.95996 32337.31836 33916.93773 37967.18359  
2.4.0 without fix X64 With Security m5.xlarge nyc_taxis/ 2 warmupIterations/ 3 testIterations 2,393.2 3,490 8,011.4 17,680.8 28,401.8 29,648.8 33,415.8  
Comparison       % diff -5.08676 -15.04083 -34.99666 -62.62098 12.95876 13.42905 12.75201  

@bbarani
Copy link
Member

bbarani commented Nov 15, 2022

Verified Docker:

{
  "name" : "bddabedcf020",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "iEpIBCqrSEqaTKshwdHVcQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.4.0",
    "build_type" : "tar",
    "build_hash" : "744ca260b892d119be8164f48d92b8810bd7801c",
    "build_date" : "2022-11-15T04:42:29.671309257Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

@bbarani
Copy link
Member

bbarani commented Nov 15, 2022

Native plugin installation (Validation):

 ./opensearch-plugin install repository-s3
warning: ignoring JAVA_TOOL_OPTIONS=-Dlog4j2.formatMsgNoLookups=true
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
[=================================================] 100%   
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed repository-s3 with folder name repository-s3

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Nov 15, 2022

Maven:
image
https://aws.oss.sonatype.org/content/repositories/releases/org/opensearch/opensearch/2.4.0/
https://repo1.maven.org/maven2/org/opensearch/opensearch/2.4.0/

RPM:


{
  "name" : "<>",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "BoSzzJc0TpmN_ZV_CTeiKA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.4.0",
    "build_type" : "rpm",
    "build_hash" : "744ca260b892d119be8164f48d92b8810bd7801c",
    "build_date" : "2022-11-15T04:42:51.211206295Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

WINDOWS:

{
  "name" : "EC2AMAZ-<>",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "Q2W1BgPoRi60C5td9Rp7Tg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.4.0",
    "build_type" : "zip",
    "build_hash" : "744ca260b892d119be8164f48d92b8810bd7801c",
    "build_date" : "2022-11-15T05:00:29.141419400Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}


@bbarani
Copy link
Member

bbarani commented Nov 17, 2022

OpenSearch 2.4.0 was released on Nov 15 2022.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release v2.4.0 'Issues and PRs related to version v2.4.0'
Projects
None yet
Development

No branches or pull requests

7 participants