Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility with segment replication #833

Closed
Tracked by #8211
dreamer-89 opened this issue Jun 29, 2023 · 8 comments
Closed
Tracked by #8211

Compatibility with segment replication #833

dreamer-89 opened this issue Jun 29, 2023 · 8 comments
Assignees
Labels
enhancement New request v2.9.0 v2.9.0 v2.10.0 Issues targeting release v2.10.0

Comments

@dreamer-89
Copy link
Member

dreamer-89 commented Jun 29, 2023

Summary

With 2.9.0 release, there are lot of enhancements going in for segment replication[1][2] feature (went GA in 2.7.0), we need to ensure different plugins are compatible with current state of this feature. Previously, we ran tests on plugin repos to verify this compatibility but want plugin owners to be aware of these changes so that required updates (if any) can be made. With 2.10.0 release, remote store feature is going GA which internally uses SEGMENT replication strategy only i.e. it enforces all indices to use SEGMENT replication strategy. So, it is important to validate plugins are compatible with segment replication feature.

What changed

1. Refresh policy behavior

  1. RefreshPolicy.IMMEDIATE will only refresh primary shards but not replica shards immediately. Instead post refresh, primary will start a round of segment replication to update the replica shard copies leading to eventual consistency.
  2. RefreshPolicy.WAIT_UNTIL ensures the indexing operation is searchable in your cluster i.e. RAW (Read after write guarantee). With segment replication, this guarantee is not promised due to delay in replica shared updates from asynchronous background refreshes.

2. Refresh lag on replicas

With segment replication, there is inherent delay in documents to be searchable on replica shard copies. This is due to the fact that replica shard copies over data (segment) files from primary. Thus, compared to document replication, there will be on average increase in amount of time the replica shards are consistent with primaries.

3. System/hidden indices support

With opensearch-project/OpenSearch#8200, system and hidden indices are now supported with SEGMENT replication strategy. We need to ensure there are no bottlenecks which prevents system/hidden indices with segment replication.

Next steps

With segment replication strong reads are not guaranteed. Thus, if the plugin needs strong reads guarantees specially as alternative to change in behavior of refresh policy and lag on replicas (point 1 and 2 above), we need to update search requests to target primary shard only. With opensearch-project/OpenSearch#7375, core now supports primary shards only based search. Please follow documentation for examples and details

Open questions

In case of any questions or issues, please post it in core issue

Reference

[1] Design

[2] Documentation

@dreamer-89
Copy link
Member Author

Request owners to add v2.9.0 label on this issue.

@gaiksaya gaiksaya added the v2.9.0 v2.9.0 label Jul 3, 2023
@bowenlan-amzn bowenlan-amzn self-assigned this Jul 10, 2023
@dreamer-89
Copy link
Member Author

Hi Plugin Owners,
Gentle reminder to look into this issue as code freeze date for 2.9.0 release is near i.e. July 11th.

@getsaurabh02
Copy link
Member

refer opensearch-project/alerting#974 (comment)
The same points for system/hidden indices and scheduling are applicable for ISM as well.
cc : @dreamer-89

@dreamer-89
Copy link
Member Author

Ping @getsaurabh02 @bowenlan-amzn as we are approaching 2.10 code freeze date. Please have a look.

@bowenlan-amzn
Copy link
Member

@dreamer-89 what's the behavior of an index created in an OS version without segRep, and later upgrade to a version that enforces segRep? It will still use docRep right?

@dreamer-89
Copy link
Member Author

@dreamer-89 what's the behavior of an index created in an OS version without segRep, and later upgrade to a version that enforces segRep? It will still use docRep right?

This is a good question! Yes, you are right. AFAIK, once an index is created, it's replication type change wouldn't be allowed i.e. index created with DOCUMENT replication wouldn't be changed to SEGMENT and vice-versa.

When cluster level settings enforces a specific replication strategy, it will govern replication type for new index creations. There wouldn't be change in replication type for existing indices. Tagging product @nandi-github @elfisher if this is not correct!

CC @mch2 @anasalkouz

@gaiksaya gaiksaya added the v2.10.0 Issues targeting release v2.10.0 label Aug 31, 2023
@dreamer-89
Copy link
Member Author

@bowenlan-amzn : Is there anything pending on this issue ? If not, do you mind closing this issue.

@bowenlan-amzn
Copy link
Member

Yes, we can close this now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New request v2.9.0 v2.9.0 v2.10.0 Issues targeting release v2.10.0
Projects
None yet
Development

No branches or pull requests

5 participants