Compatibility with segment replication

[dreamer-89]

Summary

With 2.9.0 release, there are lot of enhancements going in for segment replication[1][2] feature (went GA in 2.7.0), we need to ensure different plugins are compatible with current state of this feature. Previously, we ran tests on plugin repos to verify this compatibility but want plugin owners to be aware of these changes so that required updates (if any) can be made. With 2.10.0 release, remote store feature is going GA which internally uses SEGMENT replication strategy only i.e. it enforces all indices to use SEGMENT replication strategy. So, it is important to validate plugins are compatible with segment replication feature.

What changed

1. Refresh policy behavior

1. RefreshPolicy.IMMEDIATE will only refresh primary shards but not replica shards immediately. Instead post refresh, primary will start a round of segment replication to update the replica shard copies leading to eventual consistency.
2. RefreshPolicy.WAIT_UNTIL ensures the indexing operation is searchable in your cluster i.e. RAW (Read after write guarantee). With segment replication, this guarantee is not promised due to delay in replica shared updates from asynchronous background refreshes.

2. Refresh lag on replicas

With segment replication, there is inherent delay in documents to be searchable on replica shard copies. This is due to the fact that replica shard copies over data (segment) files from primary. Thus, compared to document replication, there will be on average increase in amount of time the replica shards are consistent with primaries.

3. System/hidden indices support

With opensearch-project/OpenSearch#8200, system and hidden indices are now supported with SEGMENT replication strategy. We need to ensure there are no bottlenecks which prevents system/hidden indices with segment replication.

Next steps

With segment replication strong reads are not guaranteed. Thus, if the plugin needs strong reads guarantees specially as alternative to change in behavior of refresh policy and lag on replicas (point 1 and 2 above), we need to update search requests to target primary shard only. With opensearch-project/OpenSearch#7375, core now supports primary shards only based search. Please follow documentation for examples and details

Open questions

In case of any questions or issues, please post it in core issue

Reference

[1] Design

[2] Documentation

[dreamer-89]

Request owners to add v2.9.0 label on this issue. Tagging @dai-chen

[penghuo]

what is strong reads? do you mena strongly consistent reads?

[penghuo]

1. does it breaking change? then plugin need to add parmater like routing=primary at query time to guarantee the strong read consisten?
2. or it backward compatible. if plugin does not enable segment replication, then there is no impact.

[penghuo]

More details in here. opensearch-project/OpenSearch#8211 (comment)

The datasource setting read api required strongly consistent reads.

[penghuo]

1. does it breaking change? then plugin need to add parmater like routing=primary at query time to guarantee the strong read consisten?

It is not a breaking change unless you use segment replication.
The reason for doing this campaign for 2.9.0 because we allow enabling segment replication at cluster level which means all indices be using segment replication. We want to verify this does not break plugins. Also with 2.10.0, remote store is going GA which internally works with segment replication only. So, with remote store feature, segment replication will be imposed on all indices. So it is better to verify this compatibility now vs during 2.10.0 release

2. or it backward compatible. if plugin does not enable segment replication, then there is no impact.

Users can enable SegRep on Cluster level, which will be applied for all plugins and system indexes

[dreamer-89]

Hi Plugin Owners,
Gentle reminder to look into this issue as code freeze date for 2.9.0 release is near i.e. July 11th.

[MaxKsyunz]

Resolved by setting _primary preference when sending REST request. Change is in the 2.x branch.

[dreamer-89]

1. With Segment Replication - Support realtime reads with GET requests OpenSearch#8536, core now supports realtime reads with get/mget APIs. Thus, if strong reads are desired plugin can use get/mget queries. get/mget is recommended over _primary based routing in order to reduce load on primary.
2. Existing OpenSearchDataSourceMetadataStorage.java uses match_all & term based search queries which does not provide strong reads and thus, for SEGMENT replication enabled indices there is no need to perform _primary based routing.

Based on above 2 points, created revert PR #2013 @penghuo @MaxKsyunz : Please have a look.

[dreamer-89]

As #2031 is merged, this plugin no longer performs _primary based search requests. If this plugin needs strong reads, owners can open up a separate task to discuss the need, possible approaches to achieve it etc. There is no action needed on this issue.

CC @penghuo @vamsi-amazon @MaxKsyunz