Compatibility with segment replication

[dreamer-89]

Summary

With 2.9.0 release, there are lot of enhancements going in for segment replication[1][2] feature (went GA in 2.7.0), we need to ensure different plugins are compatible with current state of this feature. Previously, we ran tests on plugin repos to verify this compatibility but want plugin owners to be aware of these changes so that required updates (if any) can be made. With 2.10.0 release, remote store feature is going GA which internally uses SEGMENT replication strategy only i.e. it enforces all indices to use SEGMENT replication strategy. So, it is important to validate plugins are compatible with segment replication feature.

What changed

1. Refresh policy behavior

1. RefreshPolicy.IMMEDIATE will only refresh primary shards but not replica shards immediately. Instead post refresh, primary will start a round of segment replication to update the replica shard copies leading to eventual consistency.
2. RefreshPolicy.WAIT_UNTIL ensures the indexing operation is searchable in your cluster i.e. RAW (Read after write guarantee). With segment replication, this guarantee is not promised due to delay in replica shared updates from asynchronous background refreshes.

2. Refresh lag on replicas

With segment replication, there is inherent delay in documents to be searchable on replica shard copies. This is due to the fact that replica shard copies over data (segment) files from primary. Thus, compared to document replication, there will be on average increase in amount of time the replica shards are consistent with primaries.

3. System/hidden indices support

With opensearch-project/OpenSearch#8200, system and hidden indices are now supported with SEGMENT replication strategy. We need to ensure there are no bottlenecks which prevents system/hidden indices with segment replication.

Next steps

With segment replication strong reads are not guaranteed. Thus, if the plugin needs strong reads guarantees specially as alternative to change in behavior of refresh policy and lag on replicas (point 1 and 2 above), we need to update search requests to target primary shard only. With opensearch-project/OpenSearch#7375, core now supports primary shards only based search. Please follow documentation for examples and details

Open questions

In case of any questions or issues, please post it in core issue

Reference

[1] Design

[2] Documentation

[dreamer-89]

Request owners to add v2.9.0 label on this issue.

[dreamer-89]

Hi Plugin Owners,
Gentle reminder to look into this issue as code freeze date for 2.9.0 release is near i.e. July 11th.

[heemin32]

There are one place where we use WAIT_UNTIL,

geospatial/src/main/java/org/opensearch/geospatial/action/upload/geojson/ContentBuilder.java

Line 60 in ee06307

return client.prepareBulk().setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);

This is used when customer upload their geojson file to OpenSearch cluster. After upload is completed, the data might not be available for search in replicas after segment replication is enabled. We will accept the behavior as it is. User can search using primary shards only option if they need strong read gurantee.

[dreamer-89]

Thanks @heemin32 for sharing the update and having the _primary based fix. I just wanted to check with you that using primary based search may add addtional load on primary shards and may have performance implications. I assume you do not have heavy read workload and already verified this to be not a problem in your use-case.

[heemin32]

The read is not happening internally. It will be user initiated. So, user should decide if they want to wait or use _primary option.

[dreamer-89]

@heemin32 : From linked PR #347, I understand _primary based search is used for all fetch datasource metadata search requests, so I don't understand how user would decide ?

In my last comment, I meant that using _primary search preference for high read throughput systems can have performance implications. Can you please confirm that with your change there will be no adverse impact

[heemin32]

@dreamer-89 I am talking about what will be released in 2.9.0.

geospatial/src/main/java/org/opensearch/geospatial/action/upload/geojson/ContentBuilder.java

Line 60 in ee06307

return client.prepareBulk().setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);

The PR #347 you mentioned will be released in 2.10.0.

Regarding the PR #347, yes, there is no issue as datasource read is not high throughput operation.

[dreamer-89]

Follow up for plugin owners

With #347, we also added _primary preference for search queries. I wanted to callout the downside of using _primary preference which is that request will fail if the primary is unavailable and not route to replicas that are healthy. Thus, there is availability hit. Thus, if availability is a concern then _primary_first is other option which will attempt primary first and fallback to replicas only if the node is weighed away. Thus, its a question of consistency (_primary) vs avaialbility (_primary_first). I will let plugin owners to decide on that and take necessary action.

Pinging folks for traction @heemin32 @junqiu-lei @naveentatikonda @vamshin