[Segment Replication] Expose utility to denote whether data can be stale

[dreamer-89]

With #8211, few plugins resorted to _primary preference to support strong reads for all search requests (e.g sql). This is not needed for DOCUMENT replication and thus, puts un-necessary burden on primary. To keep plugins/consumers agnostic of replication strategy, core can expose a utility to identify if replica can return stale data and thus hint consumers to add _primary preference only when needed.

In core, we can expose a utility which returns boolean denoting that client should use _primary preference if latest data is desired

Alternative

1. Do not change anything. All requests will land on primary including requests for DOCUMENT replication type indices
2. Use _primary preference only for SEGMENT replication enabled indices. This needs clients to know about underlying replication strategy.

[mch2]

Hi @dreamer-89.

A better long term solution here is to set a consistency policy in requests. I'm working on getting a proposal going for this cc @nknize

Until we have that though, clients that need a strong read with SR will need to add the _primary preference to search requests based on replication strategy. To make this lookup easy can we reuse a function on ClusterState proposed with https://github.com/opensearch-project/OpenSearch/pull/9536/files#r1304689150?