[META] Optimised prefix pattern per shard for remote store data and metadata files for higher throughput

[ashking94]

Please describe the end goal of this project

Proposing a path structure that tries to distribute all shards deterministically over multiple prefixes. The end goal of this initiative is to reduce the throttling or rate limiting done by cloud providers. The organic recommendation in such cases by the same cloud providers is to distribute the load to them over multiple prefixes. In this meta, we will breakdown the issue to smaller issues that we would be working on to complete this feature request.

Supporting References

Feature request - #12567

PRs

• Introduce remote store path type in customData in IndexMetadata #12607
• Use cluster default remote store path type during snapshot restore #12753
• Refactor remote store flow to support any path type #12822
• Add missed API visibility annotations for public APIs #12920
• Introduce remote store hash algorithm in customData in IndexMetadata #12986
• Update Shallow Snapshot flows to support remote path type & hash algo #12988
• Add implementation for remote store path types #13103
• Upload remote paths during index creation or full cluster upload #13150
• Add faster scaling composite hash value encoding for remote path #13251
• Add remote path settings to RemoteStoreSettings class #13225
• Upload remote paths during index creation or full cluster upload #13150
• Add version details in remote index path file with code enhancements #13386
• & more to follow

I will keep creating issue as and when I am starting on the pending sub tasks.

Related component

Storage:Performance

[ashking94]

Marking this meta closed since all items have been completed.