Design scalability improvements (search)

[shirey]

Design scalability improvements with input from IU and Harvard.

T9 MAJOR ACTIVITIES (for all milestones in T9, not just this milestone)

Update Elasticsearch indices and Search API to fix current issues with scalability

OUTCOMES (for all milestones in T9, not just this milestone)

We will refactor the Search API to improve the efficiency when the indices are being reindexed by allowing for incremental updates instead of having to replace everything as well as analyzing current procedures and improving the performance of these procedures when possible. Additionally we analyze the number of attributes that we track to remove those that aren’t needed and move those that aren’t needed in the main index store to a secondary index.

Tasks

Beta Give feedback

1. New reindex-all script feature with dedicated logging #781
   P +1
   
2. Troubleshooting PROD - Request size exceeded 104857600 bytes #843
   +0
   
3. Add validator to avoid infinite loop in search-api reindex procedure entity-api#695
   +0
   
4. Stash large response payload to S3 entity-api#685
   P +1
   
5. DRY - only call get_organ_types() once #841
   +0
   
6. Elastic analysis script to size attributes of indexes based on documents in provenance x-atlas-consortia/elastic-analysis#1
   +0