When CDM is updated to have new mappings, existing indices will not have them currently. Only when a new index is created do we get the updated mappings. Users can manually add mappings with http requests, but that is not practical.
Why do we have new mappings? The most common scenario is that we add mappings to the names subsection of the metric_desc indices. New names are added as we realize a new tool or benchmark needs that metadata for one of their metrics.
I suspect the easiest way to do this is when the opensearch server is started. I believe when a user updates crucible, the opensearch service is stopped and started anyway. This may, however, not be a good solution for a dedicated opensearch server elsewhere. However, that server also probably needs to run crucible update anyway, so maybe it's not a big issue.
This does highlight another issue: for add-result.js, we need to carefully inspect response messages for index failures due to missing mappings. This could also be another place where new mappings are added dynamically. Might even be better to do it here instead of when opensearch is started. From memory, the add-result makes two passes of reading all of the opensearch documents. The first pass is 'get-info' to gather important data for the second pass, which is the actual index work. In the first pass, we could assemble a list of all [sub]field names used in these docs across all indices it will index to, then query those indices to see if some of those mappings are missing, then cross reference with the index templates cdm.js has defined, and if found there, do a bulk update of the affected indices. If the mappings are not found in the index templates found in cdm.js, then the add-run should fail, or perhaps if the user passes a specific option to omit the parts of the documents which are not present in the index templates found in cdm.js. Yeah, this is the better place to do this.
When CDM is updated to have new mappings, existing indices will not have them currently. Only when a new index is created do we get the updated mappings. Users can manually add mappings with http requests, but that is not practical.
Why do we have new mappings? The most common scenario is that we add mappings to the names subsection of the metric_desc indices. New names are added as we realize a new tool or benchmark needs that metadata for one of their metrics.
I suspect the easiest way to do this is when the opensearch server is started. I believe when a user updates crucible, the opensearch service is stopped and started anyway. This may, however, not be a good solution for a dedicated opensearch server elsewhere. However, that server also probably needs to run crucible update anyway, so maybe it's not a big issue.
This does highlight another issue: for add-result.js, we need to carefully inspect response messages for index failures due to missing mappings. This could also be another place where new mappings are added dynamically. Might even be better to do it here instead of when opensearch is started. From memory, the add-result makes two passes of reading all of the opensearch documents. The first pass is 'get-info' to gather important data for the second pass, which is the actual index work. In the first pass, we could assemble a list of all [sub]field names used in these docs across all indices it will index to, then query those indices to see if some of those mappings are missing, then cross reference with the index templates cdm.js has defined, and if found there, do a bulk update of the affected indices. If the mappings are not found in the index templates found in cdm.js, then the add-run should fail, or perhaps if the user passes a specific option to omit the parts of the documents which are not present in the index templates found in cdm.js. Yeah, this is the better place to do this.