Support using ISM (instead of ILM) for index state management

[oliversalzburg]

Requirement - what kind of business use case are you trying to solve?

I'm trying to deploy Jaeger on AWS. The AWS managed ES does not support ILM, instead they offer ISM.

The existing ES rollover mechanism is prone to error during initialization and use, which is why using ILM/ISM should be strongly preferred.

Problem - what in Jaeger blocks you from solving the requirement?

Currently, using ISM is not possible to init rollover, as the policy URL is hardcoded.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Implement an "ISM Client" alongside ILM and use that if ES_USE_ISM is set.

Any open questions to address

I feel like this relates to: #3044

Also, while trying to make this work, I came up with this. ISM policy syntax differs from ILM. Maybe it helps:

curl -X PUT https://${aws_elasticsearch_domain.domain.endpoint}/_opendistro/_ism/policies/jaeger-ilm-policy \
-H 'Content-Type: application/json; charset=utf-8' \
--data-binary @- << EOF
{
	"policy": {
		"description": "Transition old records out of index.",
		"default_state": "hot",
		"schema_version": 1,
		"states": [
			{
				"name": "hot",
				"actions": [
					{
						"rollover": {
							"min_index_age": "7d"
						}
					}
				],
				"transitions": [
					{
						"state_name": "delete",
						"conditions": {
							"min_index_age": "30d"
						}
					}
				]
			},
			{
				"name": "delete",
				"actions": [
					{
						"delete": {}
					}
				]
			}
		]
	}
}

[dekimsey]

I'm guessing the error I'm seeing trying to use jaeger-collector with ILM on an Opensearch v1.0 cluster directly stems from this issue.

version: 1.32
given settings:
- SPAN_STORAGE_TYPE=opensearch
- ES_SNIFFER=false
- ES_USE_ILM=true
- ES_USE_ALIASES=true

{"level":"fatal","ts":1649710461.9009402,"caller":"./main.go:84","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings [type=illegal_argument_exception]","stacktrace":"main.main.func1\n\t./main.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/[email protected]/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/[email protected]/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/[email protected]/command.go:902\nmain.main\n\t./main.go:147\nruntime.main\n\truntime/proc.go:255"}

[ABAG603]

Are there any plans to support this? Considering OpenSearch is now under Linux Foundation maybe it makes sense to get back to it and have same level of support as Elastic Search has.

[yurishkuro]

I believe the recommended way these days is to use data streams (#4708), not any of these manual rollover mechanisms.