Improve resource association

[barkbay]

For now metrics are only associated with Pod resources:

elasticsearch-k8s-metrics-adapter/pkg/client/elasticsearch/discovery.go

Lines 195 to 200 in 39a36d9

	r.metrics[metricName] = provider.CustomMetricInfo{
	GroupResource: schema.GroupResource{ // TODO: infer resource from configuration
	Group: "",
	Resource: "pods",
	},
	Namespaced: true,

Also namespace and Pod's name fields are hardcoded in the Elasticsearch query:

{
	"query": {
		"bool": {
			"must": [{
				"exists": {
					"field": "%s"
				}
			}, {
				"match": {
					"kubernetes.namespace": "%s" -- here
				}
			}, {
				"match": {
					"kubernetes.pod.name": "%s" -- here
				}
			}]
		}
	}
}

Not all metrics make sense in the context of Pods. For example it does not make sense to expose system.memory.free as it is not possible to associate this metric to a specific Pod. The first approach so far has been to filter on the metric name and assume documents also have the kubernetes.namespace and kubernetes.pod.name fields:

metricSets:
  - indices: [ 'metricbeat-*' ]
    fields:
      - patterns: [ '^prometheus\.metrics\.' , '^kibana\.stats\.'  ] # only serve metrics with namespace and pod name

We then assume that we get that kind of document from Elasticsearch:

{
  "_index": "metricbeat-7.14.0-2021.09.21-000001",
  "_type": "_doc",
  "_id": "10V6DXwBJBC0gYXF1JYC",
  "fields": {
     "prometheus.metrics.go_memstats_frees_total": [
      123418991
    ],
   [ ... other metrics starting with prometheus.metrics* ...]
   "kubernetes.pod.name": [
      "kube-dns-b4f5c58c7-47twh"
    ],
   "kubernetes.namespace": [
      "kube-system"
    ],
    "@timestamp": [
      "2021-09-22T12:28:44.448Z"
    ]
  }
}

We may want to improve this association mechanism, for example by allowing other resources to be associated with a given metric. Users may also want to use other fields than kubernetes.namespace or kubernetes.pod.name to query for a given metric.

See here how associations are managed with the Prometheus adapter.

[Trojan295]

I would love to see this! I have a use-case, where I would like to scale based on request count coming to an Ingress:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
...yaml
spec:
    metrics:
      - type: Object
        object:
          describedObject:
            apiVersion: networking.k8s.io/v1
            kind: Ingress
            name: http-api
          metric:
            name: nginx_ingress_controller_requests
          target:
            value: "600"
            type: AverageValue

I could use such a query to get the request counts for an Ingress:

{
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "prometheus.metrics.nginx_ingress_controller_requests"
          }
        }
      ],
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-3m",
              "lte": "now"
            }
          }
        }
      ]
    }
  },
  "size": 0,
  "aggs": {
    "by_ingress": {
      "terms": {
        "field": "prometheus.labels.ingress",
        "size": 100
      },
      "aggs": {
        "by_date": {
          "date_histogram": {
            "field": "@timestamp",
            "calendar_interval": "minute",
            "order": {
              "_key": "asc"
            }
          },
          "aggs": {
            "requests": {
              "max": {
                "field": "prometheus.metrics.nginx_ingress_controller_requests"
              }
            },
            "request_rate": {
              "derivative": {
                "buckets_path": "requests"
              }
            }
          }
        }
      }
    }
  }
}

The result:

{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 9,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "by_ingress": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "prod-app",
          "doc_count": 6,
          "by_date": {
            "buckets": [
              {
                "key_as_string": "2022-08-23T11:38:00.000Z",
                "key": 1661254680000,
                "doc_count": 2,
                "requests": {
                  "value": 11189
                }
              },
              {
                "key_as_string": "2022-08-23T11:39:00.000Z",
                "key": 1661254740000,
                "doc_count": 2,
                "requests": {
                  "value": 11189
                },
                "request_rate": {
                  "value": 0
                }
              },
              {
                "key_as_string": "2022-08-23T11:40:00.000Z",
                "key": 1661254800000,
                "doc_count": 2,
                "requests": {
                  "value": 11189
                },
                "request_rate": {
                  "value": 0
                }
              }
            ]
          }
        },
        {
          "key": "stage-app",
          "doc_count": 3,
          "by_date": {
            "buckets": [
              {
                "key_as_string": "2022-08-23T11:38:00.000Z",
                "key": 1661254680000,
                "doc_count": 1,
                "requests": {
                  "value": 2
                }
              },
              {
                "key_as_string": "2022-08-23T11:39:00.000Z",
                "key": 1661254740000,
                "doc_count": 1,
                "requests": {
                  "value": 2
                },
                "request_rate": {
                  "value": 0
                }
              },
              {
                "key_as_string": "2022-08-23T11:40:00.000Z",
                "key": 1661254800000,
                "doc_count": 1,
                "requests": {
                  "value": 2
                },
                "request_rate": {
                  "value": 0
                }
              }
            ]
          }
        }
      ]
    }
  }
}

So there is a question how to associate the fields in the ES query result to the K8s resources.