feat: support metrics scrape

[jiuker]

feat: support metrics scrape
fix #2327
we just support /minio/v2/metrics/cluster before
now we can do more type for this
like

prometheusOperatorScrapeMetricsPath:
- /minio/v2/metrics/cluster
- /minio/v3/metrics/cluster
- /minio/v2/metrics/bucket

default is /minio/v2/metrics/cluster

[harshavardhana]

We should add a way to scrape v3 metrics as well

[cesnietor]

@jiuker is there a way we can create tests for this? since this is a new feature? like an integration test? Thanks.

[allanrogerr]

Why is this, and other similar changes, to the tenant crd needed?

[allanrogerr]

reflectDeepEqual will always return false due to the implementation of GetPrometheusConfig. This will trigger multiple unnecessary secret recreation.

Please consider the following option instead:

// add new scrape configs
    exceptedScrapeConfigs = append(exceptedScrapeConfigs, promCfg.ScrapeConfigs...)
    // compare scrape configs excluding bearer token to avoid continuous updates
    updateScrapeConfig := false
    if len(scrapeConfigs) != len(exceptedScrapeConfigs) {
      updateScrapeConfig = true
    } else {
      for i, _ := range scrapeConfigs {
        if scrapeConfigs[i].JobName != exceptedScrapeConfigs[i].JobName ||
          scrapeConfigs[i].MetricsPath != exceptedScrapeConfigs[i].MetricsPath ||
          scrapeConfigs[i].Scheme != exceptedScrapeConfigs[i].Scheme ||
          !reflect.DeepEqual(scrapeConfigs[i].TLSConfig, exceptedScrapeConfigs[i].TLSConfig) ||
          !reflect.DeepEqual(scrapeConfigs[i].StaticConfigs, exceptedScrapeConfigs[i].StaticConfigs) {
          updateScrapeConfig = true
          break
        }
      }
    }
    if updateScrapeConfig {
      klog.Infof("Adding MinIO tenant Prometheus scrape config")
      scrapeConfigs = exceptedScrapeConfigs
      scrapeCfgYaml, err := yaml.Marshal(scrapeConfigs)
      if err != nil {
        return err
      }
      secret.Data[miniov2.PrometheusAddlScrapeConfigKey] = scrapeCfgYaml
      _, err = c.kubeClientSet.CoreV1().Secrets(ns).Update(ctx, secret, metav1.UpdateOptions{})
      if err != nil {
        return err
      }
    }

[pjuarezd]

please take a look @jiuker