[Question]: Is it possible to scrape the Result resources through a log aggregator

[abindg]

Checklist

• I've searched for similar issues and couldn't find anything matching
• I've included steps to reproduce the behavior

Affected Components

• K8sGPT (CLI)
• K8sGPT Operator

K8sGPT Version

v0.3.48

Kubernetes Version

1.30.6

Host OS and its Version

No response

Steps to reproduce

1. When an SRE person sees the result counts on Grafana , instead of logging to k8s and running kubectl get result -n -o json | jq . , is it possible for him to access the solution part of the results in grafana itself through something like loki , which scrapes the results dynamically

Expected behaviour

The SRE should not be required to run through hundreds of results in kubernetes , but see them on grafana

Actual behaviour

The SRE person when he sees the result count in grafana , needs to login and check resuts manually

Additional Information

No response

[AlexsJones]

@abindg there is a service monitor that exists already for prometheus. This will do exactly what you want.

I will take an action to document this better.

[abindg]

The default dashboard, shows the count of reaults and the type of results using the service monitor. The contents of the results are not scrapped

[AlexsJones]

That's quite true, and somewhat intentional, we could emit the results content back into the operator on read, that would let you filter for it in Loki. How does something like that sound?

[abindg]

That sounds good.  As long as I can get the contents of results on Grafana, and the SRE doesnot need to go through running kubectl desctibe on the results - then we have a strong use case Yahoo Mail: Search, Organize, Conquer On Wed, Dec 18, 2024 at 1:56 AM, Alex ***@***.***> wrote: That's quite true, and somewhat intentional, we could emit the results content back into the operator on read, that would let you filter for it in Loki. How does something like that sound? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[AlexsJones]

That sounds good. As long as I can get the contents of results on Grafana, and the SRE doesnot need to go through running kubectl desctibe on the results - then we have a strong use case Yahoo Mail: Search, Organize, Conquer On Wed, Dec 18, 2024 at 1:56 AM, Alex @.> wrote: That's quite true, and somewhat intentional, we could emit the results content back into the operator on read, that would let you filter for it in Loki. How does something like that sound? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.>

Okay, I believe my proposal would work. I'd like it to be opt-in functionality though as in the first version of this I think we'd need to be careful about what is exposed through the operator logs.

[AlexsJones]

how does this fit your need? k8sgpt-ai/k8sgpt-operator#589

[abindg]

This looks good . How can I test this ?

[AlexsJones]

I'll get it merged in @abindg

[abindg]

Ok @AlexsJones . Once merged , then will there be a change in the helm chart ? As far as I understand with this change we can get the Solutions part of the Results into the operator logs . The result name will not be spit out . Is this understanding correct ? Then we can scrape the logs of operator .

[AlexsJones]

Ok @AlexsJones . Once merged , then will there be a change in the helm chart ? As far as I understand with this change we can get the Solutions part of the Results into the operator logs . The result name will not be spit out . Is this understanding correct ? Then we can scrape the logs of operator .

You will just need to do a helm repo update and then reinstall the new chart.. about to cut the release shortly

[abindg]

Ok @AlexsJones . Once merged , then will there be a change in the helm chart ? As far as I understand with this change we can get the Solutions part of the Results into the operator logs . The result name will not be spit out . Is this understanding correct ? Then we can scrape the logs of operator .

You will just need to do a helm repo update and then reinstall the new chart.. about to cut the release shortly

Hi @AlexsJones After installing the new version , the logs are not spitting out the results it was supposed to please find the screen shots below:

Am I missing something ? I installed using -> helm upgrade --install k8sgpt k8sgpt/k8sgpt-operator -n k8sgpt-operator-system --create-namespace --set serviceMonitor.enabled=true --set serviceMonitor.additionalLabels.release=prometheus --set grafanaDashboard.enabled=true

[abindg]

The helm chart needs to be updated to roll this out . The flag --enable-result-logging needs to be set for controllerManager container . And the value needs to be made dynamic through helm . These are not done . I have made the changes locally and it works now .

The Deployment change:

• args:
  - --health-probe-bind-address=:8081
  - --metrics-bind-address=127.0.0.1:8080
  - --leader-elect
  - --enable-result-logging={{ .Values.controllerManager.manager.enableResultLogging | default "false" }} # Added the argument to enable result logging
  command:
  - /manager
  env:
  - name: KUBERNETES_CLUSTER_DOMAIN
  value: {{ quote .Values.kubernetesClusterDomain }}
  - name: OPERATOR_SINK_WEBHOOK_TIMEOUT_SECONDS
  value: {{ quote .Values.controllerManager.manager.sinkWebhookTimeout }}
  image: {{ .Values.controllerManager.manager.image.repository }}:{{ .Values.controllerManager.manager.image.tag
  | default .Chart.AppVersion }}

Correspoding values.yaml change:

manager:
# Add the enableResultLogging variable here, which can be toggled in values.yaml
enableResultLogging: true # This is now configurable in the values.yaml file

I can give a PR . What do you think @AlexsJones