Feature/multi loader logs collection

[nosnelmil]

Summary

Extends multi-loader by collecting key logs from nodes in the cluster for the Knative platform. Users can optionally collect the following logs:

• TOP (resource usage metrics)
• Prometheus snapshots
• Logs from the Activator Pod
• Logs from the Autoscaler Pod

Implementation Notes ⚒️

• Added an additional Metrics field in the multi-loader config, accepting an array with any of the following values: top, prometheus, activator, autoscaler.
• Introduced optional fields: MasterNode, ActivatorNode, AutoscalerNode, and WorkerNodes to allow users to manually specify IPs instead of relying on multi-loader to determine them (mostly unnecessary in typical scenarios).
• Uses kubectl to automatically determine node IPs and classify them based on their roles.
• Resets TOP metrics for all nodes before starting any experiment.
• Collects Activator Pod logs from:
  /var/log/pods/knative-serving_activator-*/activator/*
• Collects Autoscaler Pod logs from:
  /var/log/pods/knative-serving_autoscaler-*/autoscaler/*
• Copies Prometheus snapshots by first triggering a snapshot via the Prometheus API on the master node and then retrieving the generated snapshot.
• Additionally, log collection logic runs during multi-loader dry run to:
  • Validate that identified IPs are reachable.
  • Ensure SSH access and necessary permissions.
  • Execute log retrieval commands to detect potential errors.
  • Delete any collected logs after validation, as no experiments have been executed yet.

External Dependencies 🍀

• N/A

Breaking API Changes ⚠️

• N/A

[leokondrashov]

Looks nice. I left some comments.

[nosnelmil]

@leokondrashov as discussed, added the log consolidation logic in 0dc0950