[Feature] Make RayJob dashboard polling interval configurable

[Future-Outlier]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, the KubeRay operator queries the Ray dashboard for every RayJob reconciliation to update its status. This feature proposes introducing a configuration option to allow users to control this polling frequency. Instead of querying on every reconciliation loop, users could define a specific interval (e.g., every 10 seconds).

Use case

When many RayJobs are submitted to a single RayCluster, the current behavior can generate a high volume of status query requests. This places a significant load on the Ray head pod, potentially leading to performance bottlenecks and increased resource consumption.

Related issues

#3907

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[Future-Outlier]

To implement this feature, we need a way to pass the polling interval to the controller from the RayJob resource.
During reconciliation, the controller can then use this value.

Based on my understanding, there are two potential approaches:

1. Modify the RayJob API: We could add a new field to the RayJob specification. However, this is generally not recommended as it involves API changes, which can be risky.
2. Use an annotation: We could use an annotation on the RayJob resource to specify the polling interval.

[Future-Outlier]

We will wait for the background go routine PR merged, then verifiy with users if this is needed.