fix(kubeflow): stream logs once, not per replica

torchx calls scheduler.log_iter(app_id, role_name, k=...) once per replica
(k = 0..num_nodes-1). The Kubeflow log_iter ignored k and re-ran
fetch_logs — which tails the entire jobset via the jobset-name selector — for
every replica, producing N independent tail streams (each with its own dedup
state) and N-fold-duplicating every console line (prefixed <role>/<k>). At 16
nodes that's 16x the log volume, which also overruns the CI job-log limit on
long runs. Stream only for k == 0; that single tail already covers all ranks
(and writes log-allranks_0.out once).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>

Author: ko3n1g

nemo_run/run/torchx_backend/schedulers/kubeflow.py

@@ -188,6 +188,14 @@ def log_iter(
         if not executor:
             return []
 
+        # fetch_logs tails ALL pods of the jobset in a single call (it powers the
+        # log-allranks_0.out capture and the cross-rank dedup). torchx invokes
+        # log_iter once per replica (k = 0..num_nodes-1); streaming on every k
+        # would re-tail the whole jobset N times — each tail with its own dedup
+        # state — and N×-duplicate every console line. Stream only for k == 0.
+        if k != 0:
+            return []
+
         logs = executor.fetch_logs(job_name=job_name, stream=should_tail)
         if isinstance(logs, str):
             if len(logs) == 0: