feat(server_utils): opt-in bounded retry and non-retryable timeouts on request()

[adil-a]

Problem

server_utils.request() retries connection errors forever: the ServerDisconnectedError and ClientOSError branches have no attempt cap (https://github.com/NVIDIA-NeMo/Gym/blob/main/nemo_gym/server_utils.py#L222-L245). MAX_NUM_TRIES only applies when _internal is false, and ServerClient always passes _internal=True. The shared session is also built with an empty ClientTimeout(), so a hung peer blocks forever, and passing timeout= per request does not help because the timeout exception is itself retried.

This is intended behavior for Gym's own localhost servers, but wrong for external endpoints, which need to fail into the failures sidecar instead of hanging the run. remote_agent (#2163) had to bypass request() and hand-roll a bounded transport loop. Four resources servers call request() for external services (tavily_search, critpt, speed_bench, browsecomp_advanced_harness) and inherit the unbounded retry today.

Proposal

Two opt-in kwargs on request(), with defaults keeping current behavior for all existing callers:

1. max_connection_tries: Optional[int] = None. Counts attempts in the two connection error branches and re-raises on exhaustion. None keeps the current infinite retry.
2. retry_on_timeout: bool = True. When False, timeout exceptions re-raise immediately.

remote_agent's transport loop can then collapse onto request(), and the external-service resources servers can opt into bounded failure.

Origin: #2163 (comment)

[sephmard]

Duplicates #2216