Skip to content

Improve URL staging resilience with configurable retry cooldown and regression coverage - #7476

Open
MikeWLloyd wants to merge 1 commit into
nextflow-io:masterfrom
MikeWLloyd:Feature-#7475
Open

Improve URL staging resilience with configurable retry cooldown and regression coverage#7476
MikeWLloyd wants to merge 1 commit into
nextflow-io:masterfrom
MikeWLloyd:Feature-#7475

Conversation

@MikeWLloyd

Copy link
Copy Markdown

Summary

This PR improves reliability for remote input staging (especially HTTP/HTTPS URLs such as raw GitHub links) by making retry cooldown configurable and refining retry classification for transient URL failures. It directly addresses #7475

Motivation

Module and CI tests that stage remote files can fail due to transient DNS/network issues (for example host resolution failures). While staging retries already existed, retry delay was hardcoded and some URL-related error shapes were treated too aggressively as non-retryable.

What changed

  1. Added configurable staging retry cooldown
    • New config key: filePorter.retryDelay
    • Default: 250ms
    • Existing filePorter.maxRetries behavior is preserved
  2. Updated staging retry wait behavior
    • Replaced small hardcoded/random sleep with configured retry delay
    • Retry log now includes the configured wait duration
  3. Refined retry classification for URL staging
    • Keeps fast-fail behavior for clearly non-retryable conditions
    • Allows retry for HTTP/HTTPS NoSuchFileException patterns that appear transient/wrapped
    • Avoids retries for clear missing-resource signals (for example 404/not found)
  4. Added targeted regression tests
    • Retries transient HTTP NoSuchFileException and succeeds
    • Retries transient DNS-like URL failure and succeeds
    • Does not retry clear missing HTTP resource (fails immediately as expected)

Files touched

  1. FilePorter.groovy
  2. FilePorterTest.groovy

Testing

Passed:

  1. JAVA_HOME=$(/usr/libexec/java_home -v 17) ./gradlew :nextflow:test --tests nextflow.file.FilePorterTest
  2. JAVA_HOME=$(/usr/libexec/java_home -v 17) ./gradlew :nextflow:test --tests 'nextflow.file.*'

Backward compatibility

  1. Existing defaults remain compatible
  2. Retry count behavior is unchanged unless users tune settings
  3. New setting is optional

Suggested reviewer focus

  1. Retry classification logic in FilePorter for HTTP/HTTPS NoSuchFileException paths
  2. Config naming/default for filePorter.retryDelay
  3. Coverage and intent of new FilePorter regression tests

@netlify

netlify Bot commented Aug 13, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs canceled.

Name Link
🔨 Latest commit e5c8239
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs/deploys/6a7dcb9156f1ea000861f20d

@pditommaso
pditommaso force-pushed the master branch 2 times, most recently from 5f935c2 to d1eae20 Compare August 20, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant