[Runtime Env] Add support for custom transport parameters in smart_open

[dkhachyan]

Description

This PR adds support for custom transport parameters in smart_open when downloading packages from remote storage systems in Ray's runtime environment. This enables users to customize how packages are downloaded, including authentication methods, SSL settings, and protocol-specific client configurations.

The implementation allows users to specify transport_params in their runtime_env configuration, which are then passed to smart_open during package downloads. For protocols with default configurations (S3, Azure, GCS, ABFSS), custom parameters are merged with defaults, with custom parameters taking precedence.

Related issues

Fixes #46833

Additional information

Implementation Details

1. Modified download_and_unpack_package() in python/ray/_private/runtime_env/packaging.py to accept transport_params
2. Updated PyModulesPlugin and WorkingDirPlugin to extract transport_params from runtime_env
3. Enhanced protocol handlers in python/ray/_private/runtime_env/protocol.py to merge custom parameters with defaults
4. Added _merge_transport_params() helper method to properly combine default and custom parameters

Usage Examples

GitLab Integration (resolves #46833)

ray.init(
    runtime_env={
        "working_dir": "https://gitlab.example.com/group/project/-/archive/main.zip",
        "config": {
            "transport_params": {
                "headers": {
                    "PRIVATE-TOKEN": "TOKEN",
                },
            },
        },
    }
)

[gemini-code-assist]

Code Review

This pull request adds valuable support for custom transport_params in smart_open, allowing for more flexible remote package downloads. The changes are well-structured across the affected files. My review includes a few suggestions for improvement: a critical fix for a potential UnboundLocalError, a suggestion to make parameter merging more robust, a refactoring to reduce code duplication, and a minor simplification. Overall, this is a great addition with the recommended changes.

[cursor]

ABFSS protocol ignores user-provided transport parameters

The open_file function in _handle_abfss_protocol accepts transport_params as a keyword argument but completely ignores it. The function always creates an AzureBlobFileSystem with hardcoded DefaultAzureCredential(), so any custom credentials, account keys, or SAS tokens that users provide via transport_params are silently discarded. This contradicts the PR's stated behavior that custom parameters would be merged and used for ABFSS protocol.

Additional Locations (1)

• python/ray/_private/runtime_env/protocol.py#L237-L241

 

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[edoakes]

@dkhachyan we have an existing "config" field that contains some configuration options related to runtime_env setup: https://docs.ray.io/en/latest/ray-core/api/doc/ray.runtime_env.RuntimeEnvConfig.html#ray.runtime_env.RuntimeEnvConfig. It's passed as a separate field in the runtime_env. I would suggest that we next "transport_options" as a subkey under "config" to avoid adding more fields to the global namespace.

[dkhachyan]

we have an existing "config" field that contains some configuration options related to runtime_env setup: https://docs.ray.io/en/latest/ray-core/api/doc/ray.runtime_env.RuntimeEnvConfig.html#ray.runtime_env.RuntimeEnvConfig. It's passed as a separate field in the runtime_env. I would suggest that we next "transport_options" as a subkey under "config" to avoid adding more fields to the global namespace.

Thanks for the feedback! I’ll update the code accordingly.

[edoakes]

LGTM. Thanks for the contribution

[edoakes]

@dkhachyan there is a merge conflict and I've kicked off the premerge CI: https://buildkite.com/ray-project/premerge/builds/57069

Ping me once CI is passing to merge

[cursor]

HTTPS headers lost when custom headers provided

Medium Severity

When custom headers are passed via transport_params for HTTPS, the default User-Agent and Accept headers are completely replaced rather than merged. The _handle_https_protocol() returns None for default transport params, so _merge_transport_params(None, custom_params) just returns the custom params unchanged. Then inside open_file, params.update(transport_params) does a shallow update that replaces the entire headers dict. This breaks the documented GitLab use case where users add authentication headers but lose the default headers that some servers may require.

Additional Locations (1)

• python/ray/_private/runtime_env/protocol.py#L301-L319