[pypi] Support pulling dependencies using direct url specifiers without using an index

[mering]

🐞 bug report

Affected Rule

Module extension:

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(...)

Is this a regression?

Not sure.

Description

The repository rule for pip dependencies doesn't use repository_ctx.download(). This breaks usages of --experimental_downloader_config for example to use a transparent dependency mirror.

🔬 Minimal Reproduction

rules_python/python/private/pypi/whl_library.bzl

Lines 245 to 256 in b6fc2a0

	repo_utils.execute_checked(
	rctx,
	# truncate the requirement value when logging it / reporting
	# progress since it may contain several ' --hash=sha256:...
	# --hash=sha256:...' substrings that fill up the console
	op = op_tmpl.format(name = rctx.attr.name, requirement = rctx.attr.requirement.split(" ", 1)[0]),
	arguments = args,
	environment = environment,
	quiet = rctx.attr.quiet,
	timeout = rctx.attr.timeout,
	logger = logger,
	)

rules_python/python/private/pypi/whl_installer/wheel_installer.py

Lines 168 to 190 in b6fc2a0

	pip_args = (
	[sys.executable, "-m", "pip"]
	+ (["--isolated"] if args.isolated else [])
	+ (["download", "--only-binary=:all:"] if args.download_only else ["wheel"])
	+ ["--no-deps"]
	+ deserialized_args["extra_pip_args"]
	)
	requirement_file = NamedTemporaryFile(mode="wb", delete=False)
	try:
	requirement_file.write(args.requirement.encode("utf-8"))
	requirement_file.flush()
	# Close the file so pip is allowed to read it when running on Windows.
	# For more information, see: https://bugs.python.org/issue14243
	requirement_file.close()
	# Requirement specific args like --hash can only be passed in a requirements file,
	# so write our single requirement into a temp file in case it has any of those flags.
	pip_args.extend(["-r", requirement_file.name])
	env = os.environ.copy()
	env.update(deserialized_args["environment"])
	# Assumes any errors are logged by pip so do nothing. This command will fail if pip fails
	subprocess.run(pip_args, check=True, env=env)

🔥 Exception or Error

Stack trace prints to the code linked above.

🌍 Your Environment

Operating System:

Linux

Output of bazel version:

7.0.2

Rules_python version:

0.37.2

Anything else relevant?

[aignas]

Please see the documentation here: https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#bazel-downloader-and-multi-platform-wheel-hub-repository.

This currently has a limitation that it needs hashes for all of the files in the requirements.txt. The stabilization of this feature is tracked by #260.

[mering]

@aignas How can I use experimental_index_url in combination with dependencies without an index like the following:

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz
tensorflow_graphics @ git+https://github.com/tensorflow/graphics@ca63cf788978db0bde2db4ce488e102c455da2e7

[aignas]

The support for direct references has not been added yet, I'll reopen this issue and rename it to scope it to supporting it, but I am not sure when I can work on it.

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz could be easy enough. Outline of the changes:

• Update the https://github.com/bazelbuild/rules_python/blob/main/python/private/pypi/index_sources.bzl file to support extracting URLs from the requirements file
• Change the code that calls the index to not call it for requirement lines that refer to URLs
• Pipe the url and the filename to the whl_library respective attributes.

For the second one it may be harder - I am not sure how to handle git repos yet, however, with bazel 7.4 the extension can pass a labels within the extension more easily, so one could look into how to do that.

For now the workaround would be to build the wheels from sdists outside bazel and host them in some PyPI compatible artifact mirror and then use experimental_index_url flag to enable the usage of the bazel-downloader.