pip.parse parsing all requirements files fails on platform specific wheel downloads

[keith]

🐞 bug report

Affected Rule

On this branch: #2449 which sets up pypi for 3 different platforms with different requirements files. Running bazel query 'deps(...)' results in a wheel download failure if you're not on x86_64 linux, since the deps for that platform are not available on other platforms:

===== stderr end ===== and referenced by '@@rules_python~~pip~pip//torch:pkg'
ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python~~pip~pip_311_torch_linux_x86_64//':
rules_python:whl_library(@@rules_python~~pip~pip_311_torch_linux_x86_64) FAIL: repo.execute: whl_library.ResolveRequirement(rules_python~~pip~pip_311_torch_linux_x86_64, torch==2.4.1+cpu): end: failure:
  command: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_host/python -m python.private.pypi.whl_installer.wheel_installer --requirement torch==2.4.1+cpu --isolated --extra_pip_args "{\"arg\":[\"--index-url\",\"https://pypi.org/simple\",\"--extra-index-url\",\"https://download.pytorch.org/whl/cpu\"]}" --pip_data_exclude "{\"arg\":[]}" --environment "{\"arg\":{}}"
  return code: 1
  working dir: <default: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~pip~pip_311_torch_linux_x86_64>
  timeout: 600
  environment:
PYTHONPATH="/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__build:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__click:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__colorama:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__importlib_metadata:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__installer:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__more_itertools:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__packaging:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pep517:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pip:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pip_tools:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pyproject_hooks:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__setuptools:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__tomli:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__wheel:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__zipp"
CPPFLAGS="-isystem /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_host/include/python3.11"
===== stdout start =====
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
===== stdout end =====
===== stderr start =====
ERROR: Could not find a version that satisfies the requirement torch==2.4.1+cpu (from versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.5.1)
ERROR: No matching distribution found for torch==2.4.1+cpu
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_aarch64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_host/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--index-url', 'https://pypi.org/simple', '--extra-index-url', 'https://download.pytorch.org/whl/cpu', '-r', '/tmp/tmprruvzh7j']' returned non-zero exit status 1.
===== stderr end =====

Is this a regression?

This was caused by this default flip: daed352

Related to https://github.com/bazelbuild/rules_python/issues/keith

Currently @ HEAD you have to revert the root cause of #2446 to repro this issue

[aignas]

This is somewhat of a known thing - bazel query forces all of the repos to be downloaded, not only the ones that are needed for a particular platform. There are multiple ways this can be solved:

• Users switch to bazel cquery
• Users start using download_only = True attribute.
• Users use download_only = True with experimental_index_url.
• rules_python strips the env marker, in this case platform_machine == foo from the requirement_line that is passed to whl_library. That will make the fetching not fail, but the result that is materialized within the repo rule would be different depending on the platform.

I personally think we as maintainers can only document what users can do and strip the markers from the whl_library because they don't add any value when fetching the wheels.

Would that be acceptable?