[pypi] Support python micro version in env marker based dependency evaluation

[psalaberria002]

🚀 feature request

Relevant Rules

The whl_installer Python program needs to be updated.

Description

whl_installer should be aware of the selected full Python version (..) when resiolving dependencies.

Today, it is only aware of the minor version.

See the Redis 5.1.1 wheel. In the METADATA file it has Requires-Dist: async-timeout>=4.0.3; python_full_version < "3.11.3".

If the Python runtime is set to 3.11.10, the async-timeout dependency should be excluded, but it isn't. The repo BUILD file looks like this:

load("@bazel_skylib//rules:copy_file.bzl", "copy_file")
load("@rules_python//python:defs.bzl", "py_library", "py_binary")

package(default_visibility = ["//visibility:public"])

filegroup(
    name = "dist_info",
    srcs = glob(["site-packages/*.dist-info/**"], allow_empty = True),
)

filegroup(
    name = "data",
    srcs = glob(["data/**"], allow_empty = True),
)

filegroup(
    name = "whl",
    srcs = ["redis-5.1.1-py3-none-any.whl"],
    data = ["@python_deps//async_timeout:whl"],
    visibility = ["//visibility:public"],
)

py_library(
    name = "pkg",
    srcs = glob(
        ["site-packages/**/*.py"],
        exclude=[],
        # Empty sources are allowed to support wheels that don't have any
        # pure-Python code, e.g. pymssql, which is written in Cython.
        allow_empty = True,
    ),
    data = [] + glob(
        ["site-packages/**/*"],
        exclude=["**/* *", "**/*.py", "**/*.pyc", "**/*.pyc.*", "**/*.dist-info/RECORD"],
    ),
    # This makes this directory a top-level in the python import
    # search path for anything that depends on this.
    imports = ["site-packages"],
    deps = ["@python_deps//async_timeout:pkg"],
    tags = ["pypi_name=redis", "pypi_version=5.1.1"],
    visibility = ["//visibility:public"],
)

Describe the solution you'd like

[aignas]

This is something of a known issue and the current design has this limitation described in the issue. To properly solve this I would like to have the following solution:

• Implement a PEP508 parser in starlark so that we can evaluate things outside the repo rule context. We can use FeatureFlagInfo to pass the result of the marker evaluation to a config_setting. I tested if it is possible to write a sane parser and have the code working for this, but did not want to create a PR unless I find a good way to use it. [pypi] PEP508 Remove the need for Python in env_marker evaluation in hub repos #2423
• Use the said method to pass markers from requirements.txt files to the hub_repository for selecting which actual whl_library gets selected. This will simplify the module extension code and will allow us to test the PEP508 parser and the config_setting machinery better. It is a simpler thing than going for the deps straight away. [pypi] PEP508 Remove the need for Python in env_marker evaluation in hub repos #2423
• Use the said method to replace the whl_installer parsing of the METADATA and create a converter from a list of METADATA Requires-Deps lines to a list of dependency specifiers.

This is a lot of yak shaving for achieving what is asked in the issue, but I do think that moving the evaluation of the markers to the analysis phase is actually the right thing to do here.

Until then, one could possibly patch the piece of code that determines the micro version used in evaluating the specifiers in the wheel_installer code. Let me know if a patch is needed.