Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(toolchain): support freethreaded toolchains #2372

Merged
merged 27 commits into from
Nov 11, 2024

Conversation

aignas
Copy link
Collaborator

@aignas aignas commented Nov 5, 2024

Before this PR freethreaded toolchains were not possible to be used,
this adds the minimum plumbing to get the things working. Coverage
support is also added.

Whilst at it:

  • Add plumbing to print checksums only for a particular python version.
  • Bump the remaining toolchain versions that used to use the 20241008 release
  • Pass around the loaded platform list so that we are only defining toolchains for the platforms that we have loaded the hermetic toolchain for.

Tested:

$ bazel run --//python/config_settings:python_version=3.13.0 --//python/config_settings:py_freethreaded="yes" //python/private:current_interpreter_executable
...
Python 3.13.0 experimental free-threading build (main, Oct 16 2024, 03:26:14) [Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Closes #2129.
Work towards #2386.

@aignas aignas marked this pull request as draft November 5, 2024 00:40
@vfdev-5
Copy link
Contributor

vfdev-5 commented Nov 5, 2024

Seems like this PR is working to build XLA with Python free-threading using the following command:

bazel build \
    --@rules_python//python/config_settings:py_freethreaded="yes" \
    --define=use_enabled_free_threading=true \
    --test_output=all --spawn_strategy=sandboxed \
    --override_repository=rules_python=/rules_python \
    //xla/...

It downloads however two versions of python:

ls /root/.cache/bazel/_bazel_root/83d0ab0b1a11dcb5adba61b76e88291b/external/python                                                                                                         
...
python_x86_64-unknown-linux-gnu-freethreaded/                                                                            
python_x86_64-unknown-linux-gnu/
...

I assume that free-threaded version is used for building python extensions...

@aignas aignas marked this pull request as ready for review November 5, 2024 13:36
CHANGELOG.md Outdated Show resolved Hide resolved
@aignas
Copy link
Collaborator Author

aignas commented Nov 5, 2024

@vfdev-5, I am not fully sure about why you have the toolchain in your cache. Could you please check it with bazel query or use --toolchain_resolution_debug=python to check?

@vfdev-5
Copy link
Contributor

vfdev-5 commented Nov 5, 2024

In my tests I deleted completely the cache and run xla build from the very beginning. In the console I saw it downloading python for platform x86_64-unknown-linux-gnu and after that x86_64-unknown-linux-gnu-freethreaded.
I'll retry with your suggestions to figure out

Copy link
Collaborator Author

@aignas aignas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done with the self-review.

python/config_settings/BUILD.bazel Show resolved Hide resolved
python/config_settings/BUILD.bazel Outdated Show resolved Hide resolved
python/private/hermetic_runtime_repo_setup.bzl Outdated Show resolved Hide resolved
python/private/hermetic_runtime_repo_setup.bzl Outdated Show resolved Hide resolved
python/private/pythons_hub.bzl Outdated Show resolved Hide resolved
python/private/toolchains_repo.bzl Outdated Show resolved Hide resolved
python/private/toolchains_repo.bzl Show resolved Hide resolved
python/versions.bzl Outdated Show resolved Hide resolved
python/versions.bzl Outdated Show resolved Hide resolved
Copy link
Collaborator

@groodt groodt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@rickeylev rickeylev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if I remember the toolchain generation logic, then all one has to do is set the flag, and it'll trigger downloading and usage of the free-threaded interpreter?

Otherwise just some small comments, LGTM

CHANGELOG.md Outdated Show resolved Hide resolved
python/config_settings/BUILD.bazel Show resolved Hide resolved
@@ -67,19 +69,23 @@ def define_hermetic_runtime_toolchain_impl(
exclude = [
# Unused shared libraries. `python` executable and the `:libpython` target
# depend on `libpython{python_version}.so.1.0`.
"lib/libpython{major}.{minor}.so".format(**version_dict),
"lib/libpython{major}.{minor}*.so".format(**version_dict),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking/curious: I'm guessing the libX.Y* pattern works because the extracted standalone interpreters only contain appropriately relevant .so files?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for exclusion, that is why libX.Y*.so is just as good as libX.Y.so.

python/private/hermetic_runtime_repo_setup.bzl Outdated Show resolved Hide resolved
python/private/python_register_toolchains.bzl Outdated Show resolved Hide resolved
@aignas aignas added this pull request to the merge queue Nov 11, 2024
Merged via the queue into bazelbuild:main with commit 2af0020 Nov 11, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants