Add python_abi output and refactor

[isuruf]

• Remove clang-19 since we are not building the experimental jit
• Remove ld_impl because we are not hacking the makefiles
• Add site_packages_path
• Make recipe.yaml identical with just the variant name difference
• Add a generate.sh to generate recipe.yaml from default variant

[ngoldbaum]

Ping @lucascolley in case you're curious how Isuru sorted out the python_abi stuff.

[ngoldbaum]

shouldn't this be cp315? Or was this choice on purpose?

[lucascolley]

nice :) and on the Pixi side we have progress towards being able to use a single recipe with pin_subpackage to handle the variants: prefix-dev/pixi#5103

[isuruf]

Can we have one recipe and pass a variant using variants.yaml?

[isuruf]

@lucascolley I had to change this from 0_${{ abi_tag }} because pixi cannot handle this correctly. Pixi would expect a build string of 0_ and complaint that it found 0_cp315. Probably an issue with conda/outputs thing in pixi build.

[lucascolley]

sounds right, a bug report would be perfect

[isuruf]

Opened prefix-dev/pixi#5250

[lucascolley]

Can we have one recipe and pass a variant using variants.yaml?

That would indeed be more simple once the variant-sensitivity is added for Pixi workspace dependencies

[ngoldbaum]

The readme probably needs an update too: https://github.com/python/cpython/tree/main/Tools/pixi-packages#maintenance

[ngoldbaum]

@crusaderky when you’re able to come back to this, let us know if this looks good.

[crusaderky]

I'm -1 on tsan_freethreading. This is because pixi environments can't have underscores, so users will end up with tsan-freethreading environments that point to the tsan_freethreading recipe, which is confusing.
I don't have a preference between tsan-free-threading and tsan-freethreading.

[crusaderky]

My pixi 0.62.2 doesn't seem to notice the workspace.build-variants?

Error:   × failed to solve requirements of environment 'h5py-freethreading' for platform 'linux-64'
  ├─▶   × failed to extract metadata for package 'python'
  │   
  ╰─▶   × Failed to parse recipe
      
      Error:
        × missing field `version`
          ╭─[recipe.yaml:16:9]
       15 │     - package:
       16 │ ╭─▶     name: python_abi
       17 │ ├─▶     version: ${{ version }}
          · ╰──── add the field `version` in between here
       18 │       build:
          ╰────
        help: the field `version` is required for `package`
      
      Error:
        × failed to parse match spec: unable to parse version spec: .* *__
          ╭─[recipe.yaml:22:9]
       21 │     run_constraints:
       22 │       - python ${{ version }}.* *_${{ variant }}_${{ abi_tag }}
          ·         ───────────────────────────┬───────────────────────────
          ·                                    ╰── error parsing `python .* *__` as a match spec
       23 │
          ╰────

[lucascolley]

My pixi 0.62.2 doesn't seem to notice the workspace.build-variants?

yes this still needs work on the Pixi side

[lucascolley]

Can you share your manifest @crusaderky ?

[crusaderky]

Can you share your manifest @crusaderky ?

[workspace]
channels = ["conda-forge"]
platforms = ["linux-64", "osx-arm64"]
preview = ["pixi-build"]

[dependencies]
python.git = "https://github.com/isuruf/cpython"
python.rev = "tsan"
python.subdirectory = "Tools/pixi-packages/default"

$ pixi --version
pixi 0.62.2
$ pixi update
Error:   × failed to solve requirements of environment 'default' for platform 'linux-64'
  ├─▶   × failed to extract metadata for package 'python'
  │   
  ╰─▶   × Failed to parse recipe
      
      Error:
        × missing field `version`
          ╭─[recipe.yaml:16:9]
       15 │     - package:
       16 │ ╭─▶     name: python_abi
       17 │ ├─▶     version: ${{ version }}
          · ╰──── add the field `version` in between here
       18 │       build:
          ╰────
        help: the field `version` is required for `package`
      
      Error:
        × failed to parse match spec: unable to parse version spec: .* *__
          ╭─[recipe.yaml:22:9]
       21 │     run_constraints:
       22 │       - python ${{ version }}.* *_${{ variant }}_${{ abi_tag }}
          ·         ───────────────────────────┬───────────────────────────
          ·                                    ╰── error parsing `python .* *__` as a match spec
       23 │
          ╰────

[baszalmstra]

Yeah the problem is that variants are local to a single workspace, so including a package from another workspace wont include those variants.

In the next pixi release (0.63) ${{ pin_subpackage(..) }} will work properly in recipes. Then it should be possible to create a single recipe with multiple outputs. You could then depend on a helper package to select the right variant, e.g. you could depend on python-asan, which would pull in an asan build of python. Similar to how you can use python-freethreading, python-gil, etc.

In a future release, it should then also be possible to write:

[dependencies]
python = { git = "..", build = "*asan" }

But I personally think that depending on named helper packages is more flexible and future-proof.

[lucascolley]

That seems fine to me for now. Unfortunately the recipe becomes more complicated with the multiple outputs, but that is balanced by having only a single Pixi manifest I guess.

[lucascolley]

I'm sure you will be the fastest out of us to refactor into using pin_subpackage Isuru, but I'm happy to take a look if you prefer

[isuruf]

Then it should be possible to create a single recipe with multiple outputs.

This would create multiple variants of python when we only really need one and seems like a waste.

[crusaderky]

@isuruf I tried hardcoding the context in each recipe as follows to bypass the above issue

context:
  # Keep up to date
  version: "3.15"
  abi_tag: cp315
  variant: default
  freethreading_tag: ${{ "t" if "freethreading" in variant else "" }}

but the recipe fails at the end of compilation:

  ╰─▶   × The build backend did not return the expected package: linux-64/python=3.15=0__. Instead the build backend returned linux-64/
        │ python=3.15=0_default_cp315.

[isuruf]

@crusaderky we need the next release of pixi for that.