Skip to content

Remove py-show and use pyproject-external project #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rgommers merged 17 commits into from
Jun 5, 2025
Merged

Remove py-show and use pyproject-external project #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
e9c5eb4
Clear root-level a bit
jaimergp Jun 4, 2025
663cfb1
Centralize workflow requirements
jaimergp Jun 4, 2025
520df06
Remove py-show in favor of external-metadata cli
jaimergp Jun 4, 2025
a16df75
Adjust GHA workflow accordingly
jaimergp Jun 4, 2025
0207f11
install git in container
jaimergp Jun 4, 2025
6c6d42e
fix package name
jaimergp Jun 4, 2025
cef1a93
retrigger
jaimergp Jun 4, 2025
3c3753e
fix mambarc location
jaimergp Jun 4, 2025
0008610
retrigger
jaimergp Jun 4, 2025
7d8f908
retrigger
jaimergp Jun 4, 2025
92b155d
quote echo
jaimergp Jun 4, 2025
109800a
move libyaml to host-requires
jaimergp Jun 4, 2025
b9e4078
quote here too
jaimergp Jun 4, 2025
66ebdc3
call eval directly
jaimergp Jun 4, 2025
552d0e3
use shlex branch temporarily
jaimergp Jun 4, 2025
e98e691
move build time patches to correct section
jaimergp Jun 4, 2025
89313b5
revert to install from main
jaimergp Jun 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 .mambarc → .github/.mambarc
View file
Open in desktop
File renamed without changes.
70 changes: 39 additions & 31 deletions .github/workflows/build_all.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,17 +41,17 @@ jobs:
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0

- name: Install system Python
- name: Install system Python and git
Copy link
Owner

@rgommers rgommers Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: why is git needed?

Copy link
Collaborator Author

@jaimergp jaimergp Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Docker images don't have it, and some of our dependencies are installing Python packages from github repositories because they are not released on PyPI yet, more specifically:

I tried using a github archive for that but it failed due to setuptools-scm shenanigans.

Copy link
Owner

@rgommers rgommers Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that explains it - makes sense. It's not an issue at all to have it here, just good to know it can be dropped again in principle ones tooling issues are resolved. I kept it out on purpose because git can also be a runtime dependency.

run: |
if [[ ${{ matrix.platform[1] }} == "Fedora" ]]; then
sudo dnf install -y python # will install 3.12
sudo dnf install -y python git # will install 3.12
elif [[ ${{ matrix.platform[1] }} == "Arch" ]]; then
pacman -Syu python sudo --noconfirm # will install 3.11
pacman -Syu python git sudo --noconfirm # will install 3.11
elif [[ ${{ matrix.platform[1] }} == "conda-forge" ]]; then
"${SHELL}" <(curl -L micro.mamba.pm/install.sh)
source ~/.bashrc
cp .mambarc ~/.mambarc
micromamba create -n cfenv python pip # will install 3.12
cp .github/.mambarc ~/.mambarc
micromamba create -n cfenv python pip git # will install 3.12
fi

- name: Download and patch sdist
Expand All @@ -63,8 +63,8 @@ jobs:
source venv/bin/activate
fi

python -m pip install pypi_json distro typer rich ./py-show
python download_and_patch_sdist.py ${{ matrix.pkgdata[0]}}
python -m pip install -r requirements.txt
python scripts/download_and_patch_sdist.py ${{ matrix.pkgdata[0]}}
ls -alh sdist/
ls -alh sdist/_amended

Expand All @@ -77,14 +77,19 @@ jobs:
fi
name=${{ matrix.pkgdata[0]}}

pyshow_args="--sdist-dir=./sdist/_amended"
cli_args=""
if [[ ${{ matrix.platform[1] }} == "conda-forge" ]]; then
pyshow_args+=" --package-manager=micromamba"
cli_args+=" --package-manager=micromamba"
fi
cli_args+=" ./sdist/_amended/$name-*.tar.gz"

py-show --external $name $pyshow_args
py-show --external --system-install-cmd $name $pyshow_args
eval "$(py-show --external --system-install-cmd $name $pyshow_args)"
echo "Raw external metadata:"
python -m pyproject_external --output=raw $cli_args
echo "Mapped external metadata:"
python -m pyproject_external --output=mapped $cli_args
echo "Install command:"
set +x
eval "$(python -m pyproject_external --output=command $cli_args)"
Copy link
Owner

@rgommers rgommers Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice.


- name: Build from sdist
run: |
Expand All @@ -93,12 +98,6 @@ jobs:
else
source venv/bin/activate
fi
# Build time patches
case ${{ matrix.pkgdata[0] }} in
matplotlib)
export AR="${GCC_AR:gcc-ar}"
;;
esac
# Dependencies may be installed from wheels, the target package from sdist
python -m pip install sdist/_amended/${{ matrix.pkgdata[0] }}-*.tar.gz -v

Expand Down Expand Up @@ -174,20 +173,19 @@ jobs:
- name: Install system Python
run: |
if [[ ${{ matrix.platform[1] }} == "Fedora" ]]; then
sudo dnf install -y python # will install 3.12
sudo dnf install -y python git # will install 3.12
elif [[ ${{ matrix.platform[1] }} == "Arch" ]]; then
pacman -Syu python sudo --noconfirm # will install 3.11
pacman -Syu python git sudo --noconfirm # will install 3.11
elif [[ ${{ matrix.platform[1] }} == "conda-forge" ]]; then
"${SHELL}" <(curl -L micro.mamba.pm/install.sh)
source ~/.bashrc
cp .mambarc ~/.mambarc
micromamba create -n cfenv python pip # will install 3.12
cp .github/.mambarc ~/.mambarc
micromamba create -n cfenv python pip git # will install 3.12
else
echo 'Unsupported distro!'
fi

- name: Download and patch sdist
if: ${{ matrix.platform[2] }}
run: |
if [[ ${{ matrix.platform[1] }} == "conda-forge" ]]; then
source ~/.bashrc && micromamba activate cfenv
Expand All @@ -196,13 +194,12 @@ jobs:
source venv/bin/activate
fi

python -m pip install pypi_json distro typer rich ./py-show
python download_and_patch_sdist.py ${{ matrix.pkgdata[0]}}
python -m pip install -r requirements.txt
python scripts/download_and_patch_sdist.py ${{ matrix.pkgdata[0]}}
ls -alh sdist/
ls -alh sdist/_amended

- name: Install external dependencies
if: ${{ matrix.platform[2] }}
run: |
if [[ ${{ matrix.platform[1] }} == "conda-forge" ]]; then
source ~/.bashrc && micromamba activate cfenv
Expand All @@ -211,14 +208,19 @@ jobs:
fi
name=${{ matrix.pkgdata[0]}}

pyshow_args="--sdist-dir=./sdist/_amended"
cli_args=""
if [[ ${{ matrix.platform[1] }} == "conda-forge" ]]; then
pyshow_args+=" --package-manager=micromamba"
cli_args+=" --package-manager=micromamba"
fi
cli_args+=" ./sdist/_amended/$name-*.tar.gz"

py-show --external $name $pyshow_args
py-show --external --system-install-cmd $name $pyshow_args
$(py-show --external --system-install-cmd $name $pyshow_args)
echo "Raw external metadata:"
python -m pyproject_external --output=raw $cli_args
echo "Mapped external metadata:"
python -m pyproject_external --output=mapped $cli_args
echo "Install command:"
set +x
eval "$(python -m pyproject_external --output=command $cli_args)"

- name: Build from sdist
if: ${{ matrix.platform[2] }}
Expand All @@ -228,6 +230,12 @@ jobs:
else
source venv/bin/activate
fi
# Build time patches
case ${{ matrix.pkgdata[0] }} in
matplotlib)
export AR="${GCC_AR:gcc-ar}"
;;
esac
# Dependencies may be installed from wheels, the target package from sdist
python -m pip install sdist/_amended/${{ matrix.pkgdata[0] }}-*.tar.gz -v

Expand Down
3 changes: 3 additions & 0 deletions external_metadata/pyyaml.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
[external]
build-requires = [
"dep:virtual/compiler/c",
]

host-requires = [
Copy link
Owner

@rgommers rgommers Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, this was a bug before

"dep:generic/libyaml",
]
1 change: 1 addition & 0 deletions external_metadata/scipy.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ build-requires = [
"dep:generic/ninja",
"dep:generic/pkg-config",
]

host-requires = [
"dep:virtual/interface/blas",
"dep:virtual/interface/lapack",
Expand Down
Loading
Loading