Skip to content

[ci] Call build_*_python_packages in ci_*.yml workflows. #1630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 1, 2025
Merged

[ci] Call build_*_python_packages in ci_*.yml workflows. #1630

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a926bb0
Call build_*_python_packages in ci_*.yml workflows.
ScottTodd Sep 29, 2025
2f6c4fc
Merge remote-tracking branch 'upstream/main' into users/scotttodd/ci-…
ScottTodd Sep 30, 2025
79bca40
Add "Python" to workflow names.
ScottTodd Sep 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build_portable_linux_python_packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ permissions:

jobs:
build:
name: Build | ${{ inputs.amdgpu_family }}
name: Build Python | ${{ inputs.amdgpu_family }}
# Note: GitHub-hosted runners run out of disk space for some gpu families
runs-on: ${{ github.repository_owner == 'ROCm' && 'azure-linux-scale-rocm' || 'ubuntu-24.04' }}
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build_windows_python_packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ permissions:

jobs:
build:
name: Build | ${{ inputs.amdgpu_family }}
name: Build Python | ${{ inputs.amdgpu_family }}
runs-on: ${{ github.repository_owner == 'ROCm' && 'azure-windows-scale-rocm' || 'windows-2022' }}
env:
ARTIFACT_RUN_ID: "${{ inputs.artifact_run_id != '' && inputs.artifact_run_id || github.run_id }}"
Expand Down
37 changes: 32 additions & 5 deletions .github/workflows/ci_linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ permissions:

jobs:
build_portable_linux_artifacts:
name: Build
name: Build Artifacts
if: ${{ inputs.use_prebuilt_artifacts == 'false' }}
uses: ./.github/workflows/build_portable_linux_artifacts.yml
secrets: inherit
Expand All @@ -30,11 +30,19 @@ jobs:
contents: read
id-token: write

# TODO: rework "artifact_run_id" and "use_prebuilt_artifacts" here?
# I don't want to copy/paste this condition and special case plumbing
# through multiple workflows. All the packaging and testing workflows need
# to know is what artifact run id to use. That could be the current
# (implicit) run id, or it could be an explicit run id.
# How about having the "build artifacts" job run as a passthrough?

test_linux_artifacts:
needs: [build_portable_linux_artifacts]
name: Test
name: Test Artifacts
# If the dependent job failed/cancelled, this job will not be run
# The use_prebuilt_artifacts "or" statement ensures that tests will run if previous build step is run or skipped.concurrency.
# The use_prebuilt_artifacts "or" statement ensures that tests will run if
# previous build step is run or skipped.concurrency.
if: >-
${{
!failure() &&
Expand All @@ -44,10 +52,29 @@ jobs:
inputs.use_prebuilt_artifacts == 'true'
)
}}
strategy:
fail-fast: false
uses: ./.github/workflows/test_artifacts.yml
with:
amdgpu_families: ${{ inputs.amdgpu_families }}
test_runs_on: ${{ inputs.test_runs_on }}
artifact_run_id: ${{ inputs.artifact_run_id }}

build_portable_linux_python_packages:
needs: [build_portable_linux_artifacts]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am aware that our CI is still flaky, but is it desired to change this to

Suggested change
needs: [build_portable_linux_artifacts]
needs: [build_portable_linux_artifacts, test_linux_artifacts]

in the future to only proceed if tests passed? Might okay to build those in parallel to get a signal earlier though. If it should be a requirement please add a comment, otherwise ignore this comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Artifact testing can take multiple hours, when I'd like early signal for package construction and sanity checks. When we upload these Python packages to an index (especially as part of release pipelines), we'll want a similar setup to what we have for the torch packages where we first upload to a staging directory/index and later promote once enough tests have passed.

name: Build Python
# If the dependent job failed/cancelled, this job will not be run
# The use_prebuilt_artifacts "or" statement ensures that tests will run if
# previous build step is run or skipped.concurrency.
if: >-
${{
!failure() &&
!cancelled() &&
(
inputs.use_prebuilt_artifacts == 'false' ||
inputs.use_prebuilt_artifacts == 'true'
)
}}
uses: ./.github/workflows/build_portable_linux_python_packages.yml
with:
artifact_run_id: "${{ inputs.artifact_run_id != '' && inputs.artifact_run_id || github.run_id }}"
amdgpu_family: ${{ inputs.amdgpu_families }}
package_version: "7.0.0.dev0" # TODO: setup version with a script/workflow
37 changes: 32 additions & 5 deletions .github/workflows/ci_windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ permissions:

jobs:
build_windows_artifacts:
name: Build
name: Build Artifacts
if: ${{ inputs.use_prebuilt_artifacts == 'false' }}
uses: ./.github/workflows/build_windows_artifacts.yml
with:
Expand All @@ -33,11 +33,19 @@ jobs:
contents: read
id-token: write

# TODO: rework "artifact_run_id" and "use_prebuilt_artifacts" here?
# I don't want to copy/paste this condition and special case plumbing
# through multiple workflows. All the packaging and testing workflows need
# to know is what artifact run id to use. That could be the current
# (implicit) run id, or it could be an explicit run id.
# How about having the "build artifacts" job run as a passthrough?

test_windows_artifacts:
needs: [build_windows_artifacts]
name: Test
name: Test Artifacts
# If the dependent job failed/cancelled, this job will not be run
# The use_prebuilt_artifacts "or" statement ensures that tests will run if previous build step is run or skipped.concurrency.
# The use_prebuilt_artifacts "or" statement ensures that tests will run if
# previous build step is run or skipped.concurrency.
if: >-
${{
!failure() &&
Expand All @@ -47,10 +55,29 @@ jobs:
inputs.use_prebuilt_artifacts == 'true'
)
}}
strategy:
fail-fast: false
uses: ./.github/workflows/test_artifacts.yml
with:
amdgpu_families: ${{ inputs.amdgpu_families }}
test_runs_on: ${{ inputs.test_runs_on }}
artifact_run_id: ${{ inputs.artifact_run_id }}

build_windows_python_packages:
needs: [build_windows_artifacts]
name: Build Python
# If the dependent job failed/cancelled, this job will not be run
# The use_prebuilt_artifacts "or" statement ensures that tests will run if
# previous build step is run or skipped.concurrency.
if: >-
${{
!failure() &&
!cancelled() &&
(
inputs.use_prebuilt_artifacts == 'false' ||
inputs.use_prebuilt_artifacts == 'true'
)
}}
uses: ./.github/workflows/build_windows_python_packages.yml
with:
artifact_run_id: "${{ inputs.artifact_run_id != '' && inputs.artifact_run_id || github.run_id }}"
amdgpu_family: ${{ inputs.amdgpu_families }}
package_version: "7.0.0.dev0" # TODO: setup version with a script/workflow
Loading