Skip to content

Add wanda builds for aarch64 architecture #59938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
andrew-anyscale wants to merge 4 commits into
base: andrew/revup/master/anyscale-image
Choose a base branch
from
Draft

Add wanda builds for aarch64 architecture #59938

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
32347fa
Migrate wheel builds to wanda (x86_64)
andrew-anyscale Jan 8, 2026
29cf5ab
Add wanda ray image builds for Docker Hub
andrew-anyscale Jan 8, 2026
4a143b7
Add wanda anyscale image builds for release tests
andrew-anyscale Jan 8, 2026
7e7a73e
Add wanda builds for aarch64 architecture
andrew-anyscale Jan 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
351 changes: 299 additions & 52 deletions .buildkite/build.rayci.yml
View file
Open in desktop

Large diffs are not rendered by default.

273 changes: 235 additions & 38 deletions .buildkite/linux_aarch64.rayci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,15 @@ steps:
MANYLINUX_VERSION: "260103.868e54c"
instance_type: builder-arm64

- name: ray-dashboard-build-aarch64
label: "wanda: dashboard (aarch64)"
wanda: ci/docker/ray-dashboard.wanda.yaml
env:
HOSTTYPE: "aarch64"
MANYLINUX_VERSION: "260103.868e54c"
tags: release_wheels
instance_type: builder-arm64

- name: raycpubase-aarch64
label: "wanda: ray.py{{matrix}}.cpu.base (aarch64)"
tags:
Expand Down Expand Up @@ -133,77 +142,265 @@ steps:
tags: release_wheels
instance_type: builder-arm64

- label: ":tapioca: build: wheel {{matrix}} (aarch64)"
- name: ray-wheel-build-aarch64
label: "wanda: wheel py{{matrix}} (aarch64)"
wanda: ci/docker/ray-wheel.wanda.yaml
matrix:
- "3.10"
- "3.11"
- "3.12"
- "3.13"
env:
PYTHON_VERSION: "{{matrix}}"
ARCH_SUFFIX: "-aarch64"
HOSTTYPE: "aarch64"
MANYLINUX_VERSION: "260103.868e54c"
tags:
- release_wheels
- linux_wheels
- oss
instance_type: builder-arm64
depends_on:
- ray-core-build-aarch64
- ray-java-build-aarch64
- ray-dashboard-build-aarch64

# Build ray-cpp core (C++ headers, libs, examples) for aarch64
- name: ray-cpp-core-build-aarch64
label: "wanda: cpp core py{{matrix}} (aarch64)"
wanda: ci/docker/ray-cpp-core.wanda.yaml
matrix:
- "3.10"
- "3.11"
- "3.12"
env:
PYTHON_VERSION: "{{matrix}}"
ARCH_SUFFIX: "-aarch64"
HOSTTYPE: "aarch64"
MANYLINUX_VERSION: "260103.868e54c"
tags:
- release_wheels
- oss
instance_type: medium-arm64
instance_type: builder-arm64

# Build ray-cpp wheel for aarch64
- name: ray-cpp-wheel-build-aarch64
label: "wanda: cpp wheel py{{matrix}} (aarch64)"
wanda: ci/docker/ray-cpp-wheel.wanda.yaml
matrix:
- "3.10"
- "3.11"
- "3.12"
env:
PYTHON_VERSION: "{{matrix}}"
ARCH_SUFFIX: "-aarch64"
HOSTTYPE: "aarch64"
MANYLINUX_VERSION: "260103.868e54c"
tags:
- release_wheels
- linux_wheels
- oss
instance_type: builder-arm64
depends_on:
- ray-core-build-aarch64
- ray-cpp-core-build-aarch64
- ray-java-build-aarch64
- ray-dashboard-build-aarch64

# Upload wheels to S3 (aarch64)
- label: ":s3: upload: wheel py{{matrix}} (aarch64)"
key: linux_wheels_upload_aarch64
instance_type: small-arm64
commands:
- bazel run //ci/ray_ci:build_in_docker -- wheel --python-version {{matrix}} --architecture aarch64 --upload
- ./ci/build/extract_wanda_wheels.sh ray-wheel-py{{matrix}}-aarch64 --upload
matrix:
- "3.10"
- "3.11"
- "3.12"
- "3.13"
depends_on:
- manylinux-aarch64
- forge-aarch64
job_env: forge-aarch64
- ray-wheel-build-aarch64
tags:
- release_wheels
- skip-on-premerge
- oss

- label: ":tapioca: build: ray-extra py{{matrix}} docker (aarch64)"
key: ray_extra_images_aarch64
# Upload cpp wheels to S3 (aarch64)
- label: ":s3: upload: cpp wheel py{{matrix}} (aarch64)"
key: linux_cpp_wheels_upload_aarch64
instance_type: small-arm64
commands:
- ./ci/build/extract_wanda_wheels.sh ray-cpp-wheel-py{{matrix}}-aarch64 --upload
matrix:
- "3.10"
- "3.11"
- "3.12"
depends_on:
- ray-cpp-wheel-build-aarch64
tags:
- release_wheels
- skip-on-premerge
- oss

- name: ray-extra-image-cpu-build-aarch64
label: "wanda: ray-extra py{{matrix}} cpu (aarch64)"
wanda: ci/docker/ray-extra-image-cpu.wanda.yaml
matrix:
- "3.10"
- "3.11"
- "3.12"
env:
PYTHON_VERSION: "{{matrix}}"
ARCH_SUFFIX: "-aarch64"
tags:
- python_dependencies
- docker
- oss
instance_type: medium-arm64
commands:
- bazel run //ci/ray_ci:build_in_docker -- docker --python-version {{matrix}}
--platform cu11.7.1-cudnn8 --platform cu11.8.0-cudnn8
--platform cu12.1.1-cudnn8 --platform cu12.3.2-cudnn9
--platform cu12.4.1-cudnn --platform cu12.5.1-cudnn
--platform cu12.6.3-cudnn --platform cu12.8.1-cudnn
--platform cu12.9.1-cudnn
--platform cpu
--image-type ray-extra --architecture aarch64 --upload
instance_type: builder-arm64
depends_on:
- manylinux-aarch64
- forge-aarch64
- raycudabaseextra-aarch64
- ray-wheel-build-aarch64
- raycpubaseextra-aarch64
Comment on lines +245 to 262
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The previous implementation for building ray-extra images (lines 154-179 in the original file) included an --upload flag, which resulted in the images being pushed. The new wanda-based build steps for ray-extra only build and cache the images in ECR, but they are not pushed to a public registry like Docker Hub. This appears to be a regression. To restore the original behavior, push steps for ray-extra images should be added, similar to the ones implemented for the ray images (e.g., lines 363-385).

job_env: forge-aarch64

- name: ray-extra-image-cuda-build-aarch64
label: "wanda: ray-extra py{{matrix.python}} cu{{matrix.cuda}} (aarch64)"
wanda: ci/docker/ray-extra-image-cuda.wanda.yaml
matrix:
setup:
python:
- "3.10"
- "3.11"
- "3.12"
cuda:
- "11.7.1-cudnn8"
- "11.8.0-cudnn8"
- "12.1.1-cudnn8"
- "12.3.2-cudnn9"
- "12.4.1-cudnn"
- "12.5.1-cudnn"
- "12.6.3-cudnn"
- "12.8.1-cudnn"
- "12.9.1-cudnn"
env:
PYTHON_VERSION: "{{matrix.python}}"
CUDA_VERSION: "{{matrix.cuda}}"
ARCH_SUFFIX: "-aarch64"
tags:
- python_dependencies
- docker
- oss
instance_type: builder-arm64
depends_on:
- ray-wheel-build-aarch64
- raycudabaseextra-aarch64

- name: ray-image-cpu-build-aarch64
label: "wanda: ray py{{matrix}} cpu (aarch64)"
wanda: ci/docker/ray-image-cpu.wanda.yaml
matrix:
- "3.10"
- "3.11"
- "3.12"
env:
PYTHON_VERSION: "{{matrix}}"
ARCH_SUFFIX: "-aarch64"
tags:
- python_dependencies
- docker
- oss
instance_type: builder-arm64
depends_on:
- ray-wheel-build-aarch64
- raycpubase-aarch64

- label: ":tapioca: build: ray py{{matrix}} docker (aarch64)"
key: ray_images_aarch64
- name: ray-image-cuda-build-aarch64
label: "wanda: ray py{{matrix.python}} cu{{matrix.cuda}} (aarch64)"
wanda: ci/docker/ray-image-cuda.wanda.yaml
matrix:
setup:
python:
- "3.10"
- "3.11"
- "3.12"
cuda:
- "11.7.1-cudnn8"
- "11.8.0-cudnn8"
- "12.1.1-cudnn8"
- "12.3.2-cudnn9"
- "12.4.1-cudnn"
- "12.5.1-cudnn"
- "12.6.3-cudnn"
- "12.8.1-cudnn"
- "12.9.1-cudnn"
env:
PYTHON_VERSION: "{{matrix.python}}"
CUDA_VERSION: "{{matrix.cuda}}"
ARCH_SUFFIX: "-aarch64"
tags:
- python_dependencies
- docker
- oss
instance_type: medium-arm64
commands:
- bazel run //ci/ray_ci:build_in_docker -- docker --python-version {{matrix}}
--platform cu11.7.1-cudnn8 --platform cu11.8.0-cudnn8
--platform cu12.1.1-cudnn8 --platform cu12.3.2-cudnn9
--platform cu12.4.1-cudnn --platform cu12.5.1-cudnn
--platform cu12.6.3-cudnn --platform cu12.8.1-cudnn
--platform cu12.9.1-cudnn
--platform cpu
--image-type ray --architecture aarch64 --upload
instance_type: builder-arm64
depends_on:
- manylinux-aarch64
- forge-aarch64
- ray-wheel-build-aarch64
- raycudabase-aarch64
- raycpubase-aarch64
job_env: forge-aarch64

# Push ray CPU images to Docker Hub (aarch64)
- label: ":docker: push: ray py{{matrix}} cpu (aarch64)"
key: ray_images_aarch64
instance_type: small-arm64
commands:
- bazel run //.buildkite:copy_files -- --destination docker_login
- bazel run //ci/ray_ci/automation:push_ray_image --
--python-version {{matrix}}
--platform cpu
--architecture aarch64
--upload
matrix:
- "3.10"
- "3.11"
- "3.12"
depends_on:
- ray-image-cpu-build-aarch64
tags:
- python_dependencies
- docker
- skip-on-premerge
- oss

# Push ray CUDA images to Docker Hub (aarch64)
- label: ":docker: push: ray py{{matrix.python}} cu{{matrix.cuda}} (aarch64)"
key: ray_images_cuda_aarch64
instance_type: small-arm64
commands:
- bazel run //.buildkite:copy_files -- --destination docker_login
- bazel run //ci/ray_ci/automation:push_ray_image --
--python-version {{matrix.python}}
--platform cu{{matrix.cuda}}
--architecture aarch64
--upload
matrix:
setup:
python:
- "3.10"
- "3.11"
- "3.12"
cuda:
- "11.7.1-cudnn8"
- "11.8.0-cudnn8"
- "12.1.1-cudnn8"
- "12.3.2-cudnn9"
- "12.4.1-cudnn"
- "12.5.1-cudnn"
- "12.6.3-cudnn"
- "12.8.1-cudnn"
- "12.9.1-cudnn"
depends_on:
- ray-image-cuda-build-aarch64
tags:
- python_dependencies
- docker
- skip-on-premerge
- oss

- label: ":ray: core: wheel-aarch64 tests"
tags: linux_wheels
Expand Down
Loading