Ray Weekly Release

[aslonnie]

We plan to do weekly Ray release, so that code changes can reach users faster.

Current. Starting from the release of Ray 2.31.0, we have unified the partial and full platform releases. For the community, this means you will receive the benefits of the Ray weekly release across all platforms: Linux, macOS, and Windows. Your feedback made this possible, and we thank you for your continuous support and patience.

Previously. Weekly release started from Ray 2.11.0. Between 2.11.0 and 2.30.0 version, there were two tiers for releases: full platform version and partial platform versions. In full platform versions, we will release Linux amd64, arm64, MacOS Apple Silicon and Windows. The minor version number will be multiples of 10s, like 2.10, 2.20, 2.30, ... In partial platform versions, we will only release Linux amd64 and MacOS Apple Silicon, and the version numbers will be not multiples of 10s, like 2.11, 2.12, 2.13, ..., 2.21, ...

[aslonnie]

cc @anyscalesam

[PhilippWillms]

Sounds good, but the advantage of "less than quarterly" releases should also reach Windows users.

[aeroaks]

I like the idea.

[DmitriGekhtman]

Will partial platform versions carry the same stability guarantees as full platform versions? (Are you planning to run the same suite of tests for both types of releases?)

[can-anyscale]

The partial platform versions maintain the same quality standards as the full platform versions yes. We've even elevated our focus on stability across teams, for example burning down flaky tests.

Our ability to now deliver Ray's weekly releases is attributable to recent enhancements in our CI system, coupled with the improved rhythm of our engineering processes.

[DmitriGekhtman]

Awesome!

[OpenCoderX]

This is great, although I had to dig to understand when the recent versions after 2.20 did not have Windows wheels. Also, side-note, congrats on the python 3.11 support, that's a big win for us. Cheers to the work you are all doing on 3.12! 🥂

[connorbrinton]

Mixing "partial" platform releases with "full" platform releases on PyPI makes it difficult for me to use Ray in all of the different environments I use it while keeping it up-to-date. I use Ray:

• Directly on my local M1 Mac
• In Linux arm64 containers on my local M1 Mac
  • Using amd64 containers isn't an option for my use case here since Rosetta doesn't support instruction sets like AVX512 that are used by packages in my environment, like TensorFlow
• In Linux amd64 containers in our production Kubernetes cluster

I use Poetry to manage dependencies, which locks Ray to a specific version. When I'm unlucky (which is more often than I would like 😂), the latest version of Ray has a minor version that isn't a multiple of 10 when I install or update it. That version is then locked in my lockfile. Then when I try to install dependencies for my app in a Linux arm64 container, the installation fails since there isn't a compatible wheel or a source package for the current environment.

I'd like to keep Ray up-to-date and use Dependabot to keep other dependencies up-to-date, but there's no way in either Poetry or Ray (or most package management tools I'm aware of) to say "Only install versions where the minor version is a multiple of 10". That's a pretty unusual constraint.

This idea of "partial" platform releases to get updates to users faster makes me think of how PyTorch has a nightly version that provides a lesser stability guarantee, but provides faster access to updates. In order to distinguish between the nightly and stable versions, PyTorch provides a separate PyPI index for nightly versions (i.e., https://download.pytorch.org/whl/nightly/cpu). This kind of separation would provide a much better user experience for my use case with Ray. Of course, it would be a non-zero amount of work to set up a separate package repository and update packaging and release scripts to push "partial" platform releases to this package index.

Another option would be to release "partial" platform releases as pre-releases that never have corresponding stable releases. This would prevent them from being selected by default, while still allowing users to opt-in to using them if they want the latest changes.

Anyways, hopefully this feedback is a useful representation of challenges that users like me are experiencing with the new release process. Thanks for maintaining Ray! 🙂

[can-anyscale]

Unification of partial and full platform release

Starting this week with the release of Ray 2.31.0, we have unified the partial and full platform releases. For the community, this means you will receive the benefits of the Ray weekly release across all platforms: Linux, macOS, and Windows. Your feedback made this possible, and we thank you for your continuous support and patience.

CC: @connorbrinton, @PhilippWillms, @OpenCoderX

[PhilippWillms]

Thanks, @can-anyscale for the heads up and the team around for enabling weekly releases also for WIndows platform!