[RFC] Create a 0.x branch

[seankao-az]

Problem Overview

Our current branching strategy faces several challenges:

1. There's no clear path for introducing new features and improvements in a stable manner between minor releases.
2. The main branch is unsuitable for continuous deployment due to potential instability and breaking changes intended for future major releases.
3. The latest minor release branch (0.6) doesn't include new features and improvements, making it inadequate for continuous deployment.
4. We're maintaining a custom branch for continuous deployment, cherry-picking desired commits from the main branch onto the 0.6 branch. This approach is unsustainable and time-consuming.

Proposed Solution

Implement a new branching strategy inspired by OpenSearch branching:

We'll maintain three primary branches:

1. main: Development for the next major release (e.g., 1.0)
2. 0.x: The next minor release (e.g., 0.7), used for continuous deployment
3. 0.6: The current stable release

The workflow is as follows:

1. Merge all new development into the main branch
2. Backport stable, non-breaking changes from main to the 0.x branch
3. Use the 0.x branch for continuous deployment
4. Cut new minor releases from the 0.x branch
5. Backport critical bug fixes to the 0.6 branch and other supported release branches

Benefits for this approach:

1. Provides a stable branch (0.x) for continuous deployment
2. Allows for new feature introduction between minor releases
3. Maintains stability in the release branches
4. Simplifies the process of managing multiple versions
5. Eliminates the need for a custom deployment branch

[YANG-DB]

Hi @seankao-az - thanks for this suggestion !

I would like to offer another approach:

Follow OpenSearch release cadence including the versioning semantics - so we are aligned with OpenSearch project.

Why is this good:

1. We will align all the artifacts that have a dependency with opensearch in an identical versioning schema that will remove versioning matching compatibility issues and potential errors.

2. We will have a known schedule of release cadence which will converge our artifact with the global project's artifacts

3. We will have a direct feature compatibility alignment for PPL language support both on Spark And OpenSearch where we can maintain same language features supported for the same version
   A new PPL feature will be presented both on SQL & Spark repositories in the same version and released in the same time

4. By aligning with the project's processes and delivery mechanism we benefit the publication and awareness of the community of our versions and take part in the docker-hub released artifacts

All the benefits mentioned above are kept

• Allows for new feature introduction between minor releases
• Maintains stability in the release branches
• Simplifies the process of managing multiple versions
• Eliminates the need for a custom deployment branch in this approach:

Why is this not so good:

The only negative implication IMO is that we will need to release in a time interval that is not always aligned with our work. Fortunately, this is the case for many other opensearch projects and by adapting the time frame of the general project we will be able to adopt its processes and schedules and enjoy its benefits

---

Branch naming strategy

In a similar way to OpenSearch supporting 3 branches :

• main: Development for the next release
• 2.x : The next minor release
• 2.18 : The current stable release

[seankao-az]

I think it's a good thing we don't tie with the OpenSearch versions. We only upgrade our dependency for OpenSearch when there's new feature or fixes when we need it.
And I think one of the motivation for separating the aysnc-core library from the sql repo is so that its release schedule can be independent of the OS release schedule

[ykmr1224]

+1 to separating versioning from OpenSearch version. That force us to make release every time when OpenSearch release new version.
I am wondering if we have strong motivation to have separate branch for next major release and minor release for now. I think that is needed once we have plan for next major release (like some feature planned only for next major release). Having separate branch would increase the maintenance cost.

[andy-k-improving]

I think it's a good thing we don't tie with the OpenSearch versions. We only upgrade our dependency for OpenSearch when there's new feature or fixes when we need it. And I think one of the motivation for separating the aysnc-core library from the sql repo is so that its release schedule can be independent of the OS release schedule

I share the same thought, especially this Flink project doesn't exist in the form of OpenSearch plugin, like what https://github.com/opensearch-project/sql do, I don't see the necessity for the coupling in release version.

[seankao-az]

Having separate branch would increase the maintenance cost.

I think that is needed once we have plan for next major release

I think this makes sense. For deployment, is there any concern just using the main branch?

[penghuo]

Separating main and 0.x makes sense to me. One use case is to align with Spark major version upgrades. For example, main can track Spark 4.0, while 0.x continues to track Spark 3.x.

[ykmr1224]

@penghuo
That use case makes sense to me. Do we want to start tracking Spark 4.0 now?

[ykmr1224]

I think the main discussion point is actually how we can maintain continuous deployment, which means keep the deployable branch.

The proposed approach to create 0.x branch and backport only stable changes from main branch has following cons.

• In case we make series of PRs for single feature and continuously merge to main branch, we need to backport all the changes after all the PRs were merged. As different commit can be merged in between each PRs, it causes conflicts.
• It would be hard to keep track of what is backported or not, since it indicates we delay backport until we make the feature stable.

Instead of having unstable main branch and stable 0.x branch, should we rather avoid merging unstable changes to main branch? I think we want to follow GitHub-Flow (ref) if we need collaboration or multiple PRs to complete single feature.

[Swiddis]

We had a discussion about this in an internal meeting, here's the notes:

As @ykmr1224 mentioned above: Long-term we're hoping to move Spark and other projects to a fully continuous deployment model, so we would like if we could develop directly on a main-like branch. My understanding is that we hope to avoid needing to do as much backport maintenance between branches. There's two concerns with that approach:

1. When we start work on 1.x and later 2.x, how do we want to support developing breaking changes without breaking our CD branch?
2. How can we guarantee that main will be deployment-ready at all times?

For (1), the tendency is toward this proposal as written. The benefit is that has precedent with other projects in the org. It will also introduce a lot of the operational backporting overhead that we've seen our other projects.

For (2), I'm not sure what mechanism we can use to ensure 0.x is safe that wouldn't also be able to detect that main is safe. Some sort of bake time? (Edit: or merge queues.)

---

One alternative that I'd like to see discussed more is, instead of tracking the "release" branch separately, tracking the "breaking changes" branch separately. i.e. push these changes to 1.0 instead of main. Instead of the usual backport-every-PR strategy, we can then merge 1.0 into main at our convenience, where we already know main is release-ready. This exchanges the work of hundreds of backports with the work of a few larger merge conflicts. The work can be reduced by doing smaller merges over time.

[ykmr1224]

The idea @Swiddis mentioned above makes sense, but I am concerned that it could cause confusion as we take different approach in sql repository.
Based on the discussion, we see requirement to separate breaking changes from stable (deployable) branch. In that case, I prefer having 0.x branch same as sql repository.

If you agree with having 0.x branch, please react with 👍 or reply +1.

[Swiddis]

Also going to leave this comment for +1s for a 1.0 branch, voting with/against the above

[ykmr1224]

Closing as there is no major objection and already got three 👍.
Conclusion: Create 0.x branch, and maintain it as stable branch. Stable commits needs to be backported to 0.x branch using backport 0.x tag.

I'll setup the tag and automation.