From 8681ab07c6e0c48d55e6a17183f39abd30b3182a Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 10 Dec 2025 21:42:17 +0530 Subject: [PATCH 1/8] spacing --- docs/docs/version-policy.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/docs/version-policy.md b/docs/docs/version-policy.md index 79c9402d5..fd7a151f0 100644 --- a/docs/docs/version-policy.md +++ b/docs/docs/version-policy.md @@ -1,13 +1,11 @@ # Icechunk Version Policy - There are three versions referred to in this document: 1. Icechunk Spec Version 2. `icechunk` Python Library Version 3. `icechunk` Rust Crate Version - ## Icechunk Spec The spec will advance with any revisions as defined in the [Spec document](./spec.md). From e91335e6bfea9af69140b9ccaa793a846655bed9 Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 10 Dec 2025 21:49:47 +0530 Subject: [PATCH 2/8] various clarifications --- docs/docs/version-policy.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/docs/version-policy.md b/docs/docs/version-policy.md index fd7a151f0..63cd17b0f 100644 --- a/docs/docs/version-policy.md +++ b/docs/docs/version-policy.md @@ -8,21 +8,22 @@ There are three versions referred to in this document: ## Icechunk Spec -The spec will advance with any revisions as defined in the [Spec document](./spec.md). +The Icechunk format specification versions are identified by a single integer. The spec version will increment with any revisions as defined in the [Spec document](./spec.md). ## `icechunk` Python Library -Versions of this library are identified by a triplet of integers with the form `..` +Versions of this library are identified by a triplet of integers with the form `..`. ### Library Version #### Major Versions -The `icechunk` Python library will change when there is an on-disk incompatible change in the Icechunk Spec. +The `icechunk` Python library major version will increment when and only when there is an on-disk incompatible change in the Icechunk Spec. ##### Reading -The latest release of `icechunk` Python will always be able to read data written against any version of the Icechunk Spec. +The latest release of `icechunk` Python will always be able to read data written against any previous version of the Icechunk Spec. +It in general will not be able to read data written using a future version of the Icechunk Spec. ##### Writing @@ -31,7 +32,7 @@ The latest release of `icechunk` Python will always be able to read data written ###### Forward Compatibility -`icechunk` Python will **not** implement forward compatible reading and writing. `icechunk` Python version 1.x.x will not be able to read or write the Icechunk 2 format. +`icechunk` Python will **not** implement forward compatible reading and writing. For example `icechunk` Python version 1.x.x will not be able to read or write data using the Icechunk format 2 (or later). #### Minor Versions From 98488d96340d482a51494fe3cf1efb6b2487a58d Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 10 Dec 2025 21:51:36 +0530 Subject: [PATCH 3/8] remove an unecessary level of indentation --- docs/docs/version-policy.md | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/docs/docs/version-policy.md b/docs/docs/version-policy.md index 63cd17b0f..65ce3050b 100644 --- a/docs/docs/version-policy.md +++ b/docs/docs/version-policy.md @@ -14,35 +14,30 @@ The Icechunk format specification versions are identified by a single integer. T Versions of this library are identified by a triplet of integers with the form `..`. -### Library Version - -#### Major Versions +### Major Versions The `icechunk` Python library major version will increment when and only when there is an on-disk incompatible change in the Icechunk Spec. -##### Reading +#### Reading The latest release of `icechunk` Python will always be able to read data written against any previous version of the Icechunk Spec. -It in general will not be able to read data written using a future version of the Icechunk Spec. -##### Writing +#### Writing `icechunk` Python will be able to, at minimum, write to the last major version of the Icechunk Spec. This means that `icechunk` Python 2 can write to Icechunk Spec version 1 format. `icechunk` Python 3 will be able write to Icechunk Spec 2, but not Icechunk Spec 1. - -###### Forward Compatibility +##### Forward Compatibility `icechunk` Python will **not** implement forward compatible reading and writing. For example `icechunk` Python version 1.x.x will not be able to read or write data using the Icechunk format 2 (or later). -#### Minor Versions +## Minor Versions Minor releases will require at most minor effort from users to update their code. Minor versions will increment with new features, significant bug fixes, or for small changes in python library api compatibility. -#### Patch Versions +## Patch Versions Patch releases will require no effort on the part of users of `icechunk` Python. They will contain bugfixes or documentation improvements. - ## `icechunk` Rust Crate No explicit versioning scheme followed here. From 233ea82019fee3c4c9dd8958508a87f8f2d5a7b1 Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 10 Dec 2025 22:00:43 +0530 Subject: [PATCH 4/8] more clarifications --- docs/docs/version-policy.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/docs/version-policy.md b/docs/docs/version-policy.md index 65ce3050b..c6693ecb3 100644 --- a/docs/docs/version-policy.md +++ b/docs/docs/version-policy.md @@ -8,15 +8,18 @@ There are three versions referred to in this document: ## Icechunk Spec -The Icechunk format specification versions are identified by a single integer. The spec version will increment with any revisions as defined in the [Spec document](./spec.md). +The Icechunk format specification ("icechunk spec") versions are identified by a single integer. +The spec version will increment if and only if there is an on-disk incompatible change in the Icechunk Spec, as defined in the [spec document](./spec.md). ## `icechunk` Python Library Versions of this library are identified by a triplet of integers with the form `..`. +We use a variant of the [SemVer](https://semver.org/) versioning scheme, but which also takes into account compatibility with the on-disk Icechunk format specification. + ### Major Versions -The `icechunk` Python library major version will increment when and only when there is an on-disk incompatible change in the Icechunk Spec. +The `icechunk` Python library major version will increment if and only if there is an on-disk incompatible change in the Icechunk Spec. #### Reading @@ -24,7 +27,9 @@ The latest release of `icechunk` Python will always be able to read data written #### Writing -`icechunk` Python will be able to, at minimum, write to the last major version of the Icechunk Spec. This means that `icechunk` Python 2 can write to Icechunk Spec version 1 format. `icechunk` Python 3 will be able write to Icechunk Spec 2, but not Icechunk Spec 1. +`icechunk` Python will be able to, at minimum, write to the last major version of the Icechunk Spec. + +For example, this means that `icechunk` Python version 2 can write to Icechunk Spec version 1 format, but while `icechunk` Python 3 will be able write to Icechunk Spec 2, is it not guaranteed to be able to write to but not Icechunk Spec 1. ##### Forward Compatibility From eed5ca26de442f9c74d49ce07cc2f4621f92acd9 Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 10 Dec 2025 22:50:10 +0530 Subject: [PATCH 5/8] rewrite based on zarr version policy --- docs/docs/version-policy.md | 76 +++++++++++++++++++++++++++---------- 1 file changed, 55 insertions(+), 21 deletions(-) diff --git a/docs/docs/version-policy.md b/docs/docs/version-policy.md index c6693ecb3..be94e812b 100644 --- a/docs/docs/version-policy.md +++ b/docs/docs/version-policy.md @@ -3,46 +3,80 @@ There are three versions referred to in this document: 1. Icechunk Spec Version -2. `icechunk` Python Library Version -3. `icechunk` Rust Crate Version +2. Icechunk Python Library Version +3. Icechunk Rust Crate Version ## Icechunk Spec The Icechunk format specification ("icechunk spec") versions are identified by a single integer. -The spec version will increment if and only if there is an on-disk incompatible change in the Icechunk Spec, as defined in the [spec document](./spec.md). -## `icechunk` Python Library +The spec version will increment if and only if there is an on-disk incompatible change in the Icechunk Spec, as defined by the [spec document](./spec.md). -Versions of this library are identified by a triplet of integers with the form `..`. +## Icechunk Python Library -We use a variant of the [SemVer](https://semver.org/) versioning scheme, but which also takes into account compatibility with the on-disk Icechunk format specification. +Versions of this library are identified by a triplet of integers with the form `..`, for example `1.1.12`. +A release of `icechunk-python` is associated with a new version identifier. +That new identifier is generated by incrementing exactly one of the components of the previous version identifier by 1. +When incrementing the `major` component of the version identifier, the `minor` and `patch` components is reset to 0. +When incrementing the minor component, the patch component is reset to 0. -### Major Versions +Releases are classified by the library changes contained in that release. +This classification determines which component of the version identifier is incremented on release. -The `icechunk` Python library major version will increment if and only if there is an on-disk incompatible change in the Icechunk Spec. +* **major** releases (for example, `1.1.12` -> `2.0.0`) are for changes that will require extensive adaptation efforts from many users and downstream projects. +For example, breaking changes to widely-used user-facing APIs should only be applied in a major release. -#### Reading + Users and downstream projects should carefully consider the impact of a major release before adopting it. + In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. -The latest release of `icechunk` Python will always be able to read data written against any previous version of the Icechunk Spec. + IMPORTANT: Any backwards-incompatible change to the Icechunk format specification is always treated as a significant change for users, and so will therefore always be accompanied by a major release. See the [section](#data-format-specification-compatibility) on data format specification compatibility below. -#### Writing +* **minor** releases (for example, `3.0.0` -> `3.1.0`) are for changes that do not require significant effort from most users or downstream downstream projects to respond to. API changes are possible in minor releases if the burden on users imposed by those changes is sufficiently small. -`icechunk` Python will be able to, at minimum, write to the last major version of the Icechunk Spec. + For example, a recently released API may need fixes or refinements that are breaking, but low impact due to the recency of the feature. Such API changes are permitted in a minor release. -For example, this means that `icechunk` Python version 2 can write to Icechunk Spec version 1 format, but while `icechunk` Python 3 will be able write to Icechunk Spec 2, is it not guaranteed to be able to write to but not Icechunk Spec 1. + Minor releases are safe for most users and downstream projects to adopt. -##### Forward Compatibility +* **patch** releases (for example, `3.1.0` -> `3.1.1`) are for changes that contain no breaking or behaviour changes for downstream projects or users. Examples of changes suitable for a patch release are bugfixes and documentation improvements. -`icechunk` Python will **not** implement forward compatible reading and writing. For example `icechunk` Python version 1.x.x will not be able to read or write data using the Icechunk format 2 (or later). + Users should always feel safe upgrading to a the latest patch release. + +Note that this versioning scheme is not consistent with [Semantic Versioning](https://semver.org/). +Contrary to SemVer, the Icechunk library may release breaking changes in `minor` releases, or even `patch` releases under exceptional circumstances. +But we strive to avoid doing so. + +A better model for our versioning scheme is [Intended Effort Versioning](https://jacobtomlinson.dev/effver/), or "EffVer". +The guiding principle off EffVer is to categorize releases based on the *expected effort required to upgrade to that release*. + +Icechunk developers endeavour to make changes as smooth as possible for users. +This means making backwards-compatible changes wherever possible. +When a backwards-incompatible change is necessary, users will be notified well in advance, e.g. via informative deprecation warnings. + +## Data format specification compatibility + +The Icechunk library is an implementation of a file format standard defined separately -- see the [Icechunk specification](./spec.md). + +If an existing Icechunk format version changes, or a new version of the Icechunk format is released, then the Icechunk library will generally require changes. +We choose to treat these changes as significant to justify a new major version of the icechunk python library. -## Minor Versions +**Therefore any update to the Icechunk format specification version will be accompanied by a _major_ release of the icechunk python library.** -Minor releases will require at most minor effort from users to update their code. Minor versions will increment with new features, significant bug fixes, or for small changes in python library api compatibility. +For example, when the new [Icechunk version 2](https://earthmover.io/blog/evolving-our-tensor-storage-engine-a-preview-of-icechunk-2) format specification is first supported, it will be via a new major icechunk-python release, with a version updated from `1.x.y` to `2.0.0`. -## Patch Versions +### Reading -Patch releases will require no effort on the part of users of `icechunk` Python. They will contain bugfixes or documentation improvements. +The latest release of `icechunk` Python will make best efforts to always be able to read data written against any previous version of the Icechunk Spec. + +### Writing + +`icechunk` Python will be able to, at minimum, write to the last major version of the Icechunk Spec. + +For example, this means that `icechunk` Python version 2 can write to Icechunk Spec version 1 format, but while `icechunk` Python 3 will be able write to Icechunk Spec version 2, is it not guaranteed to be able to write to Icechunk Spec version 1. + +### Forward Compatibility + +`icechunk` Python will **not** implement forward compatible reading and writing. For example `icechunk` Python version 1.x.x will not be able to read or write data using the Icechunk format 2 (or later). -## `icechunk` Rust Crate +## Icechunk Rust Crate -No explicit versioning scheme followed here. +No explicit versioning scheme is followed by the icechunk rust crate. From b3dec2d3d79661b86ceb22368d0eba89857fd8cb Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 10 Dec 2025 23:07:14 +0530 Subject: [PATCH 6/8] use proper markdown formatting for admonition --- docs/docs/version-policy.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/docs/version-policy.md b/docs/docs/version-policy.md index be94e812b..8307bed51 100644 --- a/docs/docs/version-policy.md +++ b/docs/docs/version-policy.md @@ -29,7 +29,13 @@ For example, breaking changes to widely-used user-facing APIs should only be app Users and downstream projects should carefully consider the impact of a major release before adopting it. In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. - IMPORTANT: Any backwards-incompatible change to the Icechunk format specification is always treated as a significant change for users, and so will therefore always be accompanied by a major release. See the [section](#data-format-specification-compatibility) on data format specification compatibility below. + :::info + + Any backwards-incompatible change to the Icechunk format specification is always treated as a significant change for users, and so will therefore always be accompanied by a major release. + + For more details see the [section](#data-format-specification-compatibility) on data format specification compatibility below. + + ::: * **minor** releases (for example, `3.0.0` -> `3.1.0`) are for changes that do not require significant effort from most users or downstream downstream projects to respond to. API changes are possible in minor releases if the burden on users imposed by those changes is sufficiently small. From c70d0fa61b964b58b81dc00a75fbeea28a84ca3d Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 10 Dec 2025 23:13:22 +0530 Subject: [PATCH 7/8] lint --- docs/docs/version-policy.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/docs/version-policy.md b/docs/docs/version-policy.md index 8307bed51..edd83f8a9 100644 --- a/docs/docs/version-policy.md +++ b/docs/docs/version-policy.md @@ -8,31 +8,31 @@ There are three versions referred to in this document: ## Icechunk Spec -The Icechunk format specification ("icechunk spec") versions are identified by a single integer. +The Icechunk format specification ("icechunk spec") versions are identified by a single integer. The spec version will increment if and only if there is an on-disk incompatible change in the Icechunk Spec, as defined by the [spec document](./spec.md). ## Icechunk Python Library -Versions of this library are identified by a triplet of integers with the form `..`, for example `1.1.12`. -A release of `icechunk-python` is associated with a new version identifier. -That new identifier is generated by incrementing exactly one of the components of the previous version identifier by 1. -When incrementing the `major` component of the version identifier, the `minor` and `patch` components is reset to 0. +Versions of this library are identified by a triplet of integers with the form `..`, for example `1.1.12`. +A release of `icechunk-python` is associated with a new version identifier. +That new identifier is generated by incrementing exactly one of the components of the previous version identifier by 1. +When incrementing the `major` component of the version identifier, the `minor` and `patch` components is reset to 0. When incrementing the minor component, the patch component is reset to 0. -Releases are classified by the library changes contained in that release. +Releases are classified by the library changes contained in that release. This classification determines which component of the version identifier is incremented on release. -* **major** releases (for example, `1.1.12` -> `2.0.0`) are for changes that will require extensive adaptation efforts from many users and downstream projects. +* **major** releases (for example, `1.1.12` -> `2.0.0`) are for changes that will require extensive adaptation efforts from many users and downstream projects. For example, breaking changes to widely-used user-facing APIs should only be applied in a major release. - Users and downstream projects should carefully consider the impact of a major release before adopting it. + Users and downstream projects should carefully consider the impact of a major release before adopting it. In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. :::info - - Any backwards-incompatible change to the Icechunk format specification is always treated as a significant change for users, and so will therefore always be accompanied by a major release. - + + Any backwards-incompatible change to the Icechunk format specification is always treated as a significant change for users, and so will therefore always be accompanied by a major release. + For more details see the [section](#data-format-specification-compatibility) on data format specification compatibility below. ::: @@ -47,22 +47,22 @@ For example, breaking changes to widely-used user-facing APIs should only be app Users should always feel safe upgrading to a the latest patch release. -Note that this versioning scheme is not consistent with [Semantic Versioning](https://semver.org/). -Contrary to SemVer, the Icechunk library may release breaking changes in `minor` releases, or even `patch` releases under exceptional circumstances. +Note that this versioning scheme is not consistent with [Semantic Versioning](https://semver.org/). +Contrary to SemVer, the Icechunk library may release breaking changes in `minor` releases, or even `patch` releases under exceptional circumstances. But we strive to avoid doing so. -A better model for our versioning scheme is [Intended Effort Versioning](https://jacobtomlinson.dev/effver/), or "EffVer". +A better model for our versioning scheme is [Intended Effort Versioning](https://jacobtomlinson.dev/effver/), or "EffVer". The guiding principle off EffVer is to categorize releases based on the *expected effort required to upgrade to that release*. -Icechunk developers endeavour to make changes as smooth as possible for users. -This means making backwards-compatible changes wherever possible. +Icechunk developers endeavour to make changes as smooth as possible for users. +This means making backwards-compatible changes wherever possible. When a backwards-incompatible change is necessary, users will be notified well in advance, e.g. via informative deprecation warnings. ## Data format specification compatibility The Icechunk library is an implementation of a file format standard defined separately -- see the [Icechunk specification](./spec.md). -If an existing Icechunk format version changes, or a new version of the Icechunk format is released, then the Icechunk library will generally require changes. +If an existing Icechunk format version changes, or a new version of the Icechunk format is released, then the Icechunk library will generally require changes. We choose to treat these changes as significant to justify a new major version of the icechunk python library. **Therefore any update to the Icechunk format specification version will be accompanied by a _major_ release of the icechunk python library.** From a9d437ad40f0e7e33861856d7a314447480390d3 Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 10 Dec 2025 23:25:40 +0530 Subject: [PATCH 8/8] Revert "use proper markdown formatting for admonition" This reverts commit b3dec2d3d79661b86ceb22368d0eba89857fd8cb. --- docs/docs/version-policy.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/docs/docs/version-policy.md b/docs/docs/version-policy.md index edd83f8a9..270a2f142 100644 --- a/docs/docs/version-policy.md +++ b/docs/docs/version-policy.md @@ -29,13 +29,7 @@ For example, breaking changes to widely-used user-facing APIs should only be app Users and downstream projects should carefully consider the impact of a major release before adopting it. In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. - :::info - - Any backwards-incompatible change to the Icechunk format specification is always treated as a significant change for users, and so will therefore always be accompanied by a major release. - - For more details see the [section](#data-format-specification-compatibility) on data format specification compatibility below. - - ::: + IMPORTANT: Any backwards-incompatible change to the Icechunk format specification is always treated as a significant change for users, and so will therefore always be accompanied by a major release. For more details see the [section](#data-format-specification-compatibility) on data format specification compatibility below. * **minor** releases (for example, `3.0.0` -> `3.1.0`) are for changes that do not require significant effort from most users or downstream downstream projects to respond to. API changes are possible in minor releases if the burden on users imposed by those changes is sufficiently small.