0.44.4

This patch release backports some important fixes that have been introduced in more recent versions
of ZenML to the 0.44.x release line.

UPDATE: a breaking update to one of our dependencies means that you should run the following update after installing this version of zenml:

pip install 'bcrypt==4.0.1'

What's Changed

• [backport/0.44] Backport important patches by @stefannica in #2042

Full Changelog: 0.44.3...0.44.4

0.46.1 [YANKED]

THIS RELEASE HAS BEEN YANKED

• Due to an unforeseen alembic migration issue and missing dashboard, this release has been yanked
• 0.47.0 is out to fix the bugs that have been uncovered

0.46.1

The 0.46.1 release introduces support for Service Accounts and API Keys that
can be used to authenticate with the ZenML server from environments that do not
support the web login flow, such as CI/CD environments, for example.

Also included in this release are some documentation updates and bug fixes,
notably moving the database migration logic deployed with the Helm chart out of
the init containers and into a Kubernetes Job, which makes it possible to scale
out the ZenML server deployments without the risk of running into database
migration conflicts.

What's Changed

• Small improvements to Hub docs page by @strickvl in #2015
• Pin OpenAI integration to <1.0.0 by @strickvl in #2027
• Make error message nicer for when two artifacts that share a prefix are found by @strickvl in #2023
• Move db-migration to job instead of init-container to allow replicas by @safoinme in #2021
• Fix stuck/broken CI by @strickvl in #2032
• Increase step.source_code Cut-Off Limit by @fa9r in #2025
• Improve artifact linkage logging in MCP by @avishniakov in #2016
• Upgrade feast so apidocs don't fail no mo by @AlexejPenner in #2028
• Remove NumPy Visualizations for 2D Arrays by @fa9r in #2033
• Fix user activation bug by @stefannica in #2037
• Remove create_new_model_version arg of ModelConfig by @avishniakov in #2030
• Extend the wait period in between PyPi package publication and Docker image building for releases by @strickvl in #2029
• Make zenml up prefill username when launching dashboard by @strickvl in #2024
• Add warning when artifact store cannot be loaded by @strickvl in #2011
• Add extra config to Kaniko docs by @safoinme in #2019
• ZenML API Keys and Service Accounts by @stefannica in #1840

Full Changelog: 0.46.0...0.46.1

0.46.0

UPDATE: a breaking update to one of our dependencies means that you should run the following update after installing this version of zenml:

pip install 'bcrypt==4.0.1'

This release brings some upgrades, documentation updates and bug fixes. Notably,
our langchain integration now supports more modern versions and has been
upgraded to a new version at the lower edge of supported packages on account of
a security vulnerability.

Other fixes related to the Model Control Plane which was updated to support the
deletion of model versions via the CLI, for example.

Breaking Change

We removed the llama_index integration in this release. This related to
unsolvable dependency clashes that relate to sqlmodel and our database. We
expect these clashes to be resolved in the future and then we will add our
integration back in. If you were using the llama_index materializer that was
part of the integration, you will have to use a custom materializer in the
meanwhile. We apologize for the inconvenience.

What's Changed

• MCP-driven E2E template by @avishniakov in #2004
• Model scoped endpoints by @avishniakov in #2003
• Delete model version in cli by @avishniakov in #2006
• Add latest version to model list response by @avishniakov in #2007
• Fix gcs bucket docs error message by @safoinme in #2018
• Fix Skypilot docs configuration by @safoinme in #2017
• Bump langchain, disable llama_index, and fix Vector Store materializer by @strickvl in #2013
• Fix Build Options of GCPImageBuilder by @fa9r in #1992
• Fix the stack component describe CLI output by @stefannica in #2001

Full Changelog: 0.45.6...0.46.0

0.45.6

UPDATE: a breaking update to one of our dependencies means that you should run the following update after installing this version of zenml:

pip install 'bcrypt==4.0.1'

This release brings an array of enhancements and refinements. Notable improvements include
allowing for disconnecting service connectors from stack components, adding connector support to the
sagemaker step operator, turning synchronous mode on by default for all orchestrators, and enabling
server-side component config validation.

What's Changed

• Updating README.md and update images by @znegrin in #1986
• Always set the active workspace to be the default workspace server side by @stefannica in #1989
• Update outdated CLI docs by @strickvl in #1990
• Turn synchronous mode on by default for all orchestrators by @stefannica in #1991
• Use docker credentials in the skypilot orchestrator by @stefannica in #1983
• Add missing space to @step warning message by @strickvl in #1994
• Fix sagemaker orchestrator and step operator env vars and other minor bugs by @stefannica in #1993
• fix: BasePyTorchMaterliazer -> Materializer by @cameronraysmith in #1969
• allow calling old base pytorch materilizzer by @safoinme in #1997
• Add connector support to sagemaker step operator. by @stefannica in #1996
• Server-Side Component Config Validation by @fa9r in #1988
• Allow disconnecting service-connector from stack component by @safoinme in #1864

New Contributors

• @znegrin made their first contribution in #1986

Full Changelog: 0.45.5...0.45.6

0.45.5

UPDATE: a breaking update to one of our dependencies means that you should run the following update after installing this version of zenml:

pip install 'bcrypt==4.0.1'

This minor release contains bugfixes and documentation improvements. Notably,
our sqlmodel dependency has been pinned to 0.0.8 which fixes installation
errors following the release of 0.0.9.

What's Changed

• Add a 'how do I...' section into docs by @strickvl in #1953
• Bump mypy, ruff and black by @strickvl in #1963
• Fix double slashes in weblogin by @schustmi in #1972
• SQLModel docs backport fixes by @strickvl in #1975
• Updated quickstart command in cloud quickstart by @AlexejPenner in #1977
• Make sure vertex job id is only lower case letter, number or dash by @AlexejPenner in #1978
• Fix DB initialization when using external authentication by @schustmi in #1965
• Pin SQLModel dependency to 0.0.8 by @strickvl in #1973
• Source code repository type not included in resolve source by @safoinme in #1982
• Prepare release 0.45.5 by @strickvl in #1981

Full Changelog: 0.45.4...0.45.5

0.45.4

UPDATE: a breaking update to two of our dependencies means that you should run the following update after installing this version of zenml:

pip install 'bcrypt==4.0.1' 'sqlmodel==0.0.8'

This minor update fixes a database migration bug that you could potentially encounter while upgrading your ZenML version and relates to use of the ExternalArtifact object.
If you are upgrading from <0.45.x version, this is the recommended release.

UPDATE (Oct 24, 2023): A new version of sqlmodel (0.0.9) might cause issues with your ZenML installation. If you encounter this error, please do a pip install 'sqlmodel==0.0.8' to fix the problem. This will be permanently fixed in higher versions of ZenML.

PROBLEMS?: If you upgraded to ZenML v0.45.2 or v0.45.3 and are experiencing issues with your database, please consider upgrading to v0.45.4 instead.

What's Changed

• Increase reuse of ModelConfig by @avishniakov in #1954
• resolve alembic branches by @avishniakov in #1964
• Fix corrupted migration for old dbs by @avishniakov in #1966

Full Changelog: 0.45.3...0.45.4

0.45.3

UPDATE: a breaking update to two of our dependencies means that you should run the following update after installing this version of zenml:

pip install 'bcrypt==4.0.1' 'sqlmodel==0.0.8'

This minor update fixes a database migration bug that you could potentially encounter while upgrading your ZenML version and relates to use of the ExternalArtifact object.

PROBLEMS?: If you upgraded to ZenML v0.45.2 and are experiencing issues with your database, please either reach out to us on Slack directly or feel free to use this migration script that will manually fix the issue. (Please do backup your database before using the migration script so as to prevent any data loss!)

This release also includes a bugfix from @cameronraysmith relating to the
resolution of our Helm chart OCI location. Thank you!

What's Changed

• fix: match chart name in docs to publish workflow by @cameronraysmith in #1942
• Evaluate YAML based config early + OSS-2511 by @avishniakov in #1876
• Fixing nullable parameter to avoid extra migrations by @bcdurak in #1955
• Pin Helm version to avoid 400 Bad Request error by @wjayesh in #1958
• external_input_artifact backward compatibility with alembic by @avishniakov in #1957
• Fix nlp template link by @htahir1 in #1960
• handle NULLs in migration by @avishniakov in #1961
• Prepare release 0.45.3 by @strickvl in #1959

New Contributors

• @cameronraysmith made their first contribution in #1942

Full Changelog: 0.45.2...0.45.3

0.45.2

UPDATE: a breaking update to two of our dependencies means that you should run the following update after installing this version of zenml:

pip install 'bcrypt==4.0.1' 'sqlmodel==0.0.8'

UPDATE NOTE: 0.45.2 introduced a minor database migration bug. To avoid this, please install 0.45.3 directly. If you already installed 0.45.2 and are running into database migration issues (especially if you're using ExternalArtifact objects), please contact us directly on Slack or (alternatively) use this migration script to fix your broken database (script should be executed directly on the server if you use remote installation).

This release replaces 0.45.0 and 0.45.1, and fixes the major migration bugs that were in
that yanked release. Please upgrade directly to 0.45.2 and avoid upgrading to
0.45.0 to avoid unexpected migration issues.

Note that 0.45.0 and 0.45.1 were removed from PyPI due to an issue with the
alembic versions + migration which could affect the database state. This release
fixes that issue.

If you have already upgraded to 0.45.0 please let us know in Slack and we'll happy to assist in rollback and recovery.

This release introduces a major upgrade to ZenML, featuring a new authentication mechanism, performance improvements, the introduction of the model control plane, and internal enhancements.

New Authentication Mechanism (#4303)

Our improved authentication mechanism offers a more secure way of connecting to the ZenML server. It initiates a device flow that prompts you to log in via the browser dashboard:

zenml connect --url <YOUR_SERVER_URL>

This eliminates the need for explicit credential input. The previous method (zenml connect --url <URL> --username <USERNAME> --password <PASSWORD>) remains operational but is less recommended due to security concerns.

Critical This change disrupts existing pipeline schedules. After upgrading, manually cancel and reschedule pipelines using the updated version of ZenML.

For more information, read about the device flow in our documentation.

Performance enhancements (#3207)

Internal API adjustments have reduced the footprint of ZenML API objects by up to 35%. This will particularly benefit users with large step and pipeline configurations. Further reductions will be implemented in our next release.

Model Control Plane debut (#5648)

ZenML now includes a preliminary version of the model control plane, a feature for registering models and their metadata on a single ZenML dashboard view. Future releases will provide more details. To test this early version, follow this example.

Breaking Changes

• Environment variables ZENML_AUTH_TYPE and ZENML_JWT_SECRET_KEY have been renamed to ZENML_SERVER_AUTH_SCHEME and ZENML_SERVER_JWT_SECRET_KEY, respectively.
• All ZenML server-issued JWT tokens now include an issuer and an audience. After the server update, current scheduled pipelines become invalidated. Reset your schedules and reconnect all clients to the server to obtain new tokens.
• UnmaterializedArtifact has been relocated to zenml.artifacts. Change your import statement from from zenml.materializers import UnmaterializedArtifact to from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact.

Deprecations

• zenml.steps.external_artifact.ExternalArtifact has moved to zenml.artifacts.external_artifact.ExternalArtifact.

And the rest:

• Discord alerter integration by @bhatt-priyadutt in #1818. Huge shoutout to you priyadutt - we're sending some swag your way!
• Update Neptune dependency: neptune-client > neptune by @fa9r in #1837
• Disable codeql on pushes to develop by @strickvl in #1842
• Template not updating due to git diff misuse by @avishniakov in #1844
• Bump feast version to fix api docs generation by @fa9r in #1845
• CI Fixes / Improvements by @fa9r in #1848
• Fix MLflow registry methods with empty metadata by @fa9r in #1843
• Use configured template REF in CI by @avishniakov in #1851
• Fix template REF in CI by @avishniakov in #1852
• Fix AWS service connector installation requirements by @stefannica in #1850
• [Docs] Improvements to custom flavor and custom orchestrator pages by @htahir1 in #1747
• Optimizing the performance through database changes by @bcdurak in #1835
• Add README for examples folder by @strickvl in #1860
• Free up disk space in CI by @strickvl in #1863
• Make Terraform Optional Again by @fa9r in #1855
• Model watchtower becomes Model control plane by @strickvl in #1868
• Update documentation by @VishalKumar-S in #1872
• Fix CI by freeing up space on runner by @strickvl in #1866
• Allow for user param to be specified (successfully) in DockerSettings by @strickvl in #1857
• Add get_pipeline_context by @avishniakov in #1870
• [Helm] Use GCP creds directly instead of a file. by @wjayesh in #1874
• External authenticator support, authorized devices and web login by @stefannica in #1814
• Connect to Service-connector at component registration by @safoinme in #1858
• Fixing the upgrade migration script after the database changes by @bcdurak in #1877
• [Model Control Plane] v0.1 mega-branch by @avishniakov in #1816
• Update to templates by @htahir1 in #1878
• Docs for orgs, rbac and sso by @AlexejPenner in #1875
• Convert network_config dict to NetworkConfig object in SageMaker orchestrator by @christianversloot in #1873
• Add missing Docker build options for GCP image builder by @strickvl in #1856
• Solve alembic branching issue by @avishniakov in #1879
• Fix typo for 0.45 release by @strickvl in #1881
• Only import ipinfo when necessary by @schustmi in #1888
• [Model Control Plane] Suppress excessive logging in model control plane by @avishniakov in #1885
• Add warning generation scripts for Gitbook docs by @strickvl in #1929
• Fix calling click decorator in model CLI command by @safoinme in #1932
• Lightweight template CI by @avishniakov in #1930
• Update Skypilot orchestrator setting docs section by @safoinme in #1931

New Contributors

• @VishalKumar-S made their first contribution in #1872

Full Changelog: 0.44.3...0.45.0

0.45.1 [YANKED]

0.45.1

Update: It's one of those days! This release is also yanked due to alembic migration issues. We are working hard to solve it. In the meanwhile please use 0.44.3

This release replaces 0.45.0, and fixes the major migration bugs that were in that yanked release. Please upgrade directly to 0.45.1 and avoid upgrading to 0.45.0 to avoid unexpected migration issues.

What's changed

This release introduces a major upgrade to ZenML, featuring a new authentication mechanism, performance improvements, the introduction of the model control plane, and internal enhancements.

New Authentication Mechanism (#4303)

Our improved authentication mechanism offers a more secure way of connecting to the ZenML server. It initiates a device flow that prompts you to log in via the browser dashboard:

zenml connect --url <YOUR_SERVER_URL>

This eliminates the need for explicit credential input. The previous method (zenml connect --url <URL> --username <USERNAME> --password <PASSWORD>) remains operational but is less recommended due to security concerns.

Critical This change disrupts existing pipeline schedules. After upgrading, manually cancel and reschedule pipelines using the updated version of ZenML.

For more information, read about the device flow in our documentation.

Performance enhancements (#3207)

Internal API adjustments have reduced the footprint of ZenML API objects by up to 35%. This will particularly benefit users with large step and pipeline configurations. Further reductions will be implemented in our next release.

Model Control Plane debut (#5648)

ZenML now includes a preliminary version of the model control plane, a feature for registering models and their metadata on a single ZenML dashboard view. Future releases will provide more details. To test this early version, follow this example.

Breaking Changes

• Environment variables ZENML_AUTH_TYPE and ZENML_JWT_SECRET_KEY have been renamed to ZENML_SERVER_AUTH_SCHEME and ZENML_SERVER_JWT_SECRET_KEY, respectively.
• All ZenML server-issued JWT tokens now include an issuer and an audience. After the server update, current scheduled pipelines become invalidated. Reset your schedules and reconnect all clients to the server to obtain new tokens.
• UnmaterializedArtifact has been relocated to zenml.artifacts. Change your import statement from from zenml.materializers import UnmaterializedArtifact to from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact.

Deprecations

• zenml.steps.external_artifact.ExternalArtifact has moved to zenml.artifacts.external_artifact.ExternalArtifact.

And the rest:

• Discord alerter integration by @bhatt-priyadutt in #1818. Huge shoutout to you priyadutt - we're sending some swag your way!
• Update Neptune dependency: neptune-client > neptune by @fa9r in #1837
• Disable codeql on pushes to develop by @strickvl in #1842
• Template not updating due to git diff misuse by @avishniakov in #1844
• Bump feast version to fix api docs generation by @fa9r in #1845
• CI Fixes / Improvements by @fa9r in #1848
• Fix MLflow registry methods with empty metadata by @fa9r in #1843
• Use configured template REF in CI by @avishniakov in #1851
• Fix template REF in CI by @avishniakov in #1852
• Fix AWS service connector installation requirements by @stefannica in #1850
• [Docs] Improvements to custom flavor and custom orchestrator pages by @htahir1 in #1747
• Optimizing the performance through database changes by @bcdurak in #1835
• Add README for examples folder by @strickvl in #1860
• Free up disk space in CI by @strickvl in #1863
• Make Terraform Optional Again by @fa9r in #1855
• Model watchtower becomes Model control plane by @strickvl in #1868
• Update documentation by @VishalKumar-S in #1872
• Fix CI by freeing up space on runner by @strickvl in #1866
• Allow for user param to be specified (successfully) in DockerSettings by @strickvl in #1857
• Add get_pipeline_context by @avishniakov in #1870
• [Helm] Use GCP creds directly instead of a file. by @wjayesh in #1874
• External authenticator support, authorized devices and web login by @stefannica in #1814
• Connect to Service-connector at component registration by @safoinme in #1858
• Fixing the upgrade migration script after the database changes by @bcdurak in #1877
• [Model Control Plane] v0.1 mega-branch by @avishniakov in #1816
• Update to templates by @htahir1 in #1878
• Docs for orgs, rbac and sso by @AlexejPenner in #1875
• Convert network_config dict to NetworkConfig object in SageMaker orchestrator by @christianversloot in #1873
• Add missing Docker build options for GCP image builder by @strickvl in #1856
• Solve alembic branching issue by @avishniakov in #1879
• Fix typo for 0.45 release by @strickvl in #1881
• Only import ipinfo when necessary by @schustmi in #1888
• [Model Control Plane] Suppress excessive logging in model control plane by @avishniakov in #1885
• Add warning generation scripts for Gitbook docs by @strickvl in #1929
• Fix calling click decorator in model CLI command by @safoinme in #1932
• Lightweight template CI by @avishniakov in #1930
• Update Skypilot orchestrator setting docs section by @safoinme in #1931
• Add missing space when creating new model version by @strickvl in #1935
• Fix alembic inconsistencies by @fa9r in #1944
• [Model Control Plane] link cached artifacts by @avishniakov in #1946
• Fixing the migration issues by @bcdurak in #1945
• Release prepped by @htahir1 in #1948

New Contributors

• @VishalKumar-S made their first contribution in #1872

Full Changelog: 0.44.3...0.45.0

0.45.0 [YANKED]

IMPORTANT THIS RELEASE HAS BEEN YANKED

• Due to an unforseen alembic migration issue, this release has been yanked.0.45.1 is on its way to being released to fix the bugs that have been uncovered
• If you have already upgraded to 0.45.0 please let us know in Slack and we'll happy to assist in rollback and recovery