0.22.0

This release comes with a new BentoML integration as well as a reworked Airflow orchestrator. Additionally, it greatly improves the server performance as well as other small fixes and updates to our docs!

BentoML integration

The new BentoML integration includes a BentoML model deployer component that allows you to deploy your models from any of the major machine learning frameworks on your local machine.

See example here.

Airflow orchestrator v2

The previous Airflow orchestrator was limited to running locally and had many additional unpleasant constraints that made it hard to work with. This release includes a completely rewritten, new version of the Airflow orchestrator that now relies on Docker images to run your pipelines and works both locally and with remote Airflow deployments.

See what changed in this video and check out the brand-new example here.

Notable bugfixes

• Further improvements to the synchronization that transfers pipeline run information from the MLMD database to the ZenML Server.
• The ZenML Label Studio integration can now be used with non-local (i.e. deployed) instances. For more information see the Label Studiodocs.
• The Spark example is fixed and now works again end-to-end.

Breaking Changes

The following changes introduces with this release mey require some manual
intervention to update your current installations:

• the Airflow orchestrator now requires a newer version of Airflow (run zenml integration install airflow to upgrade) and Docker installed
  to work.

What's Changed

• Fix bug when running non-local annotator instance. by @sheikhomar in #1045
• Introduce Permissions, Link Permissions to Roles, Restrict Access to endpoints based on Permission by @AlexejPenner in #1007
• Fix copy-pasted log message for annotator by @strickvl in #1049
• Add warning message for client server version mismatch by @schustmi in #1047
• Fix path to ingress values in ZenServer recipes by @wjayesh in #1053
• Prevent deletion/update of default entities by @stefannica in #1046
• Fix Publish API docs workflow by @AlexejPenner in #1054
• Fix multiple alembic heads warning by @fa9r in #1051
• Fix Null Step Configuration/Parameters Error by @fa9r in #1050
• Fix role permission migration by @schustmi in #1056
• Made role assignment/revokation possible through zen_server by @AlexejPenner in #1059
• Bugfix/make role assignment work with enum by @AlexejPenner in #1063
• Manually set scoped for each endpoint by @AlexejPenner in #1064
• Add run args to local docker orchestrator settings by @schustmi in #1060
• Docker ZenML deployment improvements and docs by @stefannica in #1061
• Bugfix Mlflow service cleanup configuration by @safoinme in #1067
• Rename DB Tables and Fix Foreign Keys by @fa9r in #1058
• Paginate secrets in AWSSecretsManager by @chiragjn in #1057
• Add explicit dashboard docs by @strickvl in #1052
• Added GA and Gitlab to envs by @htahir1 in #1068
• Add Inference Server Predictor to KServe and Seldon Docs by @safoinme in #1048
• Rename project table to workspace by @fa9r in #1073
• Airflow orchestrator v2 by @schustmi in #1042
• Add get_or_create_run() ZenStore method by @fa9r in #1070
• Fix the flaky fileio tests by @schustmi in #1072
• BentoML Deployer Integration by @safoinme in #1044
• Sync Speedup by @fa9r in #1055
• Fixed broken links in docs and examples. by @dnth in #1076
• Make additional stack component config options available as a setting by @schustmi in #1069
• Rename step_run_artifact table to step_run_input_artifact by @fa9r in #1075
• Update Spark Example to ZenML post 0.20.0 by @safoinme in #1071
• Always set caching to false for all Kubeflow based orchestrators by @schustmi in #1079
• Feature/eng 1402 consolidate stack sharing by @AlexejPenner in #1036

New Contributors

• @sheikhomar made their first contribution in #1045
• @chiragjn made their first contribution in #1057

Full Changelog: 0.21.1...0.22.0

0.21.1

This is an ad-hoc release to fix some bugs introduced the 0.21.0 release that
made the local ZenML dashboard unusable.

What's Changed

• Include latest (not oldest) three runs in HydratedPipelineModel by @schustmi in #1039
• Update docs to use pip install [server] by @strickvl in #1037
• Docs fix for Deepchecks by @strickvl in #1040
• Fix the pipeline run sync on sqlite and the --blocking zenml server deployment by @stefannica in #1041

Full Changelog: 0.21.0...0.21.1

0.21.0

This release primarily fixes a number of bugs that were introduced as part of
the 0.20.0 ZenServer release. These significantly improve the stability when
using ZenML with the ZenML Server.

Notable fixes include:

• Improved the synchronization that transfers pipeline run information from
  the MLMD database to the ZenML Server. This helps fix a number of issues with
  missing steps in the post-execution workflow, model deployment steps and other
  issues.
• The Label Studio example is fixed and now works again end-to-end.
• The ZenML Label Studio integration can now be used with non-local (i.e.
  deployed) instances. For more information see the Label Studiodocs.

New features and other improvements:

• ZenML now uses alembic for
  automated database migrations. The migrations happen automatically after every
  ZenML update.
• New zenml pipeline runs export / import / migrate CLI commands are now
  available to export, import and migrate pipeline runs from older, pre-0.20.0
  versions of ZenML. The ZenML server now also automatically picks up older
  pipeline runs that have been logged in the metadata store by ZenML prior to
  0.20.0.
• An MLMD gRPC service can now be deployed with the ZenML Helm chart to act
  as a proxy between clients, orchestrators and the MySQL database. This
  significantly reduces the time it takes to run pipelines locally.
• You can now specify affinity and tolerations and node selectors to all
  Kubernetes based orchestrators with the new Kubernetes Pod settings feature.

Breaking Changes

The following changes introduces with this release mey require some manual
intervention to update your current installations:

• the zenml server helm chart values.yaml file has been restructured to make
  it easier to configure and to clearly distinguish between the zenml server
  component and the newly introduced gRPC service component. Please update your
  values.yaml copies accordingly.
• the Azure integration dependency versions have been updated. Please run
  zenml integration install azure to update your current installation, if
  you're using Azure.

What's Changed

• Implement automatic alembic migration by @AlexejPenner in #990
• Fix GCP Artifact Store listdir empty path by @safoinme in #998
• Add flavors mini-video to docs by @strickvl in #999
• Remove the Client() warning when used inside a step by @stefannica in #1000
• Fix broken links caused by updated by @AlexejPenner in #1002
• Fix FileNotFoundError with remote path in HuggingFace Dataset materializer by @gabrielmbmb in #995
• Add zenml pipeline runs export / import / migrate CLI commands by @fa9r in #977
• Log message when activating a stack as part of registration by @schustmi in #1005
• Minor fixes in Migration to 0.20.0 documentation by @alvarobartt in #1009
• Doc updates by @htahir1 in #1006
• Fixing broken links in docs by @dnth in #1018
• Label Studio example fix by @strickvl in #1021
• Docs for using CUDA-enabled docker images by @strickvl in #1010
• Add social media heading on docs page by @dnth in #1020
• Add executing custom command for getting requirements by @gabrielmbmb in #1012
• Delay user instruction in dockerfile generation by @schustmi in #1004
• Update link checker configs for faster, more accurate checks by @dnth in #1022
• Add pip install zenml[server] to relevant examples by @dnth in #1027
• Add Tolerations and NodeAffinity to Kubernetes executor by @wefner in #994
• Support pydantic subclasses in BaseParameter attributes by @schustmi in #1023
• Unify run names across orchestrators by @schustmi in #1025
• Add gRPC metadata service to the ZenML helm chart by @stefannica in #1026
• Make the MLMD pipeline run information transfer synchronous by @stefannica in #1032
• Add console spinner back by @strickvl in #1034
• Fix Azure CLI auth problem by @wjayesh in #1035
• Allow non-local Label Studio instances for annotation by @strickvl in #1033
• Before deleting the global zen_server files, spin it down by @AlexejPenner in #1029
• Adding zenserver integration to stack recipe CLI by @wjayesh in #1017
• Add support for Azure ZenServer by @wjayesh in #1024
• Kubernetes Pod settings by @schustmi in #1008

New Contributors

• @alvarobartt made their first contribution in #1009
• @wefner made their first contribution in #994

Full Changelog: 0.20.5...0.21.0

0.20.5

ZenML 0.20.5 fixes another series of minor bugs, significantly improves the performance of the CLI, and adds an option to specify APT packages in Docker images.

What's Changed

• Fix accessing local zen store and artifact store in containers by @stefannica in #976
• K3d local registry pod spec updated by @wjayesh in #972
• Update readme page by @dnth in #985
• Remove beam dependency by @schustmi in #986
• Fix error message when registering secret without secrets manager by @schustmi in #981
• Update cheat sheet up to zenml==0.20.4 by @dnth in #987
• Example fixes (part 2) by @strickvl in #971
• Allow duplicate step classes inside a pipeline by @schustmi in #989
• Include deployment in azureml docker build by @schustmi in #984
• Automatically open browser upon zenml up command by @dnth in #978
• Add a just_mine flag for zenml stack list by @strickvl in #979
• Add option to specify apt packages by @schustmi in #982
• Replace old flavor references, fix the windows local ZenML server and other fixes by @stefannica in #988
• Improve docker and k8s detection by @schustmi in #991
• Update GH actions example by @schustmi in #993
• Update MissingStepParameterError exception message by @gabrielmbmb in #996
• Seprated code docs into core and integration docs by @AlexejPenner in #983
• Add docs/mkdocstrings_helper.py to format script sources by @fa9r in #997
• Further CLI optimization by @bcdurak in #992

Full Changelog: 0.20.4...0.20.5

0.20.4

This release fixes another series of minor bugs that were introduced in 0.20.0.

What's Changed

• Detect failed executions by @schustmi in #964
• Only build docker images for custom deployments by @schustmi in #960
• M1 Mac Installation Tutorial by @fa9r in #966
• Update ZenBytes links in docs by @fa9r in #968
• Fix the API docs builder by @stefannica in #967
• Fix gpu_limit condition in VertexOrchestrator by @gabrielmbmb in #963
• Add simple node affinitiy configurations by @schustmi in #973
• First iteration of the CLI optimization by @bcdurak in #962

Full Changelog: 0.20.3...0.20.4

0.20.3

This release fixes another series of minor bugs that were introduced in 0.20.0 and add full support docs for Kubeflow multi-tenancy deployments.

What's Changed

• Fixed GitHub/Colab JSON formatting error on quickstart. by @fa9r in #947
• Update YAML config template by @htahir1 in #952
• correct code from merge and fix import by @wjayesh in #950
• Check for active component using id instead of name by @schustmi in #956
• Tekton fix by @htahir1 in #955
• Improve zenml up/down UX and other fixes by @stefannica in #957
• Update kubeflow docs for multi-tenant deployments by @htahir1 in #958
• Update kubeflow.md by @abohmeed in #959
• Add additional stack validation for step operators by @schustmi in #954
• Fix pipeline run dashboard URL for unlisted runs by @fa9r in #951
• Support subclasses of registered types in recursive materialization by @fa9r in #953

New Contributors

• @abohmeed made their first contribution in #959

Full Changelog: 0.20.2...0.20.3

0.20.2

After a successful release of the new ZenML server and dashboard paradigm, we set to ironing out some bugs that slipped through.

What's Changed

• Capitalize all docs page titles. by @fa9r in #937
• Increase field sizes for docstrings and step parameters. by @fa9r in #940
• Fixing the bug in the registration of custom flavors by @bcdurak in #938
• Implemented docstring Attribute of StepModel by @fa9r in #936
• Fix shared stack emoji by @strickvl in #941
• Fix shared stacks not being allowed to be set as active. by @fa9r in #943
• Typo fix by @strickvl in #944
• Update Kubernetes Orchestrator Example by @fa9r in #942
• Add code and instructions to run quickstart on Colab. by @fa9r in #939
• Fixing the interaction in getting stacks/components by @bcdurak in #945
• Fix Kubeflow run name by @safoinme in #946
• VertexOrchestrator apply node selector constraint if gpu_limit > 0 by @gabrielmbmb in #935

Full Changelog: 0.20.1...0.20.2

0.20.0

The ZenML 0.20.0 release brings a number of big changes to its architecture and a lot of cool new features, some of which are not backwards compatible with previous versions.

These changes are only covered briefly in the release notes. For a detailed view on what happened and how you can get the most out of the 0.20.0 release, please head over to our "ZenML 0.20.0: Our Biggest Release Yet" blog post.

Warning: Breaking Changes

Updating to ZenML 0.20.0 needs to be followed by a migration of your existing ZenML Stacks and you may also need to make changes to your current ZenML pipeline code. Please read the migration guide carefully and follow the instructions to ensure a smooth transition. The guide walks you through these changes and offers instructions on how to migrate your existing ZenML stacks and pipelines to the new version with minimal effort and disruption to your existing workloads.

If you have updated to ZenML 0.20.0 by mistake or are experiencing issues with the new version, you can always go back to the previous version by using pip install zenml==0.13.2 instead of pip install zenml when installing ZenML manually or in your scripts.

Overview of Changes

• ZenML takes over the Metadata Store role. All information about your ZenML Stacks, pipelines, and artifacts is now tracked by ZenML itself directly. If you are currently using remote Metadata Stores (e.g. deployed in cloud) in your stacks, you will probably need to replace them with ZenML cloud deployments.
• the new ZenML Dashboard is now available with all ZenML deployments.
• ZenML Profiles have been removed in favor of ZenML Projects. You need to manually migrate your existing ZenML Profiles after the update.
• the configuration of Stack Components is now decoupled from their implementation. If you extended ZenML with custom stack component implementations, you may need to update the way they are registered in ZenML.
• the updated ZenML server provides a new and improved collaborative experience. When connected to a ZenML server, you can now share your ZenML Stacks and Stack Components with other users. If you were previously using the ZenML Profiles or the ZenML server to share your ZenML Stacks, you should switch to the new ZenML server and Dashboard and update your existing workflows to reflect the new features.

What's Changed

• Fix error in checking Great Expectations results when exit_on_error=True by @TimovNiedek in #889
• feat(user-dockerfile): Add user argument to DockerConfiguration by @cjidboon94 in #892
• Minor doc updates for backporting by @htahir1 in #894
• Removed feature request and replaced with hellonext board by @htahir1 in #897
• Unit tests for (some) integrations by @strickvl in #880
• Fixed integration installation command by @edshee in #900
• Pipeline configuration and intermediate representation by @schustmi in #898
• [Bugfix] Fix bug in auto-import of stack after recipe deploy by @wjayesh in #901
• Update TOC on CONTRIBUTING.md by @strickvl in #907
• ZenServer by @fa9r in #879
• Update kserve README by @strickvl in #912
• Confirmation prompts were not working by @htahir1 in #917
• Stacks can be registered in Click<8.0.0 now by @AlexejPenner in #920
• Made Pipeline and Stack optional on the HydratedPipelineRunModel by @AlexejPenner in #919
• Renamed all references from ZenServer to ZenML Server in logs and comments by @htahir1 in #915
• Prettify pipeline runs list CLI output. by @fa9r in #921
• Warn when registering non-local component with local ZenServer by @strickvl in #904
• Fix duplicate results in pipeline run lists and unlisted flag. by @fa9r in #922
• Fix error log by @htahir1 in #916
• Update cli docs by @AlexejPenner in #913
• Fix Pipeline Run Status by @fa9r in #923
• Change the CLI emoji for whether a stack is shared or not. by @fa9r in #926
• Fix running pipelines from different locations. by @fa9r in #925
• Fix zenml stack-component describe CLI command. by @fa9r in #929
• Update custom deployment to use ArtifactModel by @safoinme in #928
• Fix the CI unit test and integration test failures by @stefannica in #924
• Add gcp zenserver recipe by @wjayesh in #930
• Extend Post Execution Class Properties by @fa9r in #931
• Fixes for examples by @strickvl in #918
• Update cheat sheet by @dnth in #932
• Fix the docstring attribute of pipeline models. by @fa9r in #933
• New docs post ZenML Server by @htahir1 in #927

New Contributors

• @TimovNiedek made their first contribution in #889
• @cjidboon94 made their first contribution in #892
• @edshee made their first contribution in #900

Full Changelog: 0.13.2...0.20.0

0.13.2

ZenML 0.13.2 comes with a new local Docker orchestrator and many other improvements and fixes:

• You can now run your pipelines locally in isolated Docker containers per step
• @gabrielmbmb updated our MLFlow experiment tracker to work with Databricks deployments 🎉
• Documentation updates for cloud deployments and multi-tenancy Kubeflow support

What's Changed

• Update GitHub Actions by @fa9r in #864
• Raise zenml exception when cyclic graph is detected by @schustmi in #866
• Add source to segment identify call by @htahir1 in #868
• Use default local paths/URIs for the local artifact and metadata stores by @stefannica in #873
• Implement local docker orchestrator by @schustmi in #862
• Update cheat sheet with latest CLI commands from 0.13.0 by @dnth in #867
• Add a note about importing proper DockerConfiguration module by @jsuchome in #877
• Bugfix/misc by @schustmi in #878
• Fixed bug in tfx by @htahir1 in #883
• Mlflow Databricks connection by @gabrielmbmb in #882
• Refactor cloud guide to stack deployment guide by @wjayesh in #861
• Add cookie consent by @strickvl in #871
• Stack recipe CLI improvements by @wjayesh in #872
• Kubeflow workaround added by @htahir1 in #886

Full Changelog: 0.13.1...0.13.2

0.13.1

ZenML 0.13.1 is here and it comes with several quality of life improvements:

• You can now specify the exact order in which your pipelines steps should be
  executed, e.g., via step_b.after(step_a)
• TensorBoard was moved to a separate integration so you can use it with Pytorch
  and other modeling frameworks
• You can now configure the Evidently integration to ignore specific columns in
  your datasets.

This release also contains a lot of documentation on how to deploy
custom code (like preprocessing and postprocessing code) with our KServe and
Seldon integrations.

What's Changed

• Fix flag info on recipes in docs by @wjayesh in #854
• Fix some materializer issues by @schustmi in #852
• Add ignore columns for evidently drift detection by @SangamSwadiK in #851
• TensorBoard Integration by @fa9r in #850
• Add option to specify task dependencies by @schustmi in #858
• Custom code readme and docs by @safoinme in #853

New Contributors

• @SangamSwadiK made their first contribution in #851

Full Changelog: 0.13.0...0.13.1