Skip to content

Releases: zenml-io/zenml

0.13.0

17 Aug 15:06
@schustmi schustmi
0.13.0
0e8bf6b
Compare
Choose a tag to compare
0.13.0

ZenML version 0.13.0 is chock-full with exciting features.

Custom Code Deployment is the
continuation of the Model Deployment story that we have been working on over the last few releases. Now
it is possible to deploy custom code along with your models using Kserve or Seldon.

With Spark this release also brings distributed processing into the ZenML toolkit.

Spinning up and configuring infrastructure is a difficult part of the MLOps journey
and can easily become a barrier to entry. Using our mlops-stacks
repository, it is now possible to spin up perfectly configured infrastructure with
the corresponding ZenML stack using the ZenML CLI.

As always, we've also included various bug fixes and lots of improvements to the documentation and our examples.

Breaking Changes

This release introduces a breaking change to the CLI by adjusting the access to
the stack component specific resources for secret-managers and
model-deployers to be more explicitly linked to the component. Here is how:

# `zenml secret register ...` becomes 
zenml secrets-manager secret register ...

# `zenml served_models list` becomes 
zenml model-deployer models list

What's Changed

New Contributors

Full Changelog: 0.12.0...0.13.0

Contributors

strickvl, stefannica, and 7 other contributors
Assets 2
Loading

0.12.0

02 Aug 14:56
@schustmi schustmi
0.12.0
0d75698
This commit was signed with the committer’s verified signature.
schustmi Michael Schuster
GPG key ID: C5884EDD066312AA
Learn about vigilant mode.
Compare
Choose a tag to compare
0.12.0

The 0.12.0 release comes with the third implementation of the ZenML Model Deployer abstraction: The KServe integration allows you to deploy any PyTorch, TensorFlow or SKLearn from within your ZenML pipelines!

We also added functionality to specify hardware resources on a step level to control the amount of memory, CPUs and GPUs that each ZenML step has access to. This is curretly limited to the Kubeflow and Vertex orchestrator but will be expanded in upcoming releases.

Additionally, we've added support for scoped secrets in our AWS, GCP and Vault Secrets Managers. These updated Secrets Managers allow you to configure a scope which determines if secrets are shared with other ZenML Secrets Managers using the same backend.

As always, we've also included various bug fixes and lots of improvements to the documentation and our examples.

What's Changed

Full Changelog: 0.11.0...0.12.0

Contributors

strickvl, stefannica, and 7 other contributors
Assets 2
Loading

0.11.0

20 Jul 07:33
@stefannica stefannica
0.11.0
c29bfd1
Compare
Choose a tag to compare
0.11.0

Our 0.11.0 release contains our new annotation workflow and stack component. We've been blogging about this for a few weeks, and even started maintaining our own repository of open-source annotation tools. With ZenML 0.11.0 you can bring data labeling into your MLOps pipelines and workflows as a first-class citizen. We've started our first iteration of this functionality by integrating with Label Studio, a leader in the open-source annotation tool space.

This release also includes a ton of updates to our documentation. (Seriously, go check them out! We added tens of thousands of words since the last release.) We continued the work on our data validation story from the previous release: Deepchecks is the newest data validator we support, and we updated our Evidently and whylogs integrations to include all the latest and greatest from those tools.

Beyond this, as usual we included a number of smaller bugfixes and documentation changes to cumulatively improve experience of using ZenML as a user. For a detailed look at what's changed, give our full release notes a glance.

🏷 Data Annotation with Label Studio

We've been hard at work on our new stack component and integration with Label Studio, the first of our data annotation tools. 🥳

Object Detection with ZenML and Label Studio

Annotators are a stack component that enables the use of data annotation as part of your ZenML stack and pipelines. You can use the associated CLI command to launch annotation, configure your datasets and get stats on how many labeled tasks you have ready for use.

Data annotation/labeling is a core part of MLOps that is frequently left out of the conversation. With this release, ZenML now supports annotation as an integrated and first-class citizen as part of the MLOps lifecycle.

Got feedback or just want to let us know how you feel? Connect with us or join us for a Community Meetup 👋 which happens every Wednesday!

🔎 More Data Validation with Deepchecks, Evidently & whylogs

We continued the work started in the previous release, adding a new integration with Deepchecks. Deepchecks is a feature-rich data validation open-source library to painlessly do data validation. Deepchecks can do a variety of data validation tasks, from data integrity checks that work with a single dataset to data+model evaluation to data drift analyses. All this can be done with minimal configuration input from the user, or customized with specialized conditions that the validation checks should perform. Check out our example if you want to see it in action!

We also updated our integrations with Great Expectations, Evidently and whylogs to support their latest releases. These tools have added lots of great new features in recent weeks and we now support most of them, all under the new data validator stack component and abstraction.

📖 Documentation & User Guides

We made a significant overhaul of our documentation since the last release:

  • The developer guide section is reworked to be more complete and beginner-friendly
  • We wrote a whole new 'MLOps stack' section, which contains detailed explanations for all MLOps stack components and their various implementations in ZenML
  • A new 'Cloud Guide' section contains complete setup guides for multiple cloud stacks. This will help you get started quickly.
  • We added a new ZenML cheatsheet that you can use to remind you of important CLI commands.

⌨️ CLI Improvements

We fixed a bug that was preventing users who upgraded to 0.10.0 from pulling new examples. This now works without any problem.

➕ Other Updates, Additions and Fixes

The latest release includes several smaller features and updates to existing functionality:

  • We fixed a bug in our Feast integration that prevented registration as a stack component.
  • We updated the structure of all our examples so that they now conform to all of the 'best practices' guidance we've made available in our docs.
  • We fixed some module and path resolution errors that were happening for Windows users.
  • We have combined all the MetadataStore contexts to speed up calls to the metadata store. This speeds up pipeline execution.
  • We now prevent providing extra attributes when initializing stack components. This could have led to unexpected behaviors so we now just prevent this behavior.
  • We've built several new Docker images. You can view them all over at dockerhub.
  • The facets magic display now works on Google Colab.
  • Our Azure Secret Schema now works with the secrets manager. An issue with how Azure handles secret names was preventing this, but we encoded the secret names to get around this shortcoming on the Azure platform.
  • @Val3nt-ML added a nested MLflow parameter (on the @enable_mlflow decorator) which will allow the creation of nested runs for each step of a ZenML pipeline in MLflow.
  • We enabled the fetching of secrets from within a step.
  • We now allow the fetching of pipelines and steps by name, class or instance.
  • You can now also add optional machine specs to VertexAI orchestrators, thanks to a PR from @felixthebeard.
  • We fixed a bug that was preventing users from importing pipeline requirements via a requirements.txt file if the file ended with a newline.

Breaking Changes

The 0.11.0 release remodels the Evidently and whylogs integrations as Data Validator stack components, in an effort to converge all data profiling and validation libraries around the same abstraction. As a consequence, you now need to configure and add a Data Validator stack component to your stack if you wish to use Evidently or whylogs in your pipelines:

  • for Evidently:

    zenml data-validator register evidently -f evidently
zenml stack update -dv evidently

  • for whylogs:

    zenml data-validator register whylogs -f whylogs
zenml stack update -dv whylogs

In this release, we have also upgraded the Evidently and whylogs libraries to their latest and greatest versions (whylogs 1.0.6 and evidently 0.1.52). These versions introduce non-backwards compatible changes that are also reflected in the ZenML integrations:

  • Evidently profiles are now materialized using their original evidently.model_profile.Profile data type and the builtin EvidentlyProfileStep step now also returns a Profile instance instead of the previous dictionary representation. This may impact your existing pipelines as you may have to update your steps to take in Profile artifact instances instead of dictionaries.

  • the whylogs whylogs.DatasetProfile data type was replaced by whylogs.core.DatasetProfileView in the builtin whylogs materializer and steps. This may impact your existing pipelines as you may have to update your steps to return and take in whylogs.core.DatasetProfileView artifact instances instead of whylogs.DatasetProfile objects.

  • the whylogs library has gone through a major transformation that completely removed the session concept. As a result, the enable_whylogs step decorator was replaced by an enable_whylabs step decorator. You only need to use the step decorator if you wish to log your profiles to the Whylabs platform.

Please refer to the examples provided for Evidently and whylogs to learn more about how to use the new integration versions:

🙌 Community Contributions

We received several new community contributions during this release cycle. Here's everybody who contributed towards this release:

👩‍💻 Contribute to ZenML!

Join our Slack to let us know if you have an
idea for a feature or something you'd like to contribute to the framework.

We have a new home for our roadmap where you can vote on your favorite upcoming
feature or propose new ideas for what the core team should work on. You can vote
without needing to log in, so please do let us know what you want us to build!

Contributors

felixthebeard and Val3nt-ML
Assets 2
Loading

0.10.0

28 Jun 08:58
@stefannica stefannica
0.10.0
92931d5
Compare
Choose a tag to compare
0.10.0

The 0.10.0 release continues our streak of extending ZenML with support for new orchestrators, this time by adding the Kubernetes Native Orchestrator. This orchestrator is a lightweight alternative to other distributed orchestrators like Airflow or Kubeflow that gives our users the ability to run pipelines in any Kubernetes cluster without having to install and manage additional tools or components.

This release features another integration that we are really excited about: the popular data profiling and validation library Great Expectations is our first Data Validator, a new category of stack components that we are in the process of standardizing, that will make data quality a central feature of ZenML. The ZenML Great Expectations integration eliminates the complexity associated with configuring the store backends for Great Expectations by reusing our Artifact Store concept for that purpose and gives ZenML users immediate access to Great Expectations in both local and cloud settings.

Last but not least, the release also includes a new secrets manager implementation, courtesy of our contributor @karimhabush, that integrates ZenML with the Hashicorp Vault Server as well as a few other bug fixes and improvements.

What's Changed

New Contributors

  • @chethanuk-plutoflume made their first contribution in #716
  • @dnth made their first contribution in #722
  • @karimhabush made their first contribution in #689

Full Changelog: 0.9.0...0.10.0

Contributors

strickvl, schustmi, and 6 other contributors
Assets 2
Loading

0.9.0

14 Jun 08:33
@schustmi schustmi
0.9.0
828516a
This commit was signed with the committer’s verified signature.
schustmi Michael Schuster
GPG key ID: C5884EDD066312AA
Learn about vigilant mode.
Compare
Choose a tag to compare
0.9.0

0.9.0

It's been a couple of weeks, so it's time for a new release! 0.9.0 brings two whole new orchestrators, one of which was contributed by a community member just one day after we unveiled new documentation for orchestrator extensibility! The release also includes a new secrets manager, a Slack integration and a bunch of other smaller changes across the codebase. (Our new orchestrators are exciting enough that they'll get their own blog posts to showcase their strengths in due course.)

Beyond this, as usual we included a number of smaller bugfixes and documentation changes to cumulatively improve experience of using ZenML as a user.

What's Changed

New Contributors

Full Changelog: 0.8.1...0.9.0

Contributors

strickvl, Reed-Schimmel, and 10 other contributors
Assets 2
Loading

0.8.1

25 May 08:43
@htahir1 htahir1
0.8.1
5d24186
This commit was signed with the committer’s verified signature.
schustmi Michael Schuster
GPG key ID: C5884EDD066312AA
Learn about vigilant mode.
Compare
Choose a tag to compare
0.8.1

0.8.1

ZenML 0.8.1 is here and it comes with support for Python 3.9 🎉. It also includes major updates to our documentation, fixes some broken links in our examples and improves the zenml go command which helps you get started with ZenML.

We did this fast turn-around to fix some bugs in 0.8.0 and get a quick release out for Python 3.9 which was a highly requested feature! #502

What's Changed

Full Changelog: 0.8.0...0.8.1

Contributors

strickvl, stefannica, and 4 other contributors
Assets 2
Loading

0.8.0

19 May 06:28
@htahir1 htahir1
0.8.0
5a4099a
Compare
Choose a tag to compare
0.8.0

🧘‍♀️ Extensibility is our middle name

  • The ability to register custom stack component flavors (and renaming types to flavor (Registering custom stack component flavors by @bcdurak in #541)
  • The ability to easily extend orchestrators
  • Documentation for stacks, stack components and flavors by @bcdurak inhttps://github.com//pull/607
  • Allow configuration of s3fs by @schustmi in #532
  • Ability to use SSL to connect to MySQL clients (That allows for connecting to Cloud based MYSQL deployments)
  • New MySQL metadata stores by @bcdurak in #580!
  • Docs and messaging change
  • Make Orchestrators more extensible and simplify the interface by @AlexejPenner in #581
  • S3 Compatible Artifact Store and materializers file handling by @safoinme in #598

Manage your stacks

  • Update stack and stack components via the CLI by @strickvl in #497
  • Add stack delete confirmation prompt by @strickvl in #548
  • Add zenml stack export and zenml stack import commands by @fa9r in #560

Collaboration

CLI improvements

  • CLI speed improvement by @bcdurak in #567
  • Ensure rich CLI displays full text and wraps table text by @strickvl in #577
  • Add CLI command to remove stack component attribute by @strickvl in #590
  • Beautify CLI by grouping commands list into tags by @safoinme in #546

New integrations:

Documentation / ZenBytes etc

➕ Other Updates, Additions and Fixes

🙌 Community Contributions

Contributors

strickvl, stefannica, and 9 other contributors
Assets 2
Loading

0.7.3

28 Apr 09:01
@schustmi schustmi
0.7.3
4d523f0
This commit was signed with the committer’s verified signature.
schustmi Michael Schuster
GPG key ID: C5884EDD066312AA
Learn about vigilant mode.
Compare
Choose a tag to compare
0.7.3

📊 Experiment Tracking Components

PR #530 adds a new stack component to ZenMLs ever-growing list: experiment_trackers allows users to configure your experiment tracking tools with ZenML. Examples of experiment tracking tools are Weights&Biases, mlflow, Neptune, amongst others.

Existing users might be confused, as ZenML has had MLflow and wandb support for a while now without such a component. However, this component allows uses more control over the configuration of MLflow and wandb with the new MLFlowExperimentTracker and
WandbExperimentTracker components. This allows these tools to work in more scenarios than the currently limiting local use-cases.

🔎 XGBoost and LightGBM support

XGBoost and LightGBM are one of the most widely used boosting algorithm libraries out there. This release adds materializers for native objects for each library.

Check out both examples here and PR's #544 and #538 for more details.

📂 Parameterized S3FS support to enable non-AWS S3 storage (minio, ceph)

A big complaint of the S3 Artifact Store integration was that it was hard to parameterize it in a way that it supports non-AWS S3 storage like minio and ceph. The latest release
made this super simple! When you want to register an S3ArtifactStore from the CLI, you can now pass in client_kwargs, config_kwargs or s3_additional_kwargs as a JSON string. For example:

zenml artifact-store register my_s3_store --type=s3 --path=s3://my_bucket \
    --client_kwargs='{"endpoint_url": "http://my-s3-endpoint"}'

See PR #532 for more details.

🧱 New CLI commands to update stack components

We added functionality to allow users to update stacks that already exist. This shows the basic workflow:

zenml orchestrator register local_orchestrator2 -t local
zenml stack update default -o local_orchestrator2
zenml stack describe default
zenml container-registry register local_registry --type=default --uri=localhost:5000
zenml container-registry update local --uri='somethingelse.com'
zenml container-registry rename local local2
zenml container-registry describe local2
zenml stack rename default new_default
zenml stack update new_default -c local2
zenml stack describe new_default
zenml stack remove-component -c

More details are in the CLI docs.
Users can add new stack components to a pre-existing stack, or they can modify
already-present stack components. They can also rename their stack and individual stack components.

🐛 Seldon Core authentication through ZenML secrets

The Seldon Core Model Deployer stack component was updated in this release to allow the configuration of ZenML secrets with credentials that authenticate Seldon to access the Artifact Store. The Seldon Core integration provides 3 different secret schemas for the 3 flavors of Artifact Store: AWS, GCP, and Azure, but custom secrets can be used as well. For more information on how to use this feature please refer to our Seldon Core deployment example.

Lastly, we had numerous other changes such as ensuring the PyTorch materializer works across all artifact stores
and the Kubeflow Metadata Store can be easily queried locally.

Detailed Changelog

New Contributors

Full Changelog: 0.7.2...0.7.3
Blog Post: https://blog.zenml.io/zero-seven-two-three-release/

Contributors

strickvl, stefannica, and 7 other contributors
Assets 2
Loading

0.7.2

14 Apr 14:44
@htahir1 htahir1
0.7.2
e882df3
Compare
Choose a tag to compare
0.7.2

0.7.2 is a minor release which quickly patches some bugs found in the last release to do with Seldon and Mlflow deployment.

This release also features initial versions of two amazing new community-led integrations: HuggingFace and Weights&Biases!

richard_socher_shoutout

Continuous model deployment with MLflow has been improved with ZenML 0.7.2. A new MLflow Model Deployer Stack component is now available and needs to be part of your stack to be able to deploy models:

zenml integration install mlflow
zenml model-deployer register mlflow --type=mlflow
zenml stack register local_with_mlflow -m default -a default -o default -d mlflow
zenml stack set local_with_mlflow

The MLflow Model Deployer is yet another addition to the list of Model Deployers available in ZenML. You can read more on deploying models to production with MLflow in our [Continuous Training and Deployment documentation section](https://docs.zenml.io/features/continuous training-and-deployment) and our MLflow deployment example.

What's Changed

New Contributors

  • @Ankur3107 made their first contribution to the HuggingFace integration in #467
  • @soumik12345 from Weights&Biases also made their first contribution to the wandb integration in #486!

Full Changelog: 0.7.1...0.7.2

Contributors

strickvl, stefannica, and 6 other contributors
Assets 2
Loading

0.7.1

11 Apr 19:11
@htahir1 htahir1
0.7.1
aac824b
Compare
Choose a tag to compare
0.7.1

0.7.1

Feast-Seldon

The release introduces the Seldon Core ZenML integration, featuring the Seldon Core Model Deployer and a Seldon Core standard model deployer step. The Model Deployer is a new type of stack component that enables you to develop continuous model deployment pipelines that train models and continuously deploy them to an external model serving tool, service or platform. You can read more on deploying models to production with Seldon Core in our Continuous Training and Deployment documentation section and our Seldon Core deployment example.

We also see two new integrations with Feast as ZenML's first feature store integration. Feature stores allow data teams to serve data via an offline store and an online low-latency store where data is kept in sync between the two. It also offers a centralized registry where features (and feature schemas) are stored for use within a team or wider organization. ZenML now supports connecting to a Redis-backed Feast feature store as a stack component integration. Check out the full example to see it in action!

0.7.1 also brings an addition to ZenML training library integrations with NeuralProphet. Check out the new example for more details, and the docs for more further detail on all new features!

What's Changed

Contributors

strickvl, stefannica, and 7 other contributors
Assets 2
Loading