Skip to content

Releases: zenml-io/zenml

0.61.0

09 Jul 14:02
@schustmi schustmi
0.61.0
50334c3
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.61.0

This release comes with a new and easy way to deploy a simple ZenML stack from the dashboard and the CLI. Give it a try by going to the Stacks section in the dashboard:

image
image

or by running the zenml stack deploy command!

We hope this makes it super easy for existing and new users to set up the infrastructure required to run ZenML pipelines on the cloud in one click.

Note: Only a simple AWS stack using Skypilot supported for now but GCP + Azure to come!

Additionally, this release includes improvements to our documentation and bugfixes for some integrations.

What's Changed

Full Changelog: 0.60.0...0.61.0

Contributors

strickvl, stefannica, and 4 other contributors
Assets 2
Loading

0.60.0

26 Jun 19:06
@htahir1 htahir1
0.60.0
235f333
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.60.0

ZenML now uses Pydantic v2. 🥳

This upgrade comes with a set of critical updates. While your user experience mostly remains unaffected, you might see unexpected behavior due to the changes in our dependencies. Moreover, since Pydantic v2 provides a slightly stricter validation process, you might end up bumping into some validation errors which was not caught before, but it is all for the better 🙂 If you run into any other errors, please let us know either on GitHub or on our Slack.

Changes in some of the critical dependencies

  • SQLModel is one of the core dependencies of ZenML and prior to this upgrade, we were utilizing version 0.0.8. However, this version is relatively outdated and incompatible with Pydantic v2. Within the scope of this upgrade, we upgraded it to 0.0.18.
  • Due to the change in the SQLModel version, we also had to upgrade our SQLAlchemy dependency from V1 to v2. While this does not affect the way that you are using ZenML, if you are using SQLAlchemy in your environment, you might have to migrate your code as well. For a detailed list of changes, feel free to check their migration guide.

Changes in pydantic

Pydantic v2 brings a lot of new and exciting changes to the table. The core logic now uses Rust, and it is much faster and more efficient in terms of performance. On top of it, the main concepts like model design, configuration, validation, or serialization now include a lot of new cool features. If you are using pydantic in your workflow and are interested in the new changes, you can check the brilliant migration guide provided by the pydantic team to see the full list of changes.

Changes in our integrations changes

Much like ZenML, pydantic is an important dependency in many other Python packages. That’s why conducting this upgrade helped us unlock a new version for several ZenML integration dependencies. Additionally, in some instances, we had to adapt the functionality of the integration to keep it compatible with pydantic. So, if you are using any of these integrations, please go through the changes.

Airflow

As mentioned above upgrading our pydantic dependency meant we had to upgrade our sqlmodel dependency. Upgrading our sqlmodel dependency meant we had to upgrade our sqlalchemy dependency as well. Unfortunately, apache-airflow is still using sqlalchemy v1 and is incompatible with pydantic v2. As a solution, we have removed the dependencies of the airflow integration. Now, you can use ZenML to create your Airflow pipelines and use a separate environment to run them with Airflow. You can check the updated docs right here.

AWS

Some of our integrations now require protobuf 4. Since our previous sagemaker version (2.117.0) did not support protobof 4, we could not pair it with these new integrations. Thankfully sagemaker started supporting protobuf 4 with version 2.172.0 and relaxing its dependency solved the compatibility issue.

Evidently

The old version of our evidently integration was not compatible with Pydantic v2. They started supporting it starting from version 0.4.16. As their latest version is 0.4.22, the new dependency of the integration is limited between these two versions.

Feast

Our previous implementation of the feast integration was not compatible with Pydantic v2 due to the extra redis dependency we were using. This extra dependency is now removed and the feast integration is working as intended.

GCP

The previous version of the Kubeflow dependency (kfp==1.8.22) in our GCP integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the kfp dependency to v2 (which has no dependencies on the Pydantic library). This is why you may see some functional changes in the vertex step operator and orchestrator. If you would like to go through the changes in the kfp library, you can
find the migration guide here.

Great Expectations

Great Expectations started supporting Pydantic v2 starting from version 0.17.15 and they are closing in on their 1.0 release. Since this release might include a lot of big changes, we adjusted the dependency in our integration to great-expectations>=0.17.15,<1.0. We will try to keep it updated in the future once they release the 1.0 version

Kubeflow

Similar to the GCP integration, the previous version of the kubeflow dependency (kfp==1.8.22) in our kubeflow integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the kfp dependency to v2 (which has no dependencies on the Pydantic library). If you would like to go through the changes in the kfp library, you can find the migration guide here. ( We also are considering adding an alternative version of this integration so our users can keep using kfp V1 in their environment. Stay tuned for any updates.)

MLflow

mlflow is compatible with both Pydantic V1 and v2. However, due to a known issue, if you install zenml first and then do zenml integration install mlflow -y, it downgrades pydantic to V1. This is why we manually added the same duplicated pydantic requirement in the integration definition as well. Keep in mind that the mlflow library is still using some features of pydantic V1 which are deprecated. So, if the integration is installed in your environment, you might run into some deprecation warnings.

Label Studio

While we were working on updating our pydantic dependency, the label-studio-sdk has released its 1.0 version. In this new version, pydantic v2 is also supported. The implementation and documentation of our Label Studio integration have been updated accordingly.

Skypilot

With the switch to pydantic v2, the implementation of our skypilot integration mostly remained untouched. However, due to an incompatibility between the new version pydantic and the azurecli, the skypilot[azure] flavor can not be installed at the same time, thus our skypilot_azure integration is currently deactivated. We are working on fixing this issue and if you are using this integration in your workflows, we recommend staying on the previous version of ZenML until we can solve this issue.

Tensorflow

The new version of pydantic creates a drift between tensorflow and typing_extensions packages and relaxing the dependencies here resolves the issue. At the same time, the upgrade to kfp v2 (in integrations like kubeflow, tekton, or gcp) bumps our protobuf dependency from 3.X to 4.X. To stay compatible with this requirement, the installed version of tensorflow needs to be >=2.12.0. While this change solves the dependency issues in most settings, we have bumped into some errors while using tensorflow 2.12.0 on Python 3.8 on Ubuntu. If you would like to use this integration, please consider using a higher Python version.

Tekton

Similar to the gcp and kubeflow integrations, the old version of our tekton integration was not compatible with pydantic V1 due to its kfp dependency. With the switch from kfp V1 to v2, we have adapted our implementation to use the new version of kfp library and updated our documentation accordingly.

Additional Changes

  • We have also released a new version of mlstacks with Pydantic v2 support. If you are using it in your development environment, you have to upgrade your mlstacks package as well.
  • Added zenml.integrations.huggingface.steps.run_with_accelerate to enable running any step using accelerate. This function is supported by a utility that wraps any step function into a CLI script (which is required by most distributed training tools).
  • Fixed a memory leak that was observed while using the ZenML dashboard to view pipeline logs or artifact visualizations logged through an S3 Artifact Store linked to an AWS Service Connector.
  • Previously, we had an option called build_options that allowed users to pass arguments to the docker build command. However, these options were only applied when building the parent image. On macOS with ARM architecture, one needs to specify platform=linux/amd64 to the build command to leverage local caching of Docker image layers. We have added a way to specify these build options for the "main" ZenML build as well, not just the parent image build. Additionally, users can now specify a .dockerignore file for the parent image build, which was previously not possible.

What's Changed

Read more

Contributors

strickvl, stefannica, and 6 other contributors
Assets 2
Loading

0.58.2

10 Jun 16:16
@avishniakov avishniakov
0.58.2
b8970a0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.58.2

The 0.58.2 minor release is packed with a set of improvements to the ZenML logging and ZenML Server.

With this release ZenML logging will:

  • Offer pagination of the logs during fetching via REST API
  • Store the full logs history on GCS Artifact Stores
  • Be performant running logging-heavy tasks, like TQDM logging or logging of training in any Deep Learning framework (also TQDM-backed)

What's Changed

Full Changelog: 0.58.1...0.58.2

Contributors

strickvl, stefannica, and 4 other contributors
Assets 2
Loading

0.58.1

06 Jun 18:54
@safoinme safoinme
0.58.1
eaadf5a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.58.1

The 0.58.1 release brings a set of minor enhancements and bugfix to the ZenML framework, such as the ability to delete all pipeline versions using the Client/CLI, providing greater flexibility and control over pipeline management. Users can now specify Python package installer arguments. Furthermore, a fix has been implemented for the Sentencepiece tokenizer materializer.

We are also excited to introduce breadcrumbs to our dashboard to improve your navigation experience. This new feature allows you to easily visualize the path of your Pipelines, Models, and Artifacts, providing clear orientation, quick return to any section with a single click, and effortless navigation.

We’d like to give a special thanks to @eltociear for their first contribution.

Docs re-work

We reworked the structure of our documentation pages to make it easier to find answers to your practical questions. Please do let us know if you have any feedback on the structure or the new style of the 'How To' section!

What's Changed

New Contributors

Full Changelog: 0.58.0...0.58.1

Contributors

strickvl, schustmi, and 5 other contributors
Assets 2
Loading

0.58.0

27 May 17:00
@avishniakov avishniakov
0.58.0
21edd86
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.58.0

New Annotators

This release brings in three new integrations for our annotator stack component: Prodigy, Argilla and Pigeon.

  • Pigeon works within Jupyter notebooks and supports a limited feature set but is great for experimentation and demos.
  • Argilla works both locally-deployed and when the annotation instance lives in the cloud (i.e. in the Hugging Face Spaces deployment which they recommend).
  • Prodigy is a powerful closed-source annotation tool that allows for efficient data labeling. With this integration, users can now connect ZenML with Prodigy and leverage its annotation capabilities in their ML pipelines.

Retry configuration for steps

This release also includes new retry configuration for the steps. The following parameters can be set:

  • max_retries: The maximum number of times the step should be retried in case of failure.
  • delay: The initial delay in seconds before the first retry attempt.
  • backoff: The factor by which the delay should be multiplied after each retry attempt.

To use this in your code:

from zenml.config.retry_config import StepRetryConfig

@step(retry=StepRetryConfig(max_retries=3, delay=10, backoff=2))
def step_3() -> None:
    # Step implementation
    raise Exception("This is a test exception")

or using a config.yaml:

steps:
  my_step:
    retry:
      max_retries: 3
      delay: 10
      backoff: 2

In addition, this release includes a number of bug fixes and documentation updates, such as a new LLM finetuning template powered by PEFT and BitsAndBytes and instructions for the new annotators.

Breaking changes

  • The interface for the base class of the annotator stack component has been updated to account for the fact that not all annotators will launch with a specific URL. So there is no longer an url argument passed in.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by bumping the mlflow version to 2.12.2

What's Changed

Full Changelog: 0.57.1...0.58.0

Contributors

strickvl, stefannica, and 4 other contributors
Assets 2
Loading

0.57.1

14 May 09:54
@safoinme safoinme
0.57.1
7f97fdc
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.57.1

This a minor release that brings a variety of enhancements for
the new dashboard release, a new update to the LLMOps guide (covering the use of rerankers in RAG pipelines), and an updated README. It also introduces some new improvements to the service connectors.

We'd like to give a special thanks to @ruvilonix for their first contribution.

What's Changed

New Contributors

Full Changelog: 0.57.0...0.57.1

Contributors

strickvl, stefannica, and 7 other contributors
Assets 2
Loading

0.57.0

02 May 15:15
@avishniakov avishniakov
0.57.0
dab417c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.57.0

We're excited to announce that we're open-sourcing our new and improved dashboard. This unifies the experience for OSS and cloud users, though OSS users will initially see some dashboard features unavailable in this launch release.

We're open-sourcing our dashboard for a few reasons:

  • to ensure that the dashboard experience is consistent across all users, for both the open-source and cloud versions
  • to make it easier for us to maintain and develop the dashboard, as we can share components between the two versions
  • to allow OSS contributions (and self-hosting and modifications) to the new dashboard
  • to open up possibilities for future features, particularly for our OSS users

New users of the ZenML in the dashboard will have a better experience thanks to a much-improved onboarding sequence:

Dashboard 2
The dashboard will guide you through connecting to your server, setting up a stack, connecting to service connectors as well as running a pipeline.

We’ve also improved the ‘Settings’ section of the dashboard and this is the new home for configuration of your repositories, secrets, and connectors, along with some other options.

Dashboard 3

What It Means for You

If you're already a cloud user, not much will change for you. You're already using the new dashboard for pipelines, models and artifacts. Your experience won’t change and for the moment you’ll continue using the old dashboard for certain components (notably for stacks and components).

If you're an open-source user, the new dashboard is now available to you as part of our latest release (0.57.0). You'll notice a completely refreshed design and a new DAG visualizer.

Dashboard 4

Unfortunately, some dashboard features are not yet ready so you'll see instructions on how to access them via the CLI. We hope to have these features returned into the product soon. (If you have a strong opinion as to which you'd like to see first, please let us know!) Specifically, secrets, stacks, and service connectors are not yet implemented in the new dashboard.

How to use the legacy dashboard

The old dashboard is still available to you. To run with the legacy dashboard pass the --legacy flag when spinning it up:

zenml up --legacy

Note that you can’t use both the new and old dashboard at the same time.

If you’re self-hosting ZenML instead of using ZenML Cloud, you can specify which dashboard you want to use by setting the ZEN_SERVER_USE_LEGACY_DASHBOARD environment variable pre-deployment. Specifying a boolean value for this variable will determine which dashboard gets served for your deployment. (There’s no dynamic switching between dashboards allowed, so if you wish to change which dashboard is used for a deployed server, you’ll need to redeploy the server after updating the environment variable.)

If you’re using ZenML Cloud, your experience won’t change with this release and your use of the dashboard remains the same.

What's Changed

Full Changelog: 0.56.4...0.57.0

Contributors

strickvl, stefannica, and 5 other contributors
Assets 2
Loading

0.56.4

24 Apr 13:21
@safoinme safoinme
0.56.4
17d6209
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.56.4

This release brings a variety of bug fixes and enhancements, including a new Comet Experiment Tracker integration, additional support for the uv package installer for zenml integration ... commands which significantly improves the speed of integration installations and dependency management, and a new evaluation section in the LLMOps guide.

In addition, it includes a number of bug fixes and documentation updates, such as a fix for cached artifacts produced via save_artifact inside steps linkage to the MCP.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by bumping the mlflow version to 2.12.1

What's Changed

Full Changelog: 0.56.3...0.56.4

Contributors

strickvl, stefannica, and 6 other contributors
Assets 2
Loading

0.56.3

09 Apr 16:44
@avishniakov avishniakov
0.56.3
46d40c4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.56.3

This release comes with a number of bug fixes and enhancements.

With this release you can benefit from new Lambda Labs GPU orchestrator integration in your pipelines. Lambda Labs is a cloud provider that offers GPU instances for machine learning workloads.

In this release we have also implemented a few important security improvements to ZenML Server mostly around Content Security Policies. Also users are from now on mandated to provide previous password during the password change process.

Also the documentation was significantly improved with the new AWS Cloud guide and the LLMOps guide covering various aspects of the LLM lifecycle.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by adding support for Schedule.start_time to the HyperAI orchestrator.

What's Changed

Full Changelog: 0.56.2...0.56.3

Contributors

strickvl, stefannica, and 7 other contributors
Assets 2
Loading

0.56.2

25 Mar 22:27
@safoinme safoinme
0.56.2
68bcb3b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
0.56.2

This release introduces a wide array of new features, enhancements, and bug fixes, with a strong emphasis on elevating the user experience and streamlining machine
learning workflows. Most notably, you can now deploy models using Hugging Face inference endpoints thanks to an open-source community contribution of this model deployer stack component!

Note that 0.56.0 and 0.56.1 were yanked and removed from PyPI due to an issue with the
alembic versions + migration which could affect the database state. This release
fixes that issue.

This release also comes with a breaking change to the services
architecture.

Breaking Change

A significant change in this release is the migration of the Service (ZenML's technical term for deployment)
registration and deployment from local or remote environments to the ZenML server.
This change will be reflected in an upcoming tab in the dashboard which will
allow users to explore and see the deployed models in the dashboard with their live
status and metadata. This architectural shift also simplifies the model deployer
abstraction and streamlines the model deployment process for users by moving from
limited built-in steps to a more documented and flexible approach.

Important note: If you have models that you previously deployed with ZenML, you might
want to redeploy them to have them stored in the ZenML server and tracked by ZenML,
ensuring they appear in the dashboard.

Additionally, the find_model_server method now retrieves models (services) from the
ZenML server instead of local or remote deployment environments. As a result, any
usage of find_model_server will only return newly deployed models stored in the server.

It is also no longer recommended to call service functions like service.start().
Instead, use model_deployer.start_model_server(service_id), which will allow ZenML
to update the changed status of the service in the server.

Starting a service

Old syntax:

from zenml import pipeline, 
from zenml.integrations.bentoml.services.bentoml_deployment import BentoMLDeploymentService

@step
def predictor(
    service: BentoMLDeploymentService,
) -> None:
    # starting the service
    service.start(timeout=10)

New syntax:

from zenml import pipeline
from zenml.integrations.bentoml.model_deployers import BentoMLModelDeployer
from zenml.integrations.bentoml.services.bentoml_deployment import BentoMLDeploymentService

@step
def predictor(
    service: BentoMLDeploymentService,
) -> None:
    # starting the service
    model_deployer = BentoMLModelDeployer.get_active_model_deployer()
    model_deployer.start_model_server(service_id=service.service_id, timeout=10)

Enabling continuous deployment

Instead of replacing the parameter that was used in the deploy_model method to replace the
existing service (if it matches the exact same pipeline name and step name without
taking into accounts other parameters or configurations), we now have a new parameter,
continuous_deployment_mode, that allows you to enable continuous deployment for
the service. This will ensure that the service is updated with the latest version
if it's on the same pipeline and step and the service is not already running. Otherwise,
any new deployment with different configurations will create a new service.

from zenml import pipeline, step, get_step_context
from zenml.client import Client

@step
def deploy_model() -> Optional[MLFlowDeploymentService]:
    # Deploy a model using the MLflow Model Deployer
    zenml_client = Client()
    model_deployer = zenml_client.active_stack.model_deployer
    mlflow_deployment_config = MLFlowDeploymentConfig(
        name: str = "mlflow-model-deployment-example",
        description: str = "An example of deploying a model using the MLflow Model Deployer",
        pipeline_name: str = get_step_context().pipeline_name,
        pipeline_step_name: str = get_step_context().step_name,
        model_uri: str = "runs:/<run_id>/model" or "models:/<model_name>/<model_version>",
        model_name: str = "model",
        workers: int = 1
        mlserver: bool = False
        timeout: int = DEFAULT_SERVICE_START_STOP_TIMEOUT
    )
    service = model_deployer.deploy_model(mlflow_deployment_config, continuous_deployment_mode=True)
    logger.info(f"The deployed service info: {model_deployer.get_model_server_info(service)}")
    return service

Major Features and Enhancements:

  • A new Huggingface Model Deployer has been introduced, allowing you to seamlessly
    deploy your Huggingface models using ZenML. (Thank you so much @dudeperf3ct for the contribution!)
  • Faster Integration and Dependency Management ZenML now leverages the uv library,
    significantly improving the speed of integration installations and dependency management,
    resulting in a more streamlined and efficient workflow.
  • Enhanced Logging and Status Tracking Logging have been improved, providing better
    visibility into the state of your ZenML services.
  • Improved Artifact Store Isolation: ZenML now prevents unsafe operations that access
    data outside the scope of the artifact store, ensuring better isolation and security.
  • Adding admin user notion for the user accounts and added protection to certain operations
    performed via the REST interface to ADMIN-allowed only.
  • Rate limiting for login API to prevent abuse and protect the server from potential
    security threats.
  • The LLM template is now supported in ZenML, allowing you to use the LLM template
    for your pipelines.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @dudeperf3ct he contributed to this release
by introducing the Huggingface Model Deployer. We'd also like to thank @moesio-f
for their contribution to this release by adding a new attribute to the Kaniko image builder.
Additionally, we'd like to thank @christianversloot for his contributions to this release.

What's Changed

Read more

Contributors

strickvl, stefannica, and 10 other contributors
Assets 2
Loading