Skip to content

2.2.5+zg1.1
Compare
Choose a tag to compare
@talebzeghmi talebzeghmi released this 14 Jun 17:50
· 483 commits to feature/kfp since this release
2.2.5+zg1.1
6101ef0
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Workflow SDK Release 2.2.5+zg1.1

Release Summary:

Support for Persistent Volume Claim (PVC)

To use disk space, you can now specify persistent volume in @resource decorator per step. It is as simple as

@resources(volume="30G")
@step
def my_task():
    ...

By default the volume is mounted to /opt/metaflow_volume, and this volume is only available for the step decorated. If @retry is used, the volume will be shared across retries of this step - nice if you want to pick up from previous progress, and be sure to clean up otherwise.

You have options to customize PVC mount path, or make the volume available to all steps onwards. Two additional attributes volume_dir and volume_mode are needed:

@resources(volume="30G", volume_mode="ReadWriteMany", volume_dir=<your_preferred_path>)
@step
def my_task():
    ...

Refer to doc string here for more details.

PyTorchDistributedDecorator (@pytorch_distributed) is deprecated due to implementation similarity.

P3 GPU Instance Support

We are adding an option for P3 instance when a more powerful GPU is handy - introducing @accelerator decorator!

@accelerator sets the taints and node label for your steps. To request P3 instance:

@accelerator(type="nvidia-tesla-v100")
@resources(...)
@step
def my_task():
    ...

While other instances can be requested similarly in the future, additional work is needed to support each type. Please let us (aip teams) know if other unsupported instance types suit your use cases better.

Improve Zodiac integration and cost tracking

Services now automatically tagged with zodiac_service, zodiac_team. As a result cost will be tracked in each team's Zodiac page base on namespace profile settings. Be sure to update your team's Kubeflow profile to take advantage of this feature

Improve Datadog integration

Flow name, experiment name, run id, and step name are automatically added to K8s pod labels.
Stay tuned for dashboard filters using these attributes.

Metadata reporting fix in CICD

Fix a bug where metadata is determined at compile time, and not correctly tracking run time environment when uploading to Metaflow service.

Assets 2
Loading