Create Sagemaker pipeline schedules if specified #3271
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
github-actions
bot
added
internal
![hyperlint-ai[bot]](https://avatars.githubusercontent.com/in/718456?s=60&v=4){kind=small}
| @@ -141,7 +141,7 @@ To see a full end-to-end worked example of a custom orchestrator, [see here](htt | |||
|
|
|||
| There are some additional optional features that your orchestrator can implement: | |||
|
|
|||
| * **Running pipelines on a schedule**: if your orchestrator supports running pipelines on a schedule, make sure to handle `deployment.schedule` if it exists. If your orchestrator does not support schedules, you should either log a warning and or even raise an exception in case the user tries to schedule a pipeline. | |||
| * **Running pipelines on a schedule**: if your orchestrator supports running pipelines on a schedule, make sure to handle `deployment.schedule` if it exists. If your orchestrator ules, you should either log a warning and or even raise an exception in case the user tries to schedule a pipeline. | |||
There was a problem hiding this comment.
| * **Running pipelines on a schedule**: if your orchestrator supports running pipelines on a schedule, make sure to handle `deployment.schedule` if it exists. If your orchestrator ules, you should either log a warning and or even raise an exception in case the user tries to schedule a pipeline. | |
| * **Running pipelines on a schedule**: if your orchestrator supports running pipelines on a schedule, make sure to handle `deployment.schedule` if it exists. If your orchestrator rules, you should either log a warning and or even raise an exception in case the user tries to schedule a pipeline. |
Issues:
- Style Guide - (Spelling-error) Did you really mean 'ules'?
Fix Explanation:
The word 'ules' appears to be a typographical error for 'rules'. Correcting this spelling error improves the clarity and accuracy of the documentation.
…zenml into feature/add-sagemaker-schedule
| metadata.update( | ||
| { | ||
| "schedule_rule_name": execution["rule_name"], | ||
| "schedule_type": execution["schedule_type"], | ||
| "schedule_expression": execution["schedule_expr"], | ||
| "pipeline_name": execution["pipeline_name"], | ||
| } | ||
| ) | ||
|
|
||
| if next_execution := execution.get("next_execution"): | ||
| metadata["next_execution_time"] = next_execution.isoformat() |
| events_client = session.boto_session.client("events") | ||
| rule_name = f"zenml-{deployment.pipeline_configuration.name}" | ||
|
|
||
| # Determine first execution time based on schedule type |
There was a problem hiding this comment.
The same code is duplicated above
|
|
||
| # Create IAM policy for EventBridge | ||
| iam_client = session.boto_session.client("iam") | ||
| role_name = self.config.execution_role.split("/")[ |
There was a problem hiding this comment.
Isn't the execution role the one that should be assumed by the pipeline during execution, and not used for something around the pipeline execution like this scheduling? This seems more like it should be tied to the role of the service connector if applicable or solved in some other way.
| 3. Set up the SageMaker pipeline as the target | ||
|
|
||
| {% hint style="info" %} | ||
| If you run the same pipeline with a schedule multiple times, the existing schedule will be updated with the new settings rather than creating a new schedule. This allows you to modify schedules by simply running the pipeline again with new schedule parameters. |
There was a problem hiding this comment.
I would not do this to keep it consistent with all other orchestrators.
|
@htahir1 A sagemaker pipeline has a |
Describe changes
Add schedule support for sagemaker. Tested manually.
Pre-requisites
Please ensure you have done the following:
developand the open PR is targetingdevelop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.Types of changes