-
Notifications
You must be signed in to change notification settings - Fork 254
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Real time deck support #1704
base: master
Are you sure you want to change the base?
Real time deck support #1704
Conversation
Signed-off-by: Yicheng-Lu-llll <[email protected]>
Signed-off-by: Yicheng-Lu-llll <[email protected]>
Signed-off-by: Yicheng-Lu-llll <[email protected]>
31104aa
to
2239968
Compare
Just realizing we could use this feature to solve this problem: provide the ability to display links to e.g. wandb runs which are generated at runtime before the task ends 🎉 |
This also solves the problem with eager mode, i.e. we can re-render the Flyte deck of subtasks being kicked-off by the eager workflow in real-time. |
@@ -72,6 +72,12 @@ def name(self) -> str: | |||
def html(self) -> str: | |||
return self._html | |||
|
|||
@classmethod | |||
def persist(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think self
should be cls
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh, I see. Thank you for pointing out!
How does this effort here relate to the discussion in flyteorg/flyte#3838 btw? |
@fg91 yeah, this is one of the issues that is effectively blocked, I would rather merge this with a re-usable flytekit -> flytepropeller communication mechanism. |
Sorry to bump up on this PR, but this seems like a pretty useful feature to us. Is there any update on it? Thx! |
@ketian-indeed would love to understand the type of visualization you want |
@kumare3, thanks for the quick response. I'm thinking of a use case like dumping a link (e.g. Datadog/WandB/MLflow links that are valuable to check while a model training is ongoing) to the Deck when a task starts and being able to see the link on the Deck while the task is running. I believe the Deck is currently shown once the task is completed - please correct me if I'm wrong. |
Hi @ketian-indeed we are infact adding support for W&B, Comet and neptune natively into flyte, Check this out. #2449 Also Datadog you should be able to add already using, Please let me know what you think? But, i also like the idea of Realtime decks - we will be adding support for this, but not like this PR. slightly different. |
Thanks, @kumare3! Configuring logging links in the UI seems to be a better solution for some of our use cases. I'll test that out. |
I think that is fair. We will prioritize this work in summer. But please add ideas / use cases here that way it is very easy to ensure we address. If you are open to contributing that would be awesome too. |
This, this, and this discussion are relevant in this context. TL;DR multiple use cases are currently blocked by not having a real-time communication channel between a task pod and propeller, e.g. live updates in the eager mode deck or adding a log link dynamically at runtime: @task
def train():
run_url = my_experiment_tracking_api.init_run()
flytekit.current_context().add_log_link(run_url) |
TL;DR
This PR adds real-time deck support.
After this PR, User can use
flytekit.Deck.persist()
to generate the html based on current collected metrics/information. So that user can see the real time deck even the task is running or has already failed. See the below example:real time deck support includes:
Type
Are all requirements met?