Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Introduced User-Defined outputs_path for Pipeline Execution #402

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .docker/dev/docker-entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,7 @@ pip install -ve '/home/ftp/naas/extensions/naasai'

jupyter labextension develop --overwrite '/home/ftp/naas/extensions/naasai'

cd /home/ftp/

# Start jupyterlab.
tini -g -- "start-notebook.sh"
10 changes: 4 additions & 6 deletions naas/pipeline/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ class ExecutionContext:
output_dir: str = None
output_path: str = None

def __init__(self, output_dir: str = "pipeline_executions"):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MinuraPunchihewa, I think that what @jravenel refers to is to keep this default value when output_dir is not provided, that way we will keep the local pipeline_executions directory.

And then we can chose to completely overwrite it 👍

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Dr0p42 This default value is available in the run() method of the the Pipeline class.

def __init__(self, output_dir: str):
self.execution_id = str(uuid.uuid4())
self.output_dir = output_dir
self.timestamp = datetime.datetime.now()
Expand Down Expand Up @@ -428,7 +428,7 @@ def run(
self,
style: Literal["diagram", "progess"] = "diagram",
monitor: bool = True,
outputs_path="",
outputs_path="pipeline_executions",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Dr0p42,
This is what I meant. If the parameter is not passed, the results will be stored in the pipeline_executions directory.

):
"""Start the execution of the pipeline.

Expand All @@ -442,7 +442,7 @@ def run(
"""
if self.execution_ctx is not None:
raise PipelineAlreadyRan("This pipeline have already been executed.")
self.execution_ctx = ExecutionContext()
self.execution_ctx = ExecutionContext(outputs_path)
self.status = StepStatus.RUNNING
self.monitors = []
if monitor is True:
Expand Down Expand Up @@ -494,7 +494,6 @@ class DummyStep(Step):
"""

def __init__(self, name):

super().__init__(name)
self.output = None

Expand Down Expand Up @@ -570,7 +569,7 @@ def run(self, ctx: ExecutionContext):
)
self.status = StepStatus.COMPLETED
except Exception as e:
self.status = StepStatus.ERRORED
self.status = StepStatus.ERRORED
if not self.on_error:
raise Exception(e)

Expand All @@ -580,4 +579,3 @@ def run(self, ctx: ExecutionContext):
len(self.on_error.steps) > 0 or len(self.on_error.next_steps) > 0
):
self.on_error.run(ctx)

3 changes: 2 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,8 @@
"mprop==0.16.0",
"pydash==5.1.0",
"pyvis==0.3.0",
"rich"
"rich",
"tzlocal==2.1"
],
classifiers=[
"Programming Language :: Python :: 3.9",
Expand Down