Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replacing pipeline/notebook containers to Batch/sageMaker in AWS module(Transition to AWS environment) #43

Closed
swiri021 opened this issue Oct 1, 2021 · 7 comments · Fixed by #47, #49, #52, #74 or #77
Closed

Replacing pipeline/notebook containers to Batch/sageMaker in AWS module(Transition to AWS environment) #43

swiri021 opened this issue Oct 1, 2021 · 7 comments · Fixed by #47, #49, #52, #74 or #77
Assignees
@swiri021
Labels
Development Engineering Enhancement
Milestone
Phase 1

Comments

@swiri021
Copy link
Contributor

swiri021 commented Oct 1, 2021

Currently, we are using docker instance to deploy notebook and it could be replaced with AWS sageMaker, possibly, pipeline container can be deployed in ECS.
@swiri021 swiri021 added this to the Phase 0 milestone Oct 1, 2021
@swiri021 swiri021 self-assigned this Oct 1, 2021
@swiri021
Copy link
Contributor Author

swiri021 commented Oct 3, 2021 •
edited
Loading

For ECS deployment(Using AWS Fargate):
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html#ECS_CLI_tutorial_fargate_iam_role

@swiri021 swiri021 linked a pull request Oct 4, 2021 that will close this issue
Updated docker context structure and initial push to dockerHub registry #47
Merged
@swiri021 swiri021 pinned this issue Oct 4, 2021
@swiri021 swiri021 linked a pull request Oct 4, 2021 that will close this issue
Added SageMaker module in AWS module #49
Merged
@swiri021
Copy link
Contributor Author

swiri021 commented Oct 4, 2021 •
edited
Loading

@swiri021 swiri021 added Enhancement and removed AI/ML labels Oct 4, 2021
@swiri021 swiri021 linked a pull request Oct 5, 2021 that will close this issue
Connected pipelines to s3 bucket #52
Merged
@swiri021
Copy link
Contributor Author

swiri021 commented Oct 5, 2021 •
edited
Loading

  • Updated connection between S3 and snakemake, but has an issue in feature extaction pipeline.
  • SageMaker is fully connected with S3 buckets and github.

@swiri021
Copy link
Contributor Author

swiri021 commented Oct 6, 2021 •
edited
Loading

  • Pipelines do not need to specify output folder if we go to s3 bucket, think about updating for this point.
  • Currently pipeline output folder should not have directory path, just use the name only

@swiri021 swiri021 changed the title Replacing pipeline/notebook containers to ECS/sageMaker in AWS module Replacing pipeline/notebook containers to ECS/sageMaker in AWS module(Transition to AWS environment) Oct 6, 2021
@swiri021 swiri021 changed the title Replacing pipeline/notebook containers to ECS/sageMaker in AWS module(Transition to AWS environment) Replacing pipeline/notebook containers to Batch/sageMaker in AWS module(Transition to AWS environment) Oct 6, 2021
@swiri021
Copy link
Contributor Author

swiri021 commented Oct 6, 2021 •
edited
Loading

  • Need to implement AWS batch for pipeline

@swiri021
Copy link
Contributor Author

  • Need CloudFormation management for AWS resources

This was linked to pull requests Oct 11, 2021
Added CloudFormation and updated ec2 module #74
Merged
Added AWS lambda creation for AWS batch #77
Merged
Added AWS batch and AWS lambda testing module #79
Merged
@swiri021
Copy link
Contributor Author

swiri021 commented Oct 17, 2021 •
edited
Loading

  • Shared volume(EFS) + array job type in AWS batch could be used for sequential task execution for data pipeline.

This was linked to pull requests Oct 17, 2021
Added parallelized jobs execution part in Batch module #80
Merged
Added array job in AWS batch and partial data analysis #84
Merged
@swiri021 swiri021 linked a pull request Nov 3, 2021 that will close this issue
Merging from original engineering dev branch for feature extraction batch #96
Merged
@swiri021 swiri021 modified the milestones: Phase 0, Phase 1 Nov 5, 2021
@swiri021 swiri021 linked a pull request Nov 6, 2021 that will close this issue
Updated EFS setting for AWS batch #99
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@swiri021 swiri021

Labels
Development Engineering Enhancement
Projects
None yet
Milestone
Phase 1
Development

Successfully merging a pull request may close this issue.

Updated docker context structure and initial push to dockerHub registry OpenKBC/multiple_sclerosis_proj
Added SageMaker module in AWS module OpenKBC/multiple_sclerosis_proj
Connected pipelines to s3 bucket OpenKBC/multiple_sclerosis_proj
Added CloudFormation and updated ec2 module OpenKBC/multiple_sclerosis_proj
Added AWS lambda creation for AWS batch OpenKBC/multiple_sclerosis_proj
Added AWS batch and AWS lambda testing module OpenKBC/multiple_sclerosis_proj
Added parallelized jobs execution part in Batch module OpenKBC/multiple_sclerosis_proj
Added array job in AWS batch and partial data analysis OpenKBC/multiple_sclerosis_proj
Merging from original engineering dev branch for feature extraction batch OpenKBC/multiple_sclerosis_proj
Updated EFS setting for AWS batch OpenKBC/multiple_sclerosis_proj
1 participant
@swiri021