Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge existing workflows implementation into master #216

Open
4 of 33 tasks
mcopik opened this issue Aug 2, 2024 · 0 comments
Open
4 of 33 tasks

Merge existing workflows implementation into master #216

mcopik opened this issue Aug 2, 2024 · 0 comments
Labels
aws azure enhancement New feature or request gcp

Comments

@mcopik
Copy link
Collaborator

mcopik commented Aug 2, 2024

We have three branches with serverless workflows: original workflows, a newer branch workflow-benchmarks-updated that is up-to-date with branch, and feature/workflows

  • Finish linting
  • Support HTTP trigger for AWS step functions. Docs; issue
  • Request ID for Azure - why is it generated?
  • What is the purpose of MY_FUNCTION_NAME on Gcp?
  • Proper naming of all functions involved in a workflow; keep the same format
  • Clean up usage of request id - how is it passed and is it always clean? Split between request id of workflow and functions?
  • Avoid reuploading the same code package when sharing function on Google Cloud (possibly also AWS).
  • Clean up usage of payload
    • Verify that we don't get double nesting of payload in functions; confirmed in the choice state after failure. There, we get a double-nested payload that we extract.
    • We should safely pass errors without worrying of the nesting.
    • If AWS doesn't all allow restructuring the JSON with an optional error field without involving intrinsic functions or additional states, then we should implement it ourselves.
  • Restructurize
    • Workflow should be a module of benchmark (storage, nosql).
    • Rename benchmark directories with better numbers
    • Move workflow classes (generator, workflow) to subdirectories
  • Failure handling
    • Implement AWS
    • Implement Azure
    • Implement GCP
  • Handling of Redis config
    • CLI script with auth
    • Don't do replacement of vars, just proper envs
    • Ensure that Redis config is provided
    • Move downloading results to proper class
    • Move Redis download to processing
  • Any change in the workflow redeploys all of the functions on AWS -> any way to make it more efficient?
  • Licensing for code and data.
  • Docs
    • Workflow definition for each platform and mapping
    • Workflow sources
    • Workflow data
    • Workflow implementation
    • Redis and profiling
@mcopik mcopik added enhancement New feature or request aws azure gcp labels Aug 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws azure enhancement New feature or request gcp
Projects
None yet
Development

No branches or pull requests

1 participant