Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check and possibly reduce storage for (fargate) worker nodes #432

Open
mgeerdsen opened this issue May 23, 2022 · 1 comment
Open

check and possibly reduce storage for (fargate) worker nodes #432

mgeerdsen opened this issue May 23, 2022 · 1 comment
Assignees

Comments

@mgeerdsen
Copy link
Contributor

After switching bagit creation to EC2/EBS we should check the storage usage of the fargate based worker nodes:

  • Can we remove/reduce the ephemeral storage?
  • Do we still need EFS on worker nodes?
  • If we do, do we still need provisioned mode?

Also we should check if we can simplify the workernode scripts if we don't need to decide which storage area to use anymore.

@mgeerdsen mgeerdsen self-assigned this May 23, 2022
@mgeerdsen
Copy link
Contributor Author

mgeerdsen commented Jul 11, 2022

The following jobs use the WORKING_STORAGE_FAST exclusively if available:

  • jp2_conversion_wrapper (grok)
  • lurawave_wrapper
  • create_thumbs
  • create_jpeg
  • jpegConversionEditorial_s3
  • iadownload
  • jpylyzer
  • altoconvert

only use WORKING_STORAGE:

  • jpegConversionEditorialUpdate_s3
  • embedEditorialMeta_s3
  • layoutwizzard-cli

All these use exactly one storage area and make no decision based on size (as bagit does), so we only need one kind of storage. As WORKING_STORAGE_FAST is set to ephemeral storage, this means that nearly all jobs already use that and nearly none use the EFS.

If no jobs have had any trouble with the ephemeral storage, we should check the remaining job types and could get rid of one storage type (probably EFS).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant