Features : Optional_Files, jobTimeout and LogStream to metadata #55
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Nov 15, 2024
Closed
geertvandeweyer
changed the title
|
testing shows that : Array[File?] in call caching caused cache hit missing. => fix this before approving |
|
PR passed my testing. Ready for review & merging. |
|
Passed all functional tests in : cromwell-testing Ready for merging |
geertvandeweyer
merged commit c5c074b
into
henriqueribeiro:develop_aws
0 of 6 checks passed
geertvandeweyer
added a commit
that referenced
this pull request
Mar 13, 2025
…#58) * support for Array[File?] as job input/output * support for JobTimeout directive * expose logStreamName, logStreamGroup and Region to metadata * fix for optional files in cache_copy strategy * Added FuseMount option to runtime attributes, (re-)enabled optional localization of input files * fix globbing issues without directory prefix
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Optional Files
As raised by Peter Thomas on slack :
Added support for optional files in the AWS handling of (de)localization and caching:
Tested for input and output of individual tasks and workflows:
Note: Mixed types are not supported and cast to mandatory files:
jobTimeout
It is now possible to specify a maximal job runtime (walltime) for jobs on the AWS backend. If the time is exceeded, the job is teminated. Use this to kill hanging jobs (seen in R multicore processing in our case). Added to README for documentation
LogStream
The CloudWatch LogGroupName, LogStreamName and the AWS region the job was executed in, are now added to the task call metadata. Query cromwell for metadata to retrieve it. These logstreams also contain info on (de)-localization, in contstrast to stdout/stderr from the metadata.
** fix for globbing without foler prefix
See issue 46. Globbing now functions as expcected.
** Run jobs in privileged mode
Added option to enable fuse in AWS/Batch jobs. This allows to install & use tools like mount-s3 to "locally" access buckets instead of localizing. Usefull when extracting minor sections of big files, by tools not able to handle s3-urls as input.
** Optional Localization
(Re)-enabled support for the optional_localization flag in the WDL (see here).
extra : minor optimization on hashing through EFS/MD5 files : if considered invalid, return a random string instead of the same message each time. This forces a cache-break.
For testing : see release 87.1-AWS in my own fork : https://github.com/geertvandeweyer/cromwell/releases/tag/87.1-AWS