Features : Optional_Files, jobTimeout and LogStream to metadata

[geertvandeweyer]

Optional Files
As raised by Peter Thomas on slack :

Added support for optional files in the AWS handling of (de)localization and caching:

• support for "File?"
• support for "Array[File?] in manual assignment
• support for "Array[File?]" in globbing

Tested for input and output of individual tasks and workflows:

• optional files are allowed to be missing during (de)localization on both S3 and EFS (mentioned to log)
• optional files are considered valid in cache-hash calculation (if missing in original, and still missing => valid hit) (silent)

Note: Mixed types are not supported and cast to mandatory files:

• an array of [File_1, File_2] where _1 is optional and _2 is not, will be "non-optional" completely.

jobTimeout

It is now possible to specify a maximal job runtime (walltime) for jobs on the AWS backend. If the time is exceeded, the job is teminated. Use this to kill hanging jobs (seen in R multicore processing in our case). Added to README for documentation

LogStream

The CloudWatch LogGroupName, LogStreamName and the AWS region the job was executed in, are now added to the task call metadata. Query cromwell for metadata to retrieve it. These logstreams also contain info on (de)-localization, in contstrast to stdout/stderr from the metadata.

extra : minor optimization on hashing through EFS/MD5 files : if considered invalid, return a random string instead of the same message each time. This forces a cache-break.

For testing : see release 87.1-AWS in my own fork : https://github.com/geertvandeweyer/cromwell/releases/tag/87.1-AWS

[geertvandeweyer]

testing shows that : Array[File?] in call caching caused cache hit missing. => fix this before approving

[geertvandeweyer]

PR passed my testing. Ready for review & merging.