Adding Script to Generate a Summary of Docker Images Used by Workflows #410
Conversation
There was a problem hiding this comment.
Not sure if this should be kept if there is an updated python version
There was a problem hiding this comment.
Let's delete it when this PR gets merged.
scripts/docker/docker_usage_sum.py
Outdated
| logging.basicConfig(level=logging.INFO) | ||
|
|
||
|
|
||
| # A script to collect which dockers are in use and which latest dockers are available |
There was a problem hiding this comment.
Add to a help message
scripts/docker/docker_usage_sum.py
Outdated
| OUT_SUMMARY_TSV = os.path.join(current_dir, "dockers.in_use.tsv") | ||
|
|
||
| if os.path.exists(OUT_SUMMARY_TSV): | ||
| os.remove(OUT_SUMMARY_TSV) |
There was a problem hiding this comment.
can we backup the existing one?
scripts/docker/docker_usage_sum.py
Outdated
|
|
||
| with open(wdl_path, "r") as file_content: | ||
| content = file_content.read() | ||
| pattern = re.compile(r'.*docker:.*"') |
There was a problem hiding this comment.
Having ^ yielded no matches so removed that.
There are some instances where there is no space between : and " so changed + to *
new pattern: \s*docker:\s*"
scripts/docker/docker_usage_sum.py
Outdated
|
|
||
| # Clear the previous line and print the progress | ||
| print(f"Progress: {progress:.2f}%\r", end="") | ||
| with open(OUT_SUMMARY_TSV, "a") as tsv_file: |
There was a problem hiding this comment.
write as oppsed to append?
scripts/docker/docker_usage_sum.py
Outdated
| print(f"Progress: {progress:.2f}%\r", end="") | ||
| with open(OUT_SUMMARY_TSV, "a") as tsv_file: | ||
| # Add header | ||
| tsv_file.write(f"DOCKER_NAME\tLATEST_TAG\tUSED_TAG\tFILE_LINE\tWDL_PATH") |
There was a problem hiding this comment.
should need an \n after WDL_PATH.
There was a problem hiding this comment.
Not sure why but adding a \n creates an empty line between the header and the first line with the docker-wdl info.
scripts/docker/docker_usage_sum.py
Outdated
| # If the latest tag is not found locally, try to get it from remote | ||
| latest_tag = get_latest_remote_docker_tag(docker_name) if latest_tag == "NA" else latest_tag | ||
| docker_detail.append(f"{docker_name}\t{latest_tag}\t{used_tag}\t{line_num}\t{wdl_path_sum}") | ||
| else: |
…rate_docker_usage_md.py to repo site by way of adding it to git_page.yml
…f output file directed to a different location from current dir
bshifaw
changed the title
Added a script that generates a TSV with a list of docker images used in our wdl files. The TSV contains