ALCF fixes: tiff_to_zarr.py dynamic pixelsize, init_tiff_to_zarr_globus_flow.py, fix for issue #27 #29
Conversation
…nvert to micrometer for the multiscales step. Updated init_tiff_to_zarr_globus_flow.py to account for the new change (passing in the h5 path to the flow) as well as automatically updating the Prefect Secret Block on the server with the new flow_id and func_id. Updated instructions for running check_globus_compute.py in the docstring.
…nt was inheriting from globus_sdk.Client: als-computing#27
davramov
changed the title
scripts/polaris/tiff_to_zarr.py
Outdated
| @@ -30,6 +37,13 @@ def set_permissions_recursive(path, permissions=0o2775): | |||
| os.chmod(path, permissions) # Also set permissions for the top-level directory | |||
|
|
|||
|
|
|||
| def read_pixelsize_from_hdf5(raw_directory: str) -> dict: | |||
There was a problem hiding this comment.
This is picky, but shouldn't the parameter be named raw_file, not raw_directory?
| @@ -42,14 +39,13 @@ def conversion_wrapper(rundir, h5_file_name, folder_path): | |||
| file_name = h5_file_name[:-3] if h5_file_name.endswith('.h5') else h5_file_name | |||
| command = ( | |||
There was a problem hiding this comment.
can we remove hard-coded paths?
| @@ -76,6 +72,17 @@ def create_flow_definition(): | |||
| return flow_definition | |||
|
|
|||
|
|
|||
| @task(name="update-flow-in-Prefect") | |||
There was a problem hiding this comment.
These look to me like they should be json blocks and not Secrets, no?
There was a problem hiding this comment.
Also, let's stick to snake_case for function names. I know, picky.
| @@ -31,9 +31,6 @@ def conversion_wrapper(rundir, h5_file_name, folder_path): | |||
| """ | |||
| import os | |||
| import subprocess | |||
| import time | |||
There was a problem hiding this comment.
Was the issue that references time.run() at the import or the use of time.time()
That's super duper odd. I would be interested to see print(time.time) if it's not occuring during the import.
There was a problem hiding this comment.
I couldn't recreate the issue locally but only when the Globus Flow called the function. There error listed specifically the line start = time.time(). What's odd to me is A) it was working before and B) I am calling time.time() and nowhere do I call time.run().
…ration/_tests/test_globus_flow.py to include a test for process_new_832_file_flow() in orchestration/flows/bl832/move.py (note that this still needs work to properly validate the data pruning task). Updated move.py to accept a config as input (similar to what we did in the ALCF script) so we can inject Config832 from the test script. Fixed a typo in orchestration/flows/bl832/prune.py where there was missing parantheses causing an issue on the flow-prd agent. Updated scripts/init_tiff_to_zarr_globus_flow.py, tiff_to_zarr.py, and alcf.py to address hardcoded path concerns.
|
Fixes #27 |
tiff_to_zarr.pyto read pixelsize from the raw h5 file and convert to micrometer for the multiscales step.init_tiff_to_zarr_globus_flow.pyto account for the new change (passing in the h5 path to the flow) as well as automatically updating the Prefect Secret Block on the server with the latestflow_idandfunc_idonce the script runs. Side note: I removed the time import from the function definition because I was getting a cryptic "time.run()is not defined error" despite only callingtime.time().check_globus_compute.pyin the docstring.