-
Notifications
You must be signed in to change notification settings - Fork 35
Description
Since post processing is very slow and does a lot of HUC-level processing of rating curves, we want to move those SRC processes to the HUC pipeline. Ideally, we want to create a new bash file called synthetic_rating_curve_adjustments.sh
(or similar), and have it run after run_unit_wb.sh
here:
inundation-mapping/fim_process_unit_wb.sh
Line 94 in de07ee8
/usr/bin/time -v $srcDir/run_unit_wb.sh 2>&1 | tee $hucLogFileName |
Things to consider:
- We still want to be able to run SRC adjustments independently like we can now during a stand-alone post processing run. Commenting out the line above is an acceptable fulfillment, but make sure that SRCs are successfully reset using the base SRC as in
update_htable_src.py
fim_post_processing.sh
should only be left with processes that require data from multiple HUCs, like aggregating hydrotables and crosswalk tables- Opportunity for cleaning up bridges and roads
Update from Rob: A better place is to put it at the end of run_unit_wb.sh at the very end of the file, after processing the branches. if we put it at the end of run_unit_wb.sh, then if it fails, the fim_process_unit_wb.sh can handle the errors correctly. I try super hard ot leave any logic processing out of fim_process_unit_wb so it focuses on just catching errors and huc processing times. If we put it in fim_process_unit_wb, it can't catch errors from the new .sh file correctly. Carson's idea of making it a stand alone script that runs in fim_process_unit.wb is that if we ever do need run it again, we can as a stand alone tool.
Putting it in run_unit_wb means the huc processing metrics are good and include the calibration adjustments.
Note: One thing that will be tricky is that currently post processing, calls a number of parts that update hydrotables and we don't want to miss that. Might to do a think on that one. Maybe we let each huc do it's own calibration updates to its own huc?
Update from Rob: July 15th:
We came up with another related idea. If we do handle this with one new sh script that we can drop at run_unit_wb.sh and it can handle all stuff that came out of post processing.. .ie) src_adjustments.sh added to run_unit_wb.sh and inside that one has each of the src things that can change, the we can make a wrapper.
We could create a whole script files that can allow us to re-run src adjustments for all hucs. It would be a standalone tool, that can get a list of HUCs (or grab it), then call that new script for each HUC that has the src_adjustments.sh. This allows us to change a src attribute or test, then run all or a number of hucs which can re-calc just src adjustments for each one. It would be pretty similar to re-running post processing but it will be faster and gives us some dev options or the ability to re=run post processing for src changes if we want. The trick is how we choose to build the new scripts.
We can also decide if we want the separate PR to for this wrapper or do it here. :)