Skip to content

[21pt] Move SRC adjustments to HUC processing #1593

@CarsonPruitt-NOAA

Description

@CarsonPruitt-NOAA

Since post processing is very slow and does a lot of HUC-level processing of rating curves, we want to move those SRC processes to the HUC pipeline. Ideally, we want to create a new bash file called synthetic_rating_curve_adjustments.sh (or similar), and have it run after run_unit_wb.sh here:

/usr/bin/time -v $srcDir/run_unit_wb.sh 2>&1 | tee $hucLogFileName

Things to consider:

  • We still want to be able to run SRC adjustments independently like we can now during a stand-alone post processing run. Commenting out the line above is an acceptable fulfillment, but make sure that SRCs are successfully reset using the base SRC as in update_htable_src.py
  • fim_post_processing.sh should only be left with processes that require data from multiple HUCs, like aggregating hydrotables and crosswalk tables
  • Opportunity for cleaning up bridges and roads

Update from Rob: A better place is to put it at the end of run_unit_wb.sh at the very end of the file, after processing the branches. if we put it at the end of run_unit_wb.sh, then if it fails, the fim_process_unit_wb.sh can handle the errors correctly. I try super hard ot leave any logic processing out of fim_process_unit_wb so it focuses on just catching errors and huc processing times. If we put it in fim_process_unit_wb, it can't catch errors from the new .sh file correctly. Carson's idea of making it a stand alone script that runs in fim_process_unit.wb is that if we ever do need run it again, we can as a stand alone tool.

Putting it in run_unit_wb means the huc processing metrics are good and include the calibration adjustments.

Note: One thing that will be tricky is that currently post processing, calls a number of parts that update hydrotables and we don't want to miss that. Might to do a think on that one. Maybe we let each huc do it's own calibration updates to its own huc?


Update from Rob: July 15th:
We came up with another related idea. If we do handle this with one new sh script that we can drop at run_unit_wb.sh and it can handle all stuff that came out of post processing.. .ie) src_adjustments.sh added to run_unit_wb.sh and inside that one has each of the src things that can change, the we can make a wrapper.

We could create a whole script files that can allow us to re-run src adjustments for all hucs. It would be a standalone tool, that can get a list of HUCs (or grab it), then call that new script for each HUC that has the src_adjustments.sh. This allows us to change a src attribute or test, then run all or a number of hucs which can re-calc just src adjustments for each one. It would be pretty similar to re-running post processing but it will be faster and gives us some dev options or the ability to re=run post processing for src changes if we want. The trick is how we choose to build the new scripts.

We can also decide if we want the separate PR to for this wrapper or do it here. :)

Metadata

Metadata

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions