[2pt] PR: New SRC calibration point data w/ max extent

[RyanSpies-NOAA]

This code preprocesses the partner FIM benchmark HEC-RAS libraries and converts the inundation extent polygons into a edge point database for the input to the HAND SRC calibration/adjustment algorithm. The key changes with the new input data are the addition of the max stage/flow points as well as the removal of the 10m grid point snapping. Note that the raw data to run this code is not available for external users, so this code can only be run internally. Addresses #945, #1345.

Additions

data/nws/ahps_bench_polys_to_calb_pts.py: this script ingests the HEC-RAS partner FIM benchmark data and outputs huc level parquet files containing the water edge points with associated attributes.
data/nws/merge_nws_usgs_point_parquet.py: the script combines the nws and usgs parquet point files created seperately by the ahps_bench_polys_to_calb_pts.py script

Testing

This branch was tested on a subset of the alpha domain. Summarized results provided below:

Alpha evaluation for USGS partner fim benchmark sites
Major flood category -->

Minor flood category -->

Alpha evaluation for NWS partner fim benchmark sites
Major flood category (Note that the outlier degraded performance sites in this plot were originally dropped by the point processing scripts due to anomalies in file naming from the source data. These issues were resolved and the updated points were added to the database; however, I did not rerun the fim_pipeline and alpha test. -->

Minor flood category -->

Notes

Some additional inventory notes:

Total number of HUC8 parquet files with calibration points:

• Previous version: 152
• Updated version: 129

This reduction is largely due to a different method in prescribing the calibration points to the HUC in which the NWS lid point is attributed. The previous version of the points appears to have been generated by using a "cookie cutter" method of clipping points by the HUC8 polygons. The previous method appears to have resulted in numerous HUCs containing calibration points that were associated to a neighboring/downstream flood source which often resulted in erroneous calibration results. The new method appears to avoid those artifacts without compromising in other areas. Here is the list of 23 HUC8s that no longer have calibration point files in the new dataset:

Deployment Plan (For developer use)

How does the changes affect the product?

• Code only?
• If applicable, has a deployment plan be created with the deployment person/team?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

[RobHanna-NOAA]

Hey Ryan. Heads up.. the two variable changse in Bash_variables are already correct at 241002 for bridges and bathymetric (versus the older bathymetry) file. Do you have latest into your branch already?

[rspies]

Not sure I understand. The only change in bash_variables.env with this feature branch is the input_calib_points_dir. Everything else should be in sync with the current dev.

[hhs732]

This PR was reviewed for 6 HUCs including 04300103, 03100208, 02050103, 04100008, 07140104, and 07140102. The results were compared with dev under 3 flow files including 25- and 50-year floods and ana_flow 20230718T1313Z_ana_past_14day_max_high_flow_magnitude.csv. The following shows a summary of the results which were found.

[hhs732]

Dark purple depicts this branch and light purple shows dev.
HUC8 04300103:

[hhs732]

HUC0410008:

[hhs732]

HUC8 02050103:

[hhs732]

HUC8 07140102:

[hhs732]

HUC8 07140104:

[hhs732]

Alpha test also has been run for these 6 hucs. The following depicts the results for HUC8s 04300103 03100208 02050103 04100008 .

[hhs732]

Alpha test plots for 4 HUC8s of 04300103, 03100208, 02050103, 04100008:

[hhs732]

[hhs732]

The following depicts the alpha test results for HUC8s 07140104 and 07140102 .

[hhs732]