Exceedance metric figures

[limnoliver]

The goal is to create a more management-related analysis/visualization of model performance. To do so, we're going to use exceedance metrics, which evaluates how well the model does at predicting that stream temperatures will be over some threshold.

We settled on @aappling-usgs general visual that shows proportion of true positives, false positives, and false negatives, e.g.,

First, we establish that the hybrid models are useful for predicting exceedances, but show how limited our observations are in terms of being able to assess exceedances (you need a full summer of data to know whether you captured them all). Then, we "roll up" these predictions about exceedances to take a look at the network in terms of what habitat is available for cold water fish.

Challenges

• the exceedance metrics are currently calculated on all data, no matter whether it is from a completely observed summer or not. How do we fairly represent these data?

• Where do we spatially limit these analyses? This is only relevant to the headwaters where there is cold water habitat. I suggest limiting it to these temperature conservation reaches (on page 11).

• What temperature do we use as a threshold? The current metrics use the 75 degree threshold that managers use to guide releases, but, that is maximum temperature exceedance, and we are using mean. I suggest we lower our temperature threshold a bit. I suggest using the summer min:max ratio at Lordville to figure this number out.

Steps

• The first step is to figure out what we're working with. How many reaches have 5 or more exceedances? How many completely observed site-summers are available in these headwater reaches? This should use data from the target 2_observations/out/obs_temp_drb.rds.

• Depending on how many reaches have enough exceedances, create red/white/blue exceedance figure. If there are only a few sites, we can spatially reference them and create these barplots for each site. If a lot of sites, we'll need to figure out how to aggregate. Exceedance metrics (targets in 4_evaluation.yml) were calculated on all data, and data grouped by seg_id_nat, year, or month.

• Create a map showing what you can do with a good model! Aggregate exceedance metrics to show some metric that would be useful to managers -- average annual number of exceedances by reach