Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As a user, I want to see sampling uncertainty properly reflected in time to peak error diagrams #399

Open
HankHerr-NOAA opened this issue Feb 5, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@HankHerr-NOAA
Copy link
Contributor

James: Please adjust the title and description accordingly.

As reported in #382, I added sampling uncertainty to an event detection evaluation wherein the time to peak error was computed. As a user, I expect that, when I ask for sampling uncertainty in an evaluation that includes time to peak error, I'll see a plot of time to peak errors that includes whiskers. Either that, or, if the WRES does not support sampling uncertainty whiskers for the plot, then the plot will be identical to the results for an equivalent evaluation with no sampling uncertainty.

That was not the case here: the plot of time to peak error against earliest valid time did not include whiskers and was not the same. Example plots are in this comment. Note that James responded in three comments that followed that comment indicating that this would be another ticket and not necessarily related to event detection.

The declaration to use to reproduce this results is below. Just comment out the sampling_uncertainty component to get a "before" result. As always, the data backing this evaluation is in the VLab Redmine Large Data Sets Google Drive folder.

If any more information is needed, let me know. Thanks,

Hank

==========

label: Testing Event Based
observed:
  label: OBS Streamflow
  sources: /home/ISED/wres/wresTestData/issue92087/inputs/ABRN1_QME.xml
  variable: QME
  feature_authority: nws lid
  type: observations
  time_scale:
    function: mean
    period: 24
    unit: hours

predicted:
  label: "19161749 RetroSim CSVs"
  sources:
  - /home/ISED/wres/nwm_3_0_retro_simulations/wfo/OAX/19161749_nwm_3_0_retro_wres.csv.gz
  variable: streamflow
  feature_authority: nwm feature id
  type: simulations

features:
  - {observed: ABRN1, predicted: '19161749'}

time_scale:
  function: maximum
  #  period: 24
  period: 168
  unit: hours

event_detection: observed

sampling_uncertainty:
  sample_size: 1000
  quantiles: [0.05,0.95]

metrics:
  - time to peak error
  - sample size
  - pearson correlation coefficient

output_formats:
  - csv2
  - png
@HankHerr-NOAA
Copy link
Contributor Author

I used the declaration in the Description without event detection in staging. I think I confirmed that this issue has nothing to do with event detection. Here are the results using the declaration without event detection or sampling uncertainty:

Image

This evaluation, as declared, does not lend itself to time to peak error computations because I include no reference date window and its just a single time series for both observed and predicted. At least, I think that's why its just 0.

Here are the results with sampling uncertainty:

Image

As a user, when adding sampling_uncertainty, I expect either whiskers to be added to the plot OR I expect the same result (indicating sampling_uncertainty is not supported). Instead, I get a different result, and, as a user, that's surprising me. If there is a reasonable expectation, that's fine, I just need to know what that is.

Thanks,

Hank

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant