Calibration- High R^2, but also high stderr for individual parameters

[RTKCHydro2]

Hello,

I am running an iterative script to test different combinations of wells with a single piezometer.

An issue I have noticed is that some of the results suggest a very good fit of the model (high r^2 and EVP), but the standard deviation for individual parameters is also very high (100% to 10,000%) suggesting a poor calibration.

Since this script is iterative, and there are many combinations of wells to test, it would be challenging to fix parameters for each combination of wells.

Ultimately, I have two questions:

1. What percentage standard deviation is acceptable for a parameter?

2. Are there other metrics I should look at other than standard deviation to tell if a model is well calibrated (if so which ones, and what the cutoffs)?

I have read the 'calibration' discussion in the docs.

Thanks in advance.

[dbrakenhoff]

Hi @RTKCHydro2,

Thanks for posting your question!

First of all, are you using a NoiseModel? Before you can interpret the estimated parameter uncertainties in your model you should check whether these standard errors are reliable. These estimates are reliable when the noise (ml.noise() meets the criteria for white noise.

1. What percentage standard deviation is acceptable for a parameter?

This is difficult to answer. It depends on the goal of your model. If you're interested in specific contributions, then you might need stricter criteria than when you're using the model simulation as a whole. Note that if two parameters are highly correlated, the uncertainty in the parameter estimates might be high, but your model might still be reliable. It just means that two parameters cannot be individually estimated accurately, but taken together you can get a good estimate.

A common, somewhat arbitrary measure often used for the gain (the influence of the step response after infinite time, usually denoted with A) is to test whether the value of the parameter is smaller than 2 standard deviations.

2. Are there other metrics I should look at other than standard deviation to tell if a model is well calibrated (if so which ones, and what the cutoffs)?

There are several publications that propose criteria to check whether models are reliable, but this remains a somewhat subjective choice, and really depends on the application of your time series models. But perhaps the following publications offer some inspiration:

• Collenteur et al. (2023)
• Brakenhoff et al. (2022)
• Zaadnoordijk et al. (2019)

Unfortunately there are no hard criteria, and you'll have to think which criteria are applicable to your application. Hopefully the information above helps get you on your way! If you find criteria you think work for your case, feel free to post them here for the benefit (or review) of others!

[RTKCHydro2]

Thank you for the clarification! This is very helpful.

We haven't been using Noisemodel, but will now.