Formalize checking Pastas models and reporting results

[dbrakenhoff]

I want to create a more formal structure for checking Pastas models. These checks are performed to determine whether a model meets certain criteria, for example, to select models for further analysis.

Pastas already includes diagnostic checks in the stats module, but this new functionality should be more broad in its scope. This could include the results from these diagnostic checks in its results, but also custom user-defined functions that check some aspect of a time series model.

Now there are several options for implementing this functionality:

Option 1

Create a separate CheckModel class. This class can be instantiated empty, and subsequently checks can be added to this object.

checker = ps.CheckModel()
checker.add("R2 > 0.7", lambda ml, threshold: ml.stats.rsq() > threshold, kwargs={"threshold": 0.7"})  # lambda
checker.add("t95 < obs_period", check_response_tmax, kwargs={"cutoff": 0.95})  # custom function

checker.check(model)  

Or the checks can be provided to the class directly with a dictionary of check functions.

checklist = {
    "R2 > 0.7": {"func": lambda ml, threshold: ml.stats.rsq() > threshold, "threshold": 0.7},
    "t95 < obs_period": {"func": check_response_tmax, "cutoff": 0.95},
}
checker = CheckModel(checklist=checklist)

Applying these checks to any model:

checker.check(ml)  # returns DataFrame with checks per row, including a pass column True/False

This should return a DataFrame with results, including checks per row (some checks will output multiple rows, e.g. response t95 for each stressmodel in the model). These results should include pass/fail, the test statistic (e.g. R^2) and the threshold it is compared to.

Advantage is that the checks are kept separate from the model class, and can easily be applied to multiple models. The downside is maybe that this is slightly less intuitive (we have a lot of model related functionality stored under the model object (plots, stats, etc.) so it might be more logical to include this there as well).

Option 2

The second option is more similar to current functionality in Pastas, by adding checks submodule to the Model class. We can include some frequently performed checks by default that can be called directly from the checks submodule. But custom functions can also be applied given they follow some defined pattern (model as first argument, and returns DataFrame with a preferably formally defined structure).

# pre-programmed checks, each return a DataFrame with 1 or more rows
ml.checks.rsq_threshold(threshold=0.7)
ml.checks.response_tmax(cutoff=0.95)
ml.checks.gain_std(n_std=2)  # check gain > 2*std, reliable uncertainty estimate is modeler's responsibility

# custom check
ml.checks.custom("name", func, kwargs={"something": 2})

# apply many, using checklist defined before, concats results into 1 DataFrame
ml.checks.apply(checklist)

Advantages of this approach are that it is maybe more intuitive for users, given how the rest of pastas is designed. The downside is that it is slightly more difficult to apply the same checks to multiple models, though the last example in the snippet above addresses that somewhat.

So which option do you prefer? And should this be a core functionality or is this a good candidate for a plugin or custom extension? Let met know!

[raoulcollenteur]

I think this is related to what I proposed here: #625

Should we discuss this idea in the Issue? I think we al want this right?

[dbrakenhoff]

Forgot we already had an issue for this 😇. But yes, let's continue the development there.

[raoulcollenteur]

Option 2 but perhaps with static methods (checks) that can be called also without initiating the CheckModel class.

[martinvonk]

PR open: #829