Model diagnostics

[noamross]

We should have guidance that it should be standard practice that any projects with model building/fitting have a model diagnostics report as part of its workflow

• Model diagnostics should be in the same script/Rmd as model-fitting for single-script project
• Model diagnostics should be in an Rmd target that depends on the model for any targets workflow (with md/html comitted to repository or part of deployed documents)
• While model diagnostics vary depending on the application, they should generally include
  • High-level summaries (e.g., summary(model))
  • Convergence and fit quality. diagnostics (e.g. gam.check(gam_model), Rhat, ESS statistics and rank-histograms for MCMC chains)
  • Residual plots (standard for plots of lm models, part of gam.check())
  • Goodness-of-fit statistics (Deviance explained, R-squared)
    • For predictive models, out-of sample prediction statistics
    • For binary models, the full confusion matrix of false/true positives/negatives
  • Partial effect plots (standard for plot(gam_model), can be generated by DALEX::predict_parts
  • Variable importance plots for tree models and other black-box ML models

There should a model diagnostics Rmd template (probably comments/pseudocode only) in https://github.com/ecohealthalliance/container-template/tree/main/.github/workflows

cc @emmamendelsohn