Feature request: model validator, model polisher

[sabinala]

Model testing:

• May or may not conform to the typical unit testing convention
• May consider testing all models included in the repo

Individual model tests (IM tests)
(1) Total population tests:
-(a) Is total population information inherent to the initial conditions, or is there a separate parameter?
-(b) Can a model be made scale free? i.e. scaled to a population value of 1
-(c) Check at different time points to ensure conservation of total population

(2) “Aggregate_parameter” interpretation:
-(a) Check that a Mapping between model parameters described in literature and the corresponding aggregate parameters used for model representation/simulation
--> This should help interpret the parameters that are calibrated, as well as develop the mappings in IM_5a and IM_5b

(3) Scale/units test for parameters:
-(a) In cases of “aggregate_parameters” visible at the mira-model level, do the units agree when combining model parameters?

• (i) Can be checked for individual parameters: agreement with each other
• (ii) Can be checked for aggregated parameters: agreement with state variables & corresponding equation
  -(b) Check if parameters are explicitly being divided by population
  --> if yes, then Derivative calculations will incorrect (since those already divide by total population). Related to issue Should aggregate_parameters include total population in the denominator? gyorilab/mira#172

(4) Parameter value tests
-(a) Is a distribution provided (i.e. uniform distribution between a range of two values, or a gaussian distribution with a mean and variance provided)
-(b) Is provenance or reasonable assumptions for this distribution choice provided?
--> we hope this provides the necessary information for a posteriori adjustments to parameter choices of simulated forecasts are not satisfactory
-(c) Argument for evidence ontology?

• (i) Author statement
• (ii) Calibration result
• (iii) Transformation of existing evidence (aggregate parameters)
• (iv) WAG

-(d) Prior predictive check against data to see if data is in support of parameters

(5) Missing Identifiability of initial conditions:
-(a) Ensure information is provided to map from observed variables in data to initial conditions for model state variables

• (i) Check for the existence of a function that maps observed variables for a start time map to the initial conditions of model state variables
• (ii) Reasoning provided for this function choice
• (iii) Observed variables should be conserved if fractions are used e.g. {infected} in observed data vs. {infected, sick, affected} model states
• (iv) Check that all model state variables are provided an initial condition
• (v) Check that the initial state values are reasonable
• (vi) Check that the solution mapping applied to the start state is the same as the observed variables for that date.

(6) Observation models
-(a) Ensure information is provided to map from simulated state variable outputs from the model to observed variables

• (i) Check for the existence of a function that maps model state variables to values that agree with variables in observed data
• (ii) Reasoning provided for this function choice
• (iii) Check that values for every observed variables is calculated
  -(b) Ensure mapping from IM_3b are not redundant combinations of variables
  --> Intended idea: Flow from state to state is allowable; one state = scalar multiple of another shouldn’t be allowed
  --> Example: “Hospitalization = 10% of Infected individuals” should not be allowed

(7) Stiffness test
-(a) Check if stiffness arises from simple “patient zero” initial conditions (i.e. minimum infectious population and maximum susceptible)
-(b) Check that differences in parameter scales are reasonable to avoid introducing stiffness into the system (may not be a concern…)

(8) Permanence of Death
-(a) Make sure dead individuals stay dead; i.e. no reaction that allow dead individuals to return to a non-dead state

Ensemble Model tests (EM tests)
(1) Test scale of each model separately to test agreement
-(a) See IM tests 1 and 1a
-(b) Ensure total populations are in agreement across different models (either scale-free or not)
(2) Test that initial conditions after mapping are in agreement across different models
(3) Test post-mapping variables are reasonable across different models (How do we do this?)