6 month Evaluation Scenario 2. (August 2024 scenario)

[djinnome]

Scenario 2: Reproducing SIDARTHE and SIDARTHE-V

In 2020 the SIDARTHE model was published to describe the first wave of the Covid-19 pandemic in Italy. In 2021, this model was updated to include vaccination (SIDARTHE-V).

1. Start with the original SIDARTHE model.
   a. First, you want to make sure you have a good understanding of the original model, can execute it, and reproduce the results found in the publication describing the model. The paper doesn’t include code, but there is an SBML version of the model. Regardless of the starting point, you think it’s feasible to create an executable version of the model and reproduce the results based on the model descriptions in the paper. The paper DOI is: 10.1038/s41591-020-0883-7, pdf: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7175834/pdf/41591_2020_Article_883.pdf. The BioModels repository (where the SBML model can be found) is here: https://www.ebi.ac.uk/biomodels/BIOMD0000000955. Please complete extraction and pass unit tests under the following conditions, to test the process when the source material is of various levels of quality or completeness. Please note that one of the code versions is an accurate representation of the model, and is organized and commented. The other code version has intentional mistakes in the model definition, and is not as well organized. Please consider the following conditions:
   - i. [Challenge] Ingest model and pass unit tests from publication alone (do not start with any code as input)
   - ii. Ingest model and pass unit tests from publication and corresponding Code Version A
   - iii. Ingest model and pass unit tests from publication and corresponding Code Version B
   b. There are two ‘unit tests’ we want to pass, to ensure that we understood and can reproduce the published model:

   • i. Unit Test SIDARTHE Petri net #1: Set the initial values and parameters, as described in the Supplementary Methods section of the publication (pg. 9 of the pdf):
     1. Initial Values: I = 200/60e6, D = 20/60e6, A = 1/60e6, R = 2/60e6, T = 0, H = 0, E = 0; S = 1 – I – D – A – R – T – H – E. Let total population = 60e6.
     2. Parameters: α = 0.570, β = δ = 0.011, γ = 0.456, ε = 0.171, θ = 0.371, ζ = η = 0.125, μ = 0.017, ν = 0.027, τ = 0.01, λ = ρ = 0.034 and κ = ξ = σ = 0.017.
     3. Simulate for 100 days, and determine the day and level of peak total infections (sum over all the infected states I, D, A, R, T). Expected output: The peak should occur around day 47, when ~60% of the population is infected.
   • ii. Unit Test Adding python source code for Scenario 2 #2: Now update the parameters to reflect various interventions that Italy implemented during the first wave, as described in detail on pg. 9. Simulate for 100 days, reproduce the trajectories in Fig. 2B, and determine the day and level of peak total infections (sum over all the infected states I, D, A, R, T). Expected output: Trajectories in Fig. 2B, peak occurs around day 50, with ~0.2% of the total population infected.

   c. The difference between 1.b.i and 1.b.ii are changes in some parameter values over time. Describe the difference in outcomes between b.i and b.ii. Perform a sensitivity analysis to understand the sensitivity of the model to parameter variations and determine which parameter(s) were most responsible for the change in outcomes.

   d. Now return to the situation in b.i (constant parameters that don’t change over time). Let’s say we want to increase testing, diagnostics, and contact tracing efforts (implemented by increasing the detection parameters ε and θ). Assume that θ >= 2* ε, because a symptomatic person is more likely to be tested. What minimum constant values do these parameters need to be over the course of a 100-day simulation, to ensure that the total infected population (sum over all the infected states I, D, A, R, T) never rises above 1/3 of the total population?

2. Next, we want to explore the updated model SIDARTHE-V, which is found at https://doi.org/10.1038/s41591-021-01334-5, pdf: https://www.nature.com/articles/s41591-021-01334-5
   a. Do a structural model comparison of the original SIDARTHE and SIDARTHE-V. The structural comparison work product should include a summary or diagram describing similarities and differences between the models, with respect to parameters, variables/states, pathways, etc.
   b. Set the same initial values and parameter settings in 1.b.i. Let V(t=0) = 0, τ (in SIDARTHE) = τ2 (in SIDDARTHE-V), and τ1 = (1/3)τ2 (reflecting the fact that the mortality rate for critical conditions (state T), will always be larger than for other infected states). Assume that the vaccination rate psi is 0 to start with. The SIDARTHE-V model allows for three main types of interventions: (1) Those that impact the transmission parameters (α, β, γ and δ) – social distancing, masking, lockdown; (2) Those that impact the detection parameters (ε, θ) – testing and contact tracing; (3) Those that impact the vaccination rate psi – vaccination campaigns. Assume previously stated constraints: θ >= 2 ε, and τ1 = (1/3)*τ2.

   • i. Let’s say our goal is to ensure that the total infected population (sum over all the infected states I, D, A, R, T) never rises above 1/3 of the total population, over the course of the next 100 days. If you could choose only a single intervention (affecting only one parameter), which intervention would let us meet our goal, with minimal change to the intervention parameter? Assume that the intervention will be implemented after one month (t = day 30), and will stay constant after that, over the remaining time period (i.e. the following 70 days). What are equivalent interventions of the other two intervention types, that would have the same impact on total infections?

     • ii. Let’s say our goal is to get the reproduction number R0 below 1.0, at some point within the next 100 days. Are there interventions that will allow us to meet our goal? If there are multiple options, which single intervention would have the greatest impact on R0 and let us meet our goal with minimal change to the intervention parameter? Assume that the intervention will be implemented after one month (t = day 30), and will stay constant after that, over the remaining time period (i.e. the following 70 days).

<style> </style>

Question	Task	Equivalent TA Workflow	Metrics
1a,b	·  Model Extraction ·  Unit Testing	TA1: Model Extraction; TA1: Model Execution/Unit Testing	Time: How long does knowledge extraction take? How long does it take to get model into executable form? This includes time to iterate on unit test(s) until confident output is correct. Accuracy: ·  Were you able to faithfully reproduce results of unit tests? ·  Qualitative score on metadata quality (correctness, relevance, completeness), based on human inspection of the equations, variables, parameters, etc. ·  (TA1 only) Qualitative score on correctness of groundings/alignment
1c,d	·    Simulations ·    Produce answers to scenario questions	TA3: Simulation Workflows (incl. sensitivity analysis, interventions); TA3: Answers to Scenario Questions	Time: How long does it take to set up and execute simulations and come up with answers to each part? Quality (qualitative): Does the answer address the scenario question adequately, and does it seem reasonable?
2a	Model Comparison	TA2: Model Comparison	Time: to execute model comparison Quality (qualitative): Is model comparison output interpretable and does it capture major differences and similarities correctly?
2b	·  Simulation ·  Provide answers to scenario questions	TA3: Simulation workflows (incl. sensitivity analysis, intervention optimization); TA3: Answers to Scenario Questions	Quality (qualitative): Does the answer address the scenario question adequately, and does it seem reasonable?