Skip to content

Latest commit

 

History

History
133 lines (100 loc) · 4.13 KB

how_to_set_up_experiments.md

File metadata and controls

133 lines (100 loc) · 4.13 KB
Raw

How to set up the experiments?

The pir_run argument experiments may be the most difficult one to grasp.

Below, we show how to put some research questions into the experiments.

See https://github.com/richelbilderbeek/pirouette_examples for all https://github.com/richelbilderbeek/pirouette examples.

1. What is the error BEAST2 makes from a phylogeny using the same diversification model as it was generated by?

Here one needs only one experiment:

model_type run_if measure_evidence beautier::inference_model
generative always (TRUE or) FALSE generative_model

We assume we know the generative model, which happens to be the BEAUti default:

generative_model <- beautier::create_inference_model()
testthat::expect_true(generative_model$site_model$name == "JC69")
testthat::expect_true(generative_model$clock_model$name == "strict")
testthat::expect_true(generative_model$tree_prior$name == "yule")

We create one experiment:

experiment <- create_experiment(
  model_type = "generative",
  run_if = "always",
  do_measure_evidence = FALSE,
  inference_model = generative_model
)

experiments <- list(experiment)

pir_run(
  # ...
  experiments = experiment
)

See https://github.com/richelbilderbeek/pirouette_example_1 for the complete code.

2. What is the error BEAST2 makes from a phylogeny when picking the best inference model?

To answer this question, we create as much candidate experiments as we have inference models:

model_type run_if measure_evidence beautier::inference_model
candidate best_candidate TRUE inference_model_1
candidate best_candidate TRUE inference_model_2
... ... ... ...
candidate best_candidate TRUE inference_model_39
candidate best_candidate TRUE inference_model_40 
experiments <- create_all_experiments()

Running:

pir_run(
  # ...
  experiments = experiments
)

3. What is the error BEAST2 makes from a phylogeny, when hand-picking an inference model, compared to the background noise?

This is the same setup as the first research question, expect now we need a twin tree.

model_type run_if measure_evidence beautier::inference_model
generative always (TRUE or) FALSE generative_model 
my_favorite_inference_model <- ...

OK, put that in the experiment:

experiment <- create_experiment(
  inference_model = my_favorite_inference_model
)

experiments <- list(experiments)
pir_params <- create_pir_params(
  # ...
  experiments = list(),
  twinning_params = create_twinning_params
)
pir_run(
  # ...
  pir_params = pir_params
)

See https://github.com/richelbilderbeek/pirouette_example_11 for the complete code.

4. What is the difference in error between a known generative model and the best of the other candidates?

model_type run_if measure_evidence beautier::inference_model
generative always TRUE or FALSE generative_model
candidate best_candidate TRUE Inference model 1
candidate best_candidate TRUE Inference model 2
... ... ... ...
candidate best_candidate TRUE Inference model 38
candidate best_candidate TRUE Inference model 39

Same as 1, but there are now two models that are run: the generative model and the best candidate model.

See https://github.com/richelbilderbeek/pirouette_example_2 for the complete code.