Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snakemake workflow #14

Draft
wants to merge 36 commits into
base: main
Choose a base branch
from
Draft

Snakemake workflow #14

wants to merge 36 commits into from

Conversation

thomas-fred
Copy link
Contributor

An attempt to wrap the existing model code in a light-touch snakemake workflow. Should make it easier to reproduce analysis results in the future.

Factoring out library code into local package and trying to clean up formatting, imports, etc. as I go.

Rules written:

  • Basic damages
  • Calculation of inputs to transport flow allocation (preparation of OD, etc.)
  • Nominal flow allocation

To-do notes:

  • transport_failure_flow_allocation is the exhaustive transport rerouting rule. It's a thin wrapper of the transport_failure_scenario_setup.py script and invokes multiple instances of transport_failure_analysis.py which currently consumes on the order 70GB RAM. Each network flow allocation (using igraph_scenario_edge_failures) takes a good few seconds. Need to parallelise otherwise network with ~108k edges will take several weeks to run (or otherwise reduce set of edges to fail).
  • Water model only has single script, but there's lots more on the cluster in mistral/jamaica-ccri/drought. Don't know where to start there.
  • Electricity model core to be installed (see JEM), then flow analysis to be captured in new rule based on Nadia and Tom's work.
  • Calculation of EAD and EAEL script requires all the losses from the various network models to be present -- this to be done last.

Per-chunk loading of input data (networks, flows) now uses more modern
file formats.
Networks no longer recreated for each failed edge. Rather copied from a
master, then an edge is deleted.

These improvements give ~2x speedup.

Updated pandas groupby usage to recent API.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant