Skip to content

Latest commit

 

History

History
490 lines (350 loc) · 16.2 KB

README.md

File metadata and controls

490 lines (350 loc) · 16.2 KB
Raw

NISMOD v2

NISMOD (National Infrastructure Systems Model) is an integrated model of infrastructure systems, developed as part of the ITRC/MISTRAL project.

NISMOD v2 has several components:

  • released versions of infrastructure sector models
  • an integration framework, smif
  • smif configuration to link the models into a system-of-systems model
  • smif-compatible scenario, narrative and planning data to run linked model runs
  • (for some models) internal model data, used by the model but not exposed to smif as part of the system-of-systems configuration

This repository contains only:

  • smif configuration needed to link the models into a system-of-systems and set up model runs
  • model wrappers and various helpers to set up runs and explore results.

In particular:

  • prepared model data is not yet provided (references to Amazon S3 buckets below are not public)
  • models and dependencies must be installed separately, following instructions below (water and energy supply models are not open; transport and digital models are open)

Setup

To set up NISMOD, first clone this repository, or download a release. The directories in this folder contain or will contain everything required for a model run:

  • config: model configuration
  • data: (initially incomplete) model data
  • models: model wrappers
  • notebooks: jupyter notebooks for exploring model results
  • planning: decision models
  • provision: scripts to install models and get data
  • results: (initially empty) for model run results
  • test: integration tests
  • utilities: helpful scripts to create or migrate configuration

Then in outline (more details further below) either install everything directly to run nismod on a desktop, server or cluster:

  1. Install smif (recommend using conda)
  2. Install model dependencies:
    • FICO XPRESS (for energy supply)
    • Java (for transport)
    • PostgreSQL and ODBC (for energy supply)
  3. Configure:
    • connection details for the NISMOD AWS S3 bucket s3://nismod2-data (copy provision/template.s3.ini to provision/s3.ini and edit)
    • connection details for a Postgres database (copy provision/template.dbconfig.ini to provision/dbconfig.ini and edit for your database)
  4. Install models (run provision/install_* scripts)
  5. Get data (run provision/get_data_* scripts)

Or set up a virtual machine for testing:

  1. Check that you have VirtualBox and Vagrant installed
  2. Configure:
    • connection details for the NISMOD AWS S3 bucket (copy template.s3.ini to s3.ini and edit)
    • connection details for a Postgres database (copy template.dbconfig.ini to dbconfig.ini)
    • FICO XPRESS license (copy template.xpauth.xpr or your own license to xpauth.xpr)
  3. Setup a virtual machine (run vagrant up then connect with vagrant ssh)

Download NISMOD

Download the latest release of NISMOD v2.0 from GitHub as a .zip archive and unzip it into a directory.

Alternatively, clone the NISMOD v2.0 repository from https://github.com/nismod/nismod2.

cd path/to/git/projects  # somewhere to keep the nismod project
git clone https://github.com/nismod/nismod2.git
cd nismod2

To update NISMOD to the latest development version:

cd path/to/nismod2   # wherever you cloned the nismod folder
git checkout master  # to make sure you’re on the master branch
git pull             # pull changes from Github to your local machine

Configure connection details for the NISMOD AWS S3 bucket

Within the NISMOD folder, copy provision/template.s3.ini to provision/s3.ini and add your credentials to access the NISMOD AWS S3 bucket:

[profile nismod2-s3]
aws_access_key_id = <your-access-key-id>
aws_secret_access_key = <your-secret-access-key>
region = eu-west-2
output = json

Contact Tom Russell or Craig Robson to request credentials.

Upload data to the NISMOD AWS S3 bucket

This assumes the awscli command line client is installed and you have credentials.

pip install awscli  # install
aws configure  # configure (this will ask for access key and secret key)

To check contents of the bucket:

aws s3 ls s3://nismod2-data

To upload a zip:

aws s3 cp mydata_v1.1.zip s3://nismod2-data/model_name/mydata_v1.1.zip

Further usage and documentation: AWS Command Line Interface docs

Configure connection details for a Postgres database

Within the NISMOD folder, copy provision/template.dbconfig.ini to provision/dbconfig.ini and edit to connect to your database.

If you intend to use the vagrant virtual machine with the database as provisioned by default, leave the details unchanged.

[energy-supply]
dbname=vagrant
host=localhost
user=vagrant
password=vagrant
port=5432

Configure FICO XPRESS license

If you will install FICO XPRESS manually, skip this step.

Within the NISMOD folder, copy provision/template.xpauth.xpr to provision/xpauth.xpr.

If you are running within the University of Oxford OUCE network, the license server should be available and no changes are necessary. Otherwise, set the value to connect to your local license server or replace this file with a license file provided by FICO support (contact via https://www.fico.com)

use_server server="ouce-license.ouce.ox.ac.uk"

Running NISMOD on desktop/server/cluster

To run NISMOD directly, you will need to install smif, model dependencies and models.

The provision/vm_provision.sh script may be a useful point of reference, as it is run by the root user within the virtual machine (on Ubuntu 16.04) to automate the same process.

Install smif

See installation instructions for full details.

The recommended method uses conda):

conda config --add channels conda-forge  # enable the conda-forge channel, required for smif
conda create --name nismod python=3.6 smif  # create a conda environment for nismod
conda activate nismod  # activate the nismod conda environment

Install model dependencies

FICO XPRESS (for energy supply)

See energy supply documentation for full details.

On Linux (x64) it should be possible download and install FICO by running:

mkdir xpress_setup
wget -nc -qO- \
  "https://clientarea.xpress.fico.com/downloads/8.5.10/xp8.5.10_linux_x86_64_setup.tar" \
  --no-check-certificate | tar -C xpress_setup -xv
./xpress_setup/install.sh

Java (for transport)

See transport model documentation for full details.

To run the transport model, it should be sufficient to have the Java Runtime Environment (JRE) version 8 installed.

On Ubuntu, run:

sudo apt update && sudo apt install default-jre

PostgreSQL and ODBC (for energy supply)

See energy supply documentation for full details.

The energy supply model connects to a Postgres database via ODBC, so requires access to a Postgres server, an ODBC installation and the Postgres ODBC drivers.

On Ubuntu, install Postgres, libpq (shared library for postgres clients), ODBC and postgres connector:

sudo apt update && sudo apt install \
  postgresql postgresql-contrib \
  libpq-dev \
  unixodbc-dev odbc-postgresql

Get data

Install boto3: pip install boto3.

Run each of the provision/get_data_* scripts:

bash ./provision/get_data_digital_comms.sh .
bash ./provision/get_data_energy_demand.sh .
bash ./provision/get_data_energy_supply.sh .
bash ./provision/get_data_transport.sh .
bash ./provision/get_data_water_supply.sh .
bash ./provision/get_data_water_demand.sh .

Install models

Run each of the provision/install_* scripts:

bash ./provision/install_digital_comms.sh .
bash ./provision/install_energy_demand.sh .
bash ./provision/install_energy_supply.sh . ./path/to/xpress_install_dir
bash ./provision/install_transport.sh .
bash ./provision/install_water_supply.sh .
bash ./provision/install_water_demand.sh .

Run post-install scripts:

energy_demand setup -f ./models/energy_demand/wrapperconfig.ini

Note that install_water_supply.sh clones a repository that is currently private: setting up the water supply model may require requesting access to the repository.

Running NISMOD on a virtual machine

The Vagrantfile defines the automated setup for a virtual machine, which will provide a reproducible development environment.

In virtual machine terminology, the virtual machine (vm for short) is sometimes called the ‘guest’ machine and the physical computer is called the ‘host’ machine.

To use the NISMOD virtual machine, first install:

  1. Virtualbox
  2. Vagrant

If you see only 32-bit options in Virtualbox, please ensure that:

  1. Hardware virtualization is enabled in the BIOS
    • For Intel x64: VT-x (Intel Virtualization Technology) and VT-d are both enabled
    • For AMD x64: AMD SVM (Secure Virtual Machine) is enabled
  2. For Windows: in Windows Features, "Hyper-V platform" is disabled.

Note for Ubuntu 17.10 users: If you are experiencing the issue The box ‘bento/ubuntu-16.04’ could not be found or could not be accessed in the remote catalog. Make sure that you have the latest version of Vagrant (>v2) installed. This version is currently not in the standard package archive (PPA) but can be downloaded from the vagrant website.

Create and configure the guest machine, running this command from the NISMOD directory:

vagrant up

This will download a virtual machine image, install all the packages and software which are required to test and run NISMOD onto that virtual machine and download the data and model releases from the AWS S3 bucket.

Once it has run, you should be able to:

vagrant ssh          # log in to the virtual machine on the command line
cd /vagrant          # move to the folder that’s shared between the host and guest machines
ls                   # list files and folders
smif list            # list available model runs
smif run energy_supply_toy  # to run the energy_supply_toy model run
logout               # log out of the virtual machine (or shortcut: CTRL+D)

Then results are written to subfolders within the results directory.

Running NISMOD with smif binary (parquet) store

Parquet files are smaller, typed and faster to read and write than CSV.

Run smif csv2parquet converter after getting data and installing models, or after updates.

smif csv2parquet --noclobber data

Then run model-runs using the local_binary interface. smif will read and write parquet files.

smif run et_module_test --interface local_binary

Updating

To reload and re-provision the virtual machine after updating NISMOD:

vagrant reload --provision

Accessing services from outside the virtual machine

It is possible to connect to some services running in the guest virtual machine directly from the host. Vagrant is set to automatically forward some ports.

Viewing Results

NISMOD includes some experimental jupyter notebooks to help access and view results.

First, start the notebook server in the background with the command:

vagrant ssh
cd /vagrant
jupyter notebook &> /dev/null &

Then navigate in a browser to localhost:8910

Database Connection

Some sector models use a database which runs within the virtual machine and is set up when the virtual machine is provisioned.

The virtual machine exposes the database on port 6543 on the host machine, so you should be able to connect with the following details:

key value
Host localhost
Database vagrant
User vagrant
Password vagrant
Port 6543

For example, on the command line, assuming the psql postgres command line client is installed on the host machine, you should be able to connect with:

psql -d vagrant -h localhost -U vagrant -p 6543
Security note

This database setup is intended for testing and development only and assumes that high-numbered ports are only accessible to the local user. Strong passwords and secure login methods should be used if the host machine is expected to be accessed by multiple users.

Development

General Guidelines

  • master branch is protected against push and force-push.
  • The master branch should be treated as live and is used for major releases only
  • All development occurs on the develop branch
  • New features are never added to an existing release branch

For further detail, see this article

Developing a new feature

  1. Create a new feature branch from the develop branch git checkout -b feature/<feature_name> develop
  2. Develop your feature
  3. Submit a pull request against the develop branch
  4. Merge the changes into the develop branch

Release Process

  1. Update the changelog under the heading Version X.Y.Z

  2. Check versions of models and data specified in provision/config.ini

  3. Now, create a branch from develop e.g. checkout -b release-x.y develop

  4. Tag this as a release candidate git tag -a vx.y.z-rc1

  5. Any bugfixes are committed to this branch and should be tagged with incremented release candidate tags e.g. git tag -a vx.y.z-rc2

  6. Once the release candidate is stable, submit a pull request to the master branch

  7. Merge the pull request and tag git tag -a vx.y.z

Fixing a Bug in a release

  1. Create a branch from master git checkout -b hotfix/<bug_name> master
  2. Fix the bug by committing to the branch
  3. Submit a pull request to master and tag the fix git tag -a vx.y.z incrementing the minor z digit

Testing

The tests in this repository are intended to test the integration of the various NISMOD sector models with smif, the simulation modelling integration framework. In outline, tests here should:

  • validate possible system-of-systems model configurations
  • run each sector model through smif with only scenario dependencies (not coupling with any other models)
    1. without unexpected error
    2. with the expected type contract of results
  • run combinations of sector models
    1. without unexpected error
    2. with the expected type contract of results
    3. (optionally) with regression tests against known-good modelled outputs

Continuous integration testing is currently available through a GitLab server based in the School of Geography in Oxford. You can access the project on OUCE GitLab and monitor jobs through the web interface.

To add the remote and push a branch for automated testing, run:

git remote add ouce https://gitlab.ouce.ox.ac.uk/NISMOD/nismod2.git
git checkout feature/branch
git push -u ouce

Gitlab-runner

There is a pool of gitlab-runner workers active that pick up and run jobs on each commit/push. A local gitlab-runner can be setup using the following configuration.

concurrent = 1
check_interval = 0

[[runners]]
  name = "ubuntu-nismod2"
  url = "https://gitlab.ouce.ox.ac.uk/"
  token = "12964bfcd398c966ebf128f1b5b34f"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "ubuntu:xenial"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

Acknowledgements

NISMOD was written and developed by researchers in the Infrastructure Transition Research Consortium as part of the EPSRC sponsored MISTRAL programme.