Skip to content

Adding Model Tests to Jenkins

Jump to bottom
sambles edited this page Jun 25, 2020 · 13 revisions

Adding Model Tests to Jenkins

Each model connected to the Jenkins CI/CD server has the option to add tests using example sets of OED input files and an analysis_settings.json. This document will focus on how to extend these using the PiWind model as an example.

Tests are run on docker-compose of the full running platform using the test script api_integration.py

Test configuration file

All available test case is set in a config.ini file stored within the model repository, for PiWind this is tests/test-config.ini.

Default section

In general, this section should not need to be edited, with the possible exception of TEST_MODEL which selects which model section to pickup settings and tests cases from.

[default]
TEST_OUTPUT_DIR=output    # Sets the download path for fetching output TAR files (relative to the conf.ini location)
TEST_LOG_DIR='./'         # Sets the log storage location, (relative to the config.ini location)
TEST_MODEL=PiWind         # Sets which model sub-section to use from the test confing.ini 
CLEAN_UP=false            # Send cleanup `DELETE` calls to the API after a test has completed

Model and tests sub-sections

Each test config file can store cases for multiple models and need one section for the model parameters and multiple sub-sections; one per test. To select a default model for testing set the TEST_MODEL under the defaults section. For example if we set TEST_MODEL=my_model then we need to define the following. Where each test section is labelled my_model.<test_name>.

[my_model]
  <model params>

[my_model.first_test]
  <input file paths for 1>

  ... 

[my_model.Nth_test]
  <input file paths for n>

Model parameters

Models in the Oasis API are held under the /{version}​/models​/{id}​/ section where {id} is an integer reference. Each model is uniquely identified using a triple of:

{
  "supplier_id": "string",
  "model_id": "string",
  "version_id": "string"
}

The test script needs to know these three values to match the model under test to a running worker container. These are set in the jenkins groovy file and needs to be matched otherwise the test will timeout and fail.

[piwind]
MODEL_ID=PiWind                            # Needs to match the API `model_id`
VERSION_ID=1                               # Needs to match the API `version_id`
SUPPLIER_ID=OasisLMF                       # Needs to match the API `supplier_id`
RUN_TEST_CASES=0_case 1_case control_set   # Set of test names to run by default
EXPECTED_OUTPUT_DIR=ci/expected            # Location of directory holding expected results, (relative to the config.ini location)

Adding a new test case

Each test sub-section [<MODEL>.<TEST_NAME>] has 5 variables which hold relative paths to test input files. One analyses settings for the run, and 4 OED input files. The LOC_FILE and SETTINGS_RUN files are required, while ACC_FILE, INF_FILE and SCP_FILE are optional depending on if the test analysis requires insured loss or reinsurance.

lets assume we want to add a test called new_test.

  1. Add the test name to the RUN_TEST_CASES section under my_model if we want it to be tested by default. Note that this is a space delimited string.
[my_model]
  ...
RUN_TEST_CASES='first_test new_test' 
EXPECTED_OUTPUT_DIR=./path_to/expected
  1. Create the test sub-section in the tests/test-config.ini file.
[my_model.new_test]
SETTINGS_RUN=./path_to/analysis_settings.json
LOC_FILE=./path_to/Location_file.csv
ACC_FILE=./path_to/Account_file.csv
INF_FILE=./path_to/ReinsInfo_file.csv
SCP_FILE=./path_to/ReinsScope_file.csv
  1. Add the new input files to the model's repository. All file paths are relative to the test-config.ini file location, so we need to create a directory called path_to under OasisPiWind/tests/ and add the following files.
OasisPiWind/tests/path_to/analysis_settings.json
OasisPiWind/tests/path_to/Location_file.csv
OasisPiWind/tests/path_to/Account_file.csv
OasisPiWind/tests/path_to/ReinsInfo_file.csv
OasisPiWind/tests/path_to/ReinsScope_file.csv
  1. Adding expected results, in the my_model parameters section, we've set EXPECTED_OUTPUT_DIR to:
[my_model]
  ...
EXPECTED_OUTPUT_DIR=./path_to/expected

Create the directory OasisPiWind/tests/path_to/expected. Under this are directories which match to a test name, these hold the generated inputs files and results for verification.

OasisPiWind/tests/path_to/expected/<test_name>/input - for the generated oasis files

OasisPiWind/tests/path_to/expected/<test_name>/output - for the ktools output files

Example: control_set

The model needs to be run with either the MDK or API and these sets of files created for thenew_test example. Note however that these are optional and if a test case is missing the expected output directories the files verification step will be skipped in CI.

Clone this wiki locally