-
Notifications
You must be signed in to change notification settings - Fork 5
How to release
This guide gives instructions how to create a new data and code release for the open_eGo dataprocessing. Please follow these instructions and adjust them whenever they seem unvalid or insufficient. The relevant code to create the open_eGo data model can be found in a Github repository under https://github.com/openego/data_processing.
In the open_eGo project we follow the Git branching model defined by Vincent Driessen (https://nvie.com/posts/a-successful-git-branching-model/), which includes master and development (dev) branch as the two main branches. These main branches are complemented by feature, release and hotfix branches. To add new features or bugfixes to the repository pull requests should be created and assigned to one or two team members for a review before merging these changes into a suitable branch.
In order to initialize a new data and code release a release branch based on the current dev branch needs to be created. All bug fixes and other changes in the code which will be implemented during the release process need to be merged into this branch to be included in the release. For further guidance how to handle a release branch please refer to the Git branching model by Vincent Driessen. Additionally other developers and project member need to be informed about the release process to be started via redmine (and github).
The scenario_log used in the open_eGo dataprocessing includes the current version number of the release. Therefore the version number needs to be updated in all sql- and python-scripts included in the dataprocessing. Additionally a description of the current version has to be included in preprocessing/scenario_log/ego_pp_scenario.sql, including an update of the corresponding data set in the database. As the scenario_log also logs all runs of scripts which were executed during the debugging. To be able to distinguish between a debugging run and the final clean run two different version numbers should be assigned. The suffix ‘debug’ can be added as an distinction. In general the naming convention for release numbers differentiates between major and minor releases. A major release would be named like ‘v0.4.0’, whereas the version number of a minor release based on release ‘v0.4.0’ differs in the last digit (e.g. ‘v0.4.1’).
Github offers the option to create milestones and assign issues to these milestones. For every planned release a corresponding milestone should be created and relevant issues assigned. This helps the person responsible for the release to keep track of open issues and bugs which occur during the debugging process.
Prior to the debugging run the current data set in the model_draft needs to be versioned under a new version number for minor releases (see above and github issue #262). This ensures that intermediate changes in the data set are preserved. The debugging run deletes all data set created during prior dataprocessing runs from the model_draft. To identify and fix bugs in the dataprocessing a debugging run needs to be started by executing the script dataprocessing/eGo_data_processing.py from the release branch. In this run the versioning part of the dataprocessing, including the scripts ego_dp_versioning.sql and ego_dp_versioning_mviews.sql, needs to be excluded by commenting them out. The correct version number needs to be implemented in the scripts which are about to be executed. Whenever the executing python script stops due to a bug in the current script a fix needs to be added to the release branch by the person responsible for this script. Github issues should be used to assign this task to the relevant person. After adding an appropriate bug fix the executing python script can be started again from the script on were it stopped before. This is done until the whole dataprocessing went through. Please note that this process of debugging can take several days depending on the extent of changes implemented in the dev branch.
Before executing the REA part of the dataprocessing the number of grid districts needs to be examined, for example using this SQL query: SELECT COUNT(*) FROM model_draft.ego_grid_mv_griddistrict;
The so called clean run is executed after all bugs where removed in the debugging run which ensures a flawless clean run. Prior to the clean run the version numbers in the scripts have to be adjusted. In the clean run also the versioning part in the ego_dp_versioning.sql needs to be included to copy the data created to the versioned schemata of the database.
After the clean run the release branch need to be merged into the master. This merge shouldn‘t cause any trouble as no development happens on the master branch. As a next step a new release needs to be created on github including a description about new features and changes in this release. Additionally the updated master branch needs to be merged into the dev branch to include all recent changes here. Depending on the development which happened on the dev branch meanwhile this merge might be a bit bigger.