WMS-benchmark reports the code used to benchmark workflow management systems (WMS) on a bioinformatics use case, LodSeq.
The following WMS were tested:
Please cite WMS-benchmark using this citation:
E. Larsonneur, J. Mercier, N. Wiart, E. Le Floch, O. Delhomme and V. Meyer, "Evaluating Workflow Management Systems: A Bioinformatics Use Case," 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Madrid, Spain, 2018, pp. 2773-2775. doi: 10.1109/BIBM.2018.8621141.
A supplementary table and a corrected figure are available in the figures/ directory.
See scripts/all.sh for details.
The full dataset is available here or here.
We proposed ten metrics for evaluating the efficiency of workflow management systems,
they were extracted from pidstat, fg_sar and benchme output files. fg_sar uses sar, and benchme uses time to compute metrics. fg_sar and benchme are a part of fgtools toolkit and can be found in the fgtools/ directory. fg_sar requires sysstat<=11.5.2.
To know how to run the workflow management systems with a toy dataset, please see the .gitlab-ci.yml file and its execution log.
Elise Larsonneur, Centre National de Recherche en Génomique Humaine, CEA, Evry, France, [email protected]
Edith Le Floch, Centre National de Recherche en Génomique Humaine, CEA, Evry, France, [email protected]
Jonathan Mercier, Centre National de Recherche en Génomique Humaine, CEA, Evry, France, [email protected]
Nicolas Wiart, Centre National de Recherche en Génomique Humaine, CEA, Evry, France, [email protected]
WMS-benchmark is released under the terms of the CeCILL license, a free software license agreement adapted to both international and French legal matters that is fully compatible with the GNU GPL, GNU Affero GPL and/or EUPL license.
For further details see LICENSE file or check out http://www.cecill.info/.