2012-08-03-habits-of-open-scientist-2.html

---
layout: post
name: habits-of-open-scientist-2
title: ! '7 Habits of the Open Scientist: #2 -- Reproducible Research'
time: 2012-08-03 19:49:00.000000000 +03:00
tags: open-science reproducible-research research science scientific-computing software-distribution
categories: community
---
Note: This post was originally written by <a href="http://www.davidketcheson.info/2012/08/03/habits-of-open-scientist-2.html" target="_blank">David Ketcheson</a>.

<p><span style="font-family: Palatino;"><span style="font-size: 18px;">Note: this post is part of <a href="http://scienceinthesands.blogspot.co.uk/2012/07/7-habits-of-open-scientist.html">a series on habits of the open scientist</a>.  Here I discuss the second habit, <strong>reproducible research</strong>.  The previous post was on <a href="http://scienceinthesands.blogspot.co.uk/2012/08/7-habits-of-open-scientist-1-open.html"><strong>open scientific publishing</strong></a>.</span></span></p><p style="font-size: 18px;"><span style="font-size: 24px;"><strong><span style="font-family: Helvetica;"><span style="font-size: 24px;"><span style="font-family: Palatino;"><span style="font-size: 24px;">Reproducible research</span></span></span></span></strong></span></p><p><span>Reproducibility is part of the definition of science: if the results of your experiment cannot be replicated by different people in a different location, then you're not doing science.  Far from being a mere philosophic concern, reproducible research has been a key issue in prominent controversies like <a href="http://en.wikipedia.org/wiki/Climatic_Research_Unit_email_controversy">climategate</a> and <a href="http://magazine.amstat.org/blog/2011/01/01/scipolicyjan11/">cancer research clinical trials</a>.</span></p><p><span>Especially disconcerting is the typical irreproducibility of scientific work involving computer code:</span></p><blockquote><p><span>“Computational science is facing a credibility crisis: it’s impossible to verify most of the computational results presented at conferences and in papers today.” (LeVeque, Mitchell, Stodden, <em>CiSE </em>2012)</span></p></blockquote><p>Frankly, I used to find that I was often unable to reproduce <em>my own</em> computational results after a few months, because I had not maintained sufficiently detailed notes about my code and my computing environment.</p><p><strong>The open scientist ensures that the entire <em>research compendium</em> -- including not only the paper but the data, source code, parameters, post-processing, and computing environment -- is made freely available, preferably in a way that facilitates its reuse by others.</strong></p><p>I won't spend more time motivating reproducible research, since <a href="http://www.bloomberg.com/news/2012-01-10/scientists-share-secrets-or-lose-funding-stodden-and-arbesman.html">others have done that</a> <a href="http://biostatistics.oxfordjournals.org/content/11/3/385.full">much better than I could</a>.  Instead, let me focus on the relatively easy first steps you can take to make your research more reproducible.</p><p><span style="font-size: 24px;"><strong>The bare minimum: publish your code and data</strong></span></p><p><span style="font-family: Palatino; font-size: 18px;">If you wish to set an example of good reproducible computational research practices, I have good news for you: the bar is very low at the moment.  The reason why "it's impossible to verify most of the computational results" is that </span><span style="font-family: Palatino; font-size: 18px;"><em>most researchers don't release their code and data.</em></span><span style="font-family: Palatino; font-size: 18px;">  The first step toward working reproducibly is simply to put the code and data that is used in your published research out in the open.</span></p><p><span style="font-family: Palatino; font-size: 18px;">If you don't want to release your code to the public, please read about <a href="http://jarrodmillman.com/talks/siam2011/ms148/leveque.pdf">why you should</a> and <a href="http://www.nature.com/news/2010/101013/full/467753a.html">why you can</a>.  Once you're convinced, go endorse the <a href="http://sciencecodemanifesto.org/">Science Code Manifesto</a>.</span></p><p>Releasing your code and data can be as simple as posting a tarball on your website with a reference to the paper it pertains to.  Or you may wish to start putting all your code out in the open on <a href="http://bitbucket.org">Bitbucket</a> or <a href="http://github.com">Github</a>, <a href="http://github.com/ketch/">like I do</a>.  I don't claim that these are the <em>best</em> solutions possible, but they are a big step forward from keeping everything on your own hard drive.</p><p>When you release your code and data, it is important to use an appropriate license.  <a href="http://stodden.net">Victoria Stodden</a>, a leader in the reproducible research movement, recommends the use of a permissive license like <a href="http://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_.28.22New_BSD_License.22_or_.22Modified_BSD_License.22.29">modified BSD</a> for code and <a href="http://sciencecommons.org/resources/faq/database-protocol/">Science Commons Database Protocol</a> for data.  Together with the <a href="http://creativecommons.org/licenses/by/2.0/">Creative Commons BY</a> license for media (that I mentioned in <a href="http://scienceinthesands.blogspot.co.uk/2012/08/7-habits-of-open-scientist-1-open.html">my last post</a>), these comprise the <a href="http://www.ijclp.net/files/ijclp_web-doc_1-13-2009.pdf"><em>Reproducible Research Standard</em></a>, a convenient amalgamation of licenses for open science.</p><p>Be sure to include a mention of reproducibility in your paper, along with links to the code and data.  If you release your work under the RRS, I suggest using <a href="http://dx.doi.org/10.1109/MCSE.2009.19">this citation</a>.</p><p><span style="font-size: 24px;"><strong>Real benefits</strong></span></p><p><span>The open scientist may adopt reproducible research practices for philosophical reasons, but he soon finds that they bring more direct benefits.  Because he writes code and prepares data with the expectation that it will be seen by others, the open scientist finds it much easier for himself, students, and colleagues to build on past work.  New collaborations are formed when others discover his work through openly released code and data.  And (as in the case of <a href="http://numerics.kaust.edu.sa/papers/pyclaw-sisc/pyclaw-sisc.html">this paper</a>, for example) the code itself may be the main subject of publications in <a href="http://scicomp.stackexchange.com/questions/660/venues-for-publishing-papers-that-emphasize-software">journals that have come to recognize the importance</a> of scientific software.</span></p><p><span style="font-size: 24px;"><strong>Taking it further</strong></span></p><div>Like free and open scientific publishing, reproducible research has become a very large movement, and only a book could hope to cover it all.  Here I've merely distilled some basic practical suggestions.</div><p>Openly releasing code and data is only the first step.  Open scientists may wish to adopt tools that track code provenance and ensure a fully reproducible workflow, such as</p><ul><li><a href="http://www.reproducibility.org/wiki/Main_Page">Madagascar</a></li><li><a href="http://www.vistrails.org">VisTrails</a></li><li><a href="http://yihui.name/knitr/">knitr</a></li><li><a href="http://neuralensemble.org/trac/sumatra">Sumatra</a></li><li><a href="http://vcr.stanford.edu/">VCR</a></li></ul><div>Next up: Habit #3 -- <a href="http://scienceinthesands.blogspot.co.uk/2012/08/7-habits-of-open-scientist-3-pre.html"><strong>Pre-publication dissemination of research</strong></a>.</div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/4660225022390848760-9151334296275912840?l=scienceinthesands.blogspot.com' alt='' /></div>