Five selfish reasons for working reproducibly

And so, my fellow scientists: Ask not what you can do for reproducibility — ask what reproducibility can do for you!

The following is a summary of a talk I gave in my institute and at the Gurdon in Cambridge. My job was to motivate why working reproducibly is a good strategy for ambitious scientists. Right after my talk, Gordon Brown (CRUK CI) and Stephen Eglen (Cambridge DAMTP)  presented tools and case studies of reproducible work.

All materials are on github and below are my slides, thanks to slideshare:

[Slides 2+3] I started with an intro to my lab, which has computational and experimental biologists working side by side. The message here: I tell my experimental students exactly the same things as my computational students. Reproducibility is not an issue only for bioinformaticians, but for all scientists.

[Slide 4] This famous cartoon had been used to advertise the talk. Complex stuff on the left, complex stuff on the right, in the middle “a miracle occurs”. This is exactly how it seems when you want to figure out how the authors got from a large and complex data set to a very dense paper with lots of busy figures. Without access to the data and the analysis code, a miracle occurred. And there should be no miracles in science.

A motivating example: The Duke breast cancer scandal

[Slides 5-12] I used the Duke breast cancer scandal as a striking example how non-reproducibility (and fraud) destroy scientific careers.

I don’t want to describe the full Duke story here again. You can watch Keith Baggerly give a great talk on videolectures or read the Baggerly & Coombes paper in the Annals of Applied Statistics, or even read about them in the New York Times. It’s not often that biostatisticians are featured in the NYT – this alone shows how big the Duke scandal was!

I have heard Keith’s talk several times. He is very entertaining and had us all laughing out loud many times. But halfway through the talk most people had fallen silent. They were thinking about their own projects and how they can avoid a disaster like this in their own work and their own group.

Reason Nr 0: Reproducibility is the right thing to do

[Slide 13] Here is one reason why you should work reproducibly that gets thrown around quite often:

Because it is the right thing to do!
Because the world would be a better place if everyone did it!
Because it is the foundation of science!
yaddah, yaddah, yaddah …

I called this ‘Reason Nr 0’ because I don’t think it carries that much weight.

Sure, it will appeal to altruistic and enthusiastic do-gooders who still hold on to their ideals and illusions of how science should be instead of how it really is.

But you, dear reader, you are not like these people.

You are like me: a cutthroat alpha-scientist who ruthlessly exploits every opportunity to have more publications, more impact factor, more money and more career. More, more, more – because in the real world that’s what science is all about.

And why is reproducibly good for winners like you and me? Read on, five reasons are coming:

1. Reproducibility helps to avoid disaster
2. Reproducibility makes it easier to write papers
3. Reproducibility helps arguing with reviewers
4. Reproducibility enables continuity of your work/in the lab
5. Reproducibility helps to build your reputation

Reason Nr 1: Reproducibility helps to avoid disaster

[Slides 14-19] First of all, if you get hit by a Duke-like disaster, that’s the end of your career.

And that’s not only true for the first and last authors of a paper, but for the middle authors, too. It will leave a stain on your CV.

So it is important to demonstrate that you, your students and postdocs, as well as your collaboration partners all acted in the most scientific way possible and any mistakes were honest and are easily corrected.

And even disasters much smaller than the Duke train wreck can be embarrassing. Here is an example from my own lab, where lack of reproducibility killed a project:

Our experimental collaboration partners had spent almost two years on successfully validating a pathway model we had generated computationally. We all were very pleased with the outcome and already started to throw around the names of glamorous journals to submit our paper to.

There was only one tiny problem: No matter how hard we tried, we could not reproduce our initial pathway model. We had many hypotheses on what had gone wrong: the data might have changed, or maybe the code, or maybe we just couldn’t remember the parameter settings of our method correctly.

We couldn’t figure out how it had happened and at the end of the day it didn’t matter, the result was just the same: without a reproducible model, the project was dead. Rest In Peace, beautiful pathway!

It is good we spotted the problem before we submitted the paper. Without a proof of how we got to the initial hypothesis we would have had a hard time to defend ourselves if someone suggested that we had made it up just to fit the validation experiments.

This experience showed me two things:

A project is more than a beautiful result. You need to protocol in detail how you got there.

Starting to work reproducibly early on will save you time later. In the project I described above we wasted years of our and our collaborators time by not being able to reproduce our own results.

Reason Nr 2: Reproducibility helps writing papers

[Slides 20-25] In our own work we strongly rely on R and dynamic tools like Sweave, knitR and Rmarkdown (I often use these terms synonymously) which are all very easily accessible through RStudio.

As an example I presented in my talk a knitR file I prepared as a supplementary file for a recent paper in PLOS Medicine.

Why is it useful to have well-documented code and data embedded in dynamic documents?

• It is easy to look up numbers and put them in manuscript
• You can be confident your figures and tables are up-to-date
• Numbers, results and tables automatically update when data change.
• It is engaging and more eyes can look over the analysis.
• Easier to spot mistakes.

Here is another example from my own work. In a different project, my collaboration partner and I were discussing why some survival results did not come out as expected. Because all the data and analysis code were available to use in a knitR file, we could just start to explore the data directly.

By simply generating a table of the variable describing tumor stage we spotted the problem: Instead of numbers 1 to 4, the variable contained entries like ‘XXX’, ‘Fred’ and ‘999’ which showed that whoever had prepared this data set had done a poor job in curating it.

Looking into the data ourselves was much quicker and more engaging than going to the postdoc working on the project and saying ‘figure this out for us’. It was key that the data and scripts were available to us in an easily accessible format. My collaborator and I are way too busy to spend too much time on low level data analysis and without the knitR file we would not have been able to contribute. But because we had very transparent data and code it cost us just five minutes to spot a mistake.

Reason Nr 3: Reproducibility helps arguing with reviewers

[Slides 26-27] When people moan about peer review, one of the complaints I hear most often is: the reviewers didn’t even read the paper and had no idea what we were really doing.

This starkly contrasts with my experience during the review process of our PLOS Medicine paper. Because we had made the data and code fully available and easily accessible, one of the reviewers directly tried out his ideas on our data and the discussion was about very specific questions of data analysis.

The reviewer was completely on board, the only thing left to discuss was the best way to analyze the data. Exactly how a constructive review should be. And it would have been impossible without a knitr file.

Reason Nr 4: Reproducibility enables continuity

[Slides 28-31] I would be surprised if you hadn’t heard the following things or said them yourself:

“I am so busy, I can’t remember all the details of all my projects”

“I did this analysis 6 months ago. Of course I can’t remember all the details any more …”

“My PI said I should continue the project of a previous postdoc. But that postdoc is long gone and hasn’t saved any scripts or data.”

Think about it, all of this can be solved by documenting data and code well and making them easily accessible.

In my own group I don’t even discuss “results” with students if they are not documented in a Sweave/knitr/Rmarkdown file. No proof of reproducibility, no result!

Reason Nr 5: Reproducibility helps to build your reputation

[Slides 32-37] For several papers we have made our data, code and analyses available as an Experiment Package on Bioconductor (e.g here). When I came up for tenure I cited all of them as research output of my lab.

Making your analyses available in this way will help you to build a reputation for being an honest and careful researcher. Should there ever be a problem with one of your papers, you will be in a very good position to defend yourself and show that you reported everything in good faith.

The recent paper “Promoting an open research culture” in Science summarizes eight standards and three levels of reproducibility guidelines. Using tools like R and knitr will make it likely that you comply easily with the highest level – and that again is good for your reputation.

Currently, there is a window of opportunity to be a trendsetter by working reproducibly. In a couple of years, everyone will do it – and ‘open science’ will just be called ‘science’ again.

Frequent excuses for not working reproducibly

[Slides 38-45] Have I convinced you? Maybe not. Here is a collection of responses I sometimes get to my insistence on reproducible research (as well as my answers to them):

“Mind your own business! I document my data the way I want!”

• Yes, please do! There are many ways to work reproducibly.

“Excel works just fine. I don’t need any fancy R or Python or whatever.”

• The tool you mention might work well if lots of manual curation are needed, but as soon as you do data analysis less clicking and more scripting are the way to go.

“Reproducibility sounds alright, but my code and data are spread over so many hard drives and directories that it would just be too much work to collect them all in one place.”

• Just think about what you just said. Your disorganization puts you and your project in grave danger.

“My field is very competitive and I can’t risk wasting time”

• And that is exactly WHY you should start working reproducibly early on, so you don’t waste time in the long run.

“We can always sort out the code and data after submission”

• My pathway example above shows the danger of this strategy. Also, preparing a manuscript for submission can take a long time and you might not even remember all the details of your analysis by the time you submit your results.

“It’s only the result that matters!”

• You are wrong.

“I’d rather do real science than tidy up my data”

• If you don’t work reproducibly you are not doing science at all.

When do you need to worry about reproducibility?

[Slides 47+48]

• Before you start the project, because you might have to learn tools like R or git.
• While you do the analysis, because if you wait too long you might lose a lot of time trying to remember what you did 2 months ago.
• When you write the paper, because you want your numbers, tables and figures to be up-to-date.
• When you co-author a paper, because you don’t want to be the victim of a Duke disaster and want to make sure that the analyses presented in a paper with your name on are sound.
• When you review a paper, because you can’t judge the results if you don’t know how the authors got there.

In short: ALWAYS!

[Slide 49] Working reproducibly is part of what I call scientific soft skills. Not the ‘hard’ molecular biology or algorithmic skills that produce your results, but ‘soft’ skills of how to organise, manage and communicate your data, code and results in a transparent way.

Who is Reproducibility important for?

[Slide 50]

• Students and postdocs, because you are the people who need to learn the tools and actually apply them in your daily work.
• PIs, group leaders, professors, team leaders, because if you are short-sighted and don’t create a “culture of reproducibility” in your lab, then your students and postdocs will see no advantage in working reproducibly and you will miss out on the large scientific pay-offs reproducibility offers in the long run.

Florian

Postscript 1: I have highlighted the supplement I wrote for a recent PLOS Medicine paper. I wrote it, and the first and co-last authors checked it and contributed to it. Still, when I downloaded the .Rnw from the PLOS webpage and tried to run it again on my laptop it failed to compile. Because of a silly mistake: I had linked to an external BibTeX file instead of copying this info into the .Rnw. Sigh! It just shows how hard reproducibility is …

Postscript 2: I  would love writing (parts of) this post up as an Opinion piece in some nice journal. Calling all journal editors, when you read this, contact me – I’ll make you a good price.