Sustaining reproducibility

Our paper on tumor evolution in ovarian cancer (see here) came with a nice knitR file to reproduce the survival results, which I used as an example in my recent talk about reproducibility (see here).

I thought that was a nice test scenario to see if I could reproduce the results I got more than a year ago.

How reproducible am I?

Downloading the Rnw from the journal webpage (link) was easy, but -of course- it didn’t run through smoothly.

LaTeX failed and there were several R error messages.

The joys and frustrations of reproducibility

First of all, I had linked to a BibTeX file instead of just copying the bibliography in to the Rnw as I should have done.

Second, I ran into problems with the survival analysis, because one of the packages had changed.

rms::survplot() used to allow plotting a survfit object through survplot.survfit() function. However, this function has been deprecated as of version 4.2.

Luckily I found an easy workaround, just use npsurv() instead of survfit().

The updated Rnw is here on my webpage:

Together with a PDF so you can see what the output should look like.

Click to access journal.pmed.1001789.s020-v2.pdf

Take-home message for me: Even with a knitR file I did myself, reproducibility is not a one-click thing.

To make reproducibility sustainable I would have to check all published analysis scripts in regular intervals (e.g. once every year or every 6 months). Am I prepared to do this? And for how long?


2 thoughts on “Sustaining reproducibility

  1. Implementing the knitr document as a package vignette could help. While it will not fix the problem with changing dependencies, a package provides a well defined framework to automate building, checking and versioning using Travis on github or submitting the (data/experiment) package to Bioconductor.

    You could also distribute the document or package and the complete (fixed) environment as a docker container.

    But you are absolutely right: reproducibility and maintaining it over time is neither trivial nor free.


  2. We posted Rnw files to produce all the plots in our DESeq2 paper, and R scripts for long running analyses. But I did something which I wish I could go back and change. For some simulations, I used something like this pseudocode:

    for (i in 1:100) {
    data <- makeData()
    for (a in algorithms) {
    runAlgorithm(data, a)

    At least one of the algorithms we evaluated used random sampling, so then if the packages change what they do across versions, they might be changing how the state is incremented in the internal for-loop and affecting other algorithms' output. A better way to keep this contained would have been to also set the seed before each algorithm call, to limit the impact of version changes.

    All that said, I think the proper way going forward is to bundle up all the necessary packages with something like Docker.


You gotta talk to me!

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s