“How reproducible is cancer biology?” stood in bold letters on a poster I had designed to advertise a talk by Tim Errington, one of the leaders of the Reproducibility Project: Cancer Biology (RP:CB), in Cambridge a few weeks ago. And I had told everyone “Come and learn who’s good, who’s bad and who’s ugly in cancer research!”
The RP:CB results are collected at eLife and the splash they made was big enough to be covered by Nature and Science. So it was great to finally meet somebody leading this project to learn first-hand which ideas are guiding their work.
Tim’s talk had the rather technical title “Improving Openness and Reproducibility of Scientific Research.” You can easily see why I felt the need to spice things up. To get a lecture theatre full of people you need to promise them blood, not balanced and nuanced views (limitations I luckily have never been accused of myself).
With all the effort I had put into advertising the talk, the people at my institute knew what to expect: A witchhunt by a posse of replication vigilantes, who abuse money diverted from real science to name and shame the actually successfull researchers! Hang them higher! Yihaah!
When Tim arrived, he took one look at the way I had advertised the talk, gently shook his head, and said “Well, you can of course do this, but I wouldn’t. It’s not really important which study reproduces and which doesn’t.”
Wasn’t that what the Reproducibility Project was all about?
Uh oh .. how disappointing. It seems I was wrong. Apparently there is more to it.
Here, have a look at the talk yourself. The kind people organising the Cambridge Open Access week 2017 filmed it and put it up on Youtube (Thank you, Danny Kingsley!).
There were several take-home messages for me:
First of all, the RP:CB is not very expensive. The total budget is just 2 million Dollar. For comparision, my own group gets about £500k from CRUK each year, so we would burn through the total RP:CB budget in less than 3 years.
So even if it is not a big investment, is it necessary? Couldn’t my group, well funded as it is, not just do all these replication experiments as part of our normal research? Yes, we could and we do. All our projects, both experimental and computational, involve a phase where we replicate known results. Always a good sanity check to make sure the methods work and we know what we are talking about. And I am sure most other labs do the same. So why do we need the RP:CB at all?
Tim explained that the RP:CB offers insights that go beyond a simple Yes/No answer and that go beyond what is achievable by intra-lab reproducibility efforts. Because the RP:CB reproducibility efforts are much more transparent and much better controlled than any ‘Let me quickly check if that is true’ experiments that most of us are doing. I mean, who ever writes a pre-registered report for themselves? I know my lab doesn’t.
The RP:CB is about answering questions like: What roadblocks to reproducibility exist? What went wrong (if something went wrong)? What actually does it mean ‘to replicate the previous result’? When are two results ‘the same’? The RP:CB results will address all this questions (and they started discussing them for example here). Answering these questions is meta-science and thus outside the scope of most of us do during our normal science.
At the same time, Tim was not very interested in which study replicates and which doesn’t. In particular, he was sceptical of turning the nuances of replication into YES/NO tables like the one Nature has produced. This nuance also extended to the the RP:CB’s own approach, which is based on pre-registered reports, and Tim freely admitted that one of the outcomes so far is that preregistered reports might be too restrictive. Variations should be allowed as long as they are well documented.
Finally, in the Q&A Tim gave a very good answer when challenged by what must be now a classical question: Some people are just better at science than others; the same way some can play the violin better than others. Some have Golden Hands, some are crap. Isn’t this fact confounding all reproducibility efforts?. Tim said: If you can’t play the violin, it is important to find out why: Did you ever take lessons? What else could you do wrong? Only through transparency science can move beyond Golden Hands.