Duty Calls, Science

Replication of results in statistics and biology


Do differences between biology and statistics explain some of our diverging attitudes regarding criticism and replication of scientific claims?asks Andrew Gelman on his blog.

I was not very impressed with the post or the comments it received. Here is what I posted as a response:

The ‘Us vs Them’, ‘Statisticians vs Biologists’, ‘Level Playing Field vs Toxic Politics’, ‘Reproducibility And Truth vs Impatient Career Optimizers’ is just lazy thinking. Why not read Dr Bissell’s paper and discuss her arguments?

1) You would find that she speaks a lot about reproducing results – in her own lab and collaborating with others. She is certainly not an enemy of reproducibility.

2) And her arguments are not half bad. For example (I quote from her paper): “[W]ho will evaluate the evaluators?” I can fail to reproduce YOUR results just by being sloppy and lazy. Does that make you a bad scientist?

3) “[I]t is sometimes much easier not to replicate than to replicate studies, because the techniques and reagents are sophisticated, time-consuming and difficult to master.”

In my field, computational biology, the statistical/computational/data analysis part of a paper can often attempted to be reproduced by an undergrad student with knowledge of R and the right set of Bioconductor packages. Statistics is the easy bit, because it can be communicated in equations and code – the actual experiments are often much harder, especially in Dr Bissell’s field. If it takes years to hone 3D cell culturing skills then naturally only a small number of people are out there who could potentially replicate results and most others will fail.

4) Dr Bissell is indeed not a good ‘representative of biologists as a profession’, but for different reasons than you might think. For decades she was an outsider and was being ignored by the biology establishment (the genome people). What they thought of her was (I quote from her life story linked to from her webpage): ‘Oh, it’s cute, there is this little excitable Persian woman over there screaming about whatever.’

If she is successful now, it is after years and decades of her and her work being scrutinized and by now I’d assume she knows a thing or two about reproducibility. Some of the comments read like: ‘Oh, she only says that because she is an elitist career scientists who is more interested in publishing papers than truth’. Dismissing her views in this way is lazy.

Needless to say, I found Mina Bissell’s article ‘Reproducibility: The risks of the replication drive‘ insightful and well argued. And I’m the guy who bullies his students and postdocs into writing Sweave files to reproduce every single number in a paper.

Florian

Update: I later added a second comment:

Andrew

all three editions of BDA stare at me reproachfully while I write this. This ‘little blog’ is a widely read megaphone of its own. So it’s good to have this discussion here:

1) In the title of your post you claim ‘diverging attitudes’ and further imply that statisticians are pro reproducibility while biologists are against it. From working in both fields I can say this is not true.

My own work has often been constrained by badly documented statistical analyses, unavailability of code and badly discussed parameter choices. At the same time all the biologists I work with routinely replicate other people’s work and benchmark new methods. My own wetlab, as small as it is, does the same. I see no evidence whatsoever that biologists lack awareness of replication (as some commentators believe) or need to be lectured on it.

2) Basing the claim of ‘diverging attitudes’ on Mina Bissell’s paper would be wrong, because she actually does not argue against reproducing results. Her article is about risks and limitations – this is different from saying ‘it’s a waste of time, don’t do it’.

The first and third paragraph of her paper describe how much effort her own group puts into reproducing their previous results (I assume it’s a way to train newcomers to the lab). In the last paragraph she even proposes peer-reviewed reports to settle reproducibility problems. Nowhere does she say reproducibility was a bad thing. To conclude like commentators here do that she proposes to ‘skip verification’ is wrong.

3) No one disagrees with your statement: “it’s in everyone’s interest if replication can be done as fast and reliably as possible”. I certainly don’t; all my big papers have a data package on Bioconductor for exactly this reason. And if Mina Bissell disagrees with this statement she did not write it in her article.

The problem is your general claim: “[I]f a published finding cannot be _easily_ replicated, this is at best a failure of communication” (my emphasis).

Depending on what you mean by ‘easily’, your claim might be true in a field where all relevant information can be communicated in writng (equations and code) and where execution is cheap and fast.

Biologists are not counting peas anymore. In modern experimental sciences replication is much harder (no one says ‘impossible’ or ‘do not try’ or ‘it’s not important’) because reproducing complicated experiments in highly specialized fields needs lots of experience, lots of training, lots of technical skills, lots of time, lots of money, lots of reagents and lots of other resources. No matter how good your communication skills are, these are real limiting factors.

This is different from statistics, where I see my friends having careers with nothing more than a whiteboard, a laptop and sometimes a cluster, ie much less specialized commodities. Proposing to use ‘multiple replicators’ (as one commentator did) severely underestimates the complexity of many biological replication tasks.

4) In summary, the technical and training requirements in highly-specialized fields provide limitations for reproducibility that need to be acknowledged. Often replication can not be done easily, fast or reliably. This is not due to the lack of awareness of a group of scientists, but a feature of the work they do.

(That said, many of the technological limitations are not permanent and what was impossible to easily reproduce 10 years ago can suddenly be done over night – sequencing a human genome for example.)

So while there is no evidence that the attitudes regarding criticism and replication of scientific claims are any different in biology compared to statistics, the differences in the scientific work being done (not necessarily the social or economic structure) lead to constraints on how much reproducibility is easily possible.

Florian

Advertisements

You gotta talk to me!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s