Manuscript review and the alleged reproducibility crisis

Feb 22 2018 Published by under ReplicationCrisis

If you believe there is a “replication and reproducibility crisis” in science, you have limited proposal options.

Multiple independent replications of each phenomenon in a paper. Not n-up, but triplicate or more full power repeats.

Are you demanding this? Are you doing this in your own research?

Or, we create some way to encourage and enhance direct replications and simple tests of generalization from other labs or in subsequent papers.

I favor this one.

As it happens,I have had multiple incidences in the past few years which address this. As both an author and as a reviewer.

Editors and reviewers for decidedly modest JIF journals are overtly and explicitly saying replications and simple tests of generalization of a finding should not be published.

I can't stress this enough. We're not talking "this isn't cool enough for this 2.5 JIF journal". These are opinions that such things do not merit publication at all.

Findings that fail to replicate a prior finding (that is actually poorly supported) *simultaneously* take heat for not getting the same result.

Direct replication is too incremental and refutation / modification is too.....doubtful?

As my longer term Readers know, I tend to think this is just the way science works. If you keep at it your manuscript will find a home eventually. It is a PIA but it is not damning of the entire enterprise.

But if there is any validity to the reproducibility claims and you keep braying on about it...I want to know a lot more about how your reviewing behavior matches your fine talk.

I'm also looking forward to NIH grant review in about 3-4 years. We should be expecting the experienced PI to have some specific and concrete examples about their rigor.

Even if it is "this preprint shows our replication attempts, even if idiotic reviewers prevented them from being published".

7 responses so far

  • David says:

    The only way I've seen this routinely accepted is when multiple labs got together and did a round robin. Typically it has been for new equipment where they were trying to show that the equipment produced the same results across multiple labs. My lab once did it for a method to show that different equipment produced equivalent results when following the method.

  • Morgan Price says:

    PLoS ONE and some of the other mega journals should be willing to take these kinds of papers. (Although if many scientists really think that these papers should not be published anywhere, I’m not sure if this would actually work in practice.)

  • Microscientist says:

    I was just a reviewer for Scientific Reports and had just this issue. The paper in question had no novel results at all, but did do a nice job replicating results in a minorly different cell line. Is it worthy of publication? I ended up kicking it to the editor, as this is where the line of not evaluating significance gets extremely hazy. My opinion was they they could have easily done some novel experiments to add to the data they had. The editor felt they had done enough.

  • drugmonkey says:

    Why does your opinion on what they could “easily do” come into it? Why are you not merely reviewing merit of what was presented?

  • DNAman says:

    I've noticed this isn't usually issue in areas that rely on quantitative measurements.

    If you measure a parameter to be 110, (95% CI 100-120) and then another lab measures the same parameter more accurately as 115 (95% CI 113-117), that's often viewed to be a significant advance.

    In areas that rely on binary outcomes, perhaps more focus should be put on the level of certainty. One lab finds that X causes Y, with a certainty at the 95% level. Then when another lab finds that X causes Y with a certainty at the 99% level, that should be viewed as a valuable contribution, and not ignored.

  • Stephen says:

    Every paper should have an online comments section where one can leave comments under ones real name e.g. couldnt get this to work, works well if X is done, bad stats etc. If that was done then bad papers would quickly get flagged

  • qaz says:

    If one lab finds that X causes Y with a certainty at 95% and another lab finds that X causes Y with a certainty at 95%, then we have learned something - we now have increased our certainty that X causes Y. That's an improvement in the literature. We need to accept that no paper is ever the final say in a result. Papers are always only a step on the journey.

    PLoS ONE started with the mantra that they would publish anything as long as it was experimentally correctly done. They explicitly said that they would *not* take into account any questions of "novelty" or "importance". That is a good thing.

Leave a Reply