Replication costs money

I ran across a curious finding in a very Glamourous publication. Being that it was in a CNS journal, the behavior sucked. The data failed to back up the central claim about that behavior*. Which was kind of central to the actual scientific advance of the entire work.

So I contemplated an initial, very limited check on the behavior. A replication of the converging sort.

It's going to cost me about $15K to do it.

If it turns out negative, then where am I? Where am I going to publish a one figure tut-tut negative that flies in the face of a result published in CNS?

If it turns out positive, this is almost worse. It's a "yeah we already knew that from this CNS paper, dumbass" rejection waiting to happen.

Either way, if I expect to be able to publish in even a dump journal I'm gong to need to throw some more money at the topic. I'd say at least $50K.

At least.

Spent from grants that are not really related to this topic in any direct way.

If the NIH is serious about the alleged replication problem then it needs to be serious about the costs and risks involved.
*a typical problem with CNS pubs that involve behavioral studies.

35 responses so far

  • dr24hours says:

    Do you have "slush funds" that can be used this way, so you're not charging irrelevant work to a grant? At the school where I *almost* got my Asst. Prof. gig (approved by faculty, took too long, took job at MECMC), they gave investigators unrestricted use of a %age of indirects from their grants for such projects.

  • drugmonkey says:

    No, I don't.

  • odyssey says:

    Such slush funds are not uncommon, but the %age of indirects returned is typically pretty low. So coming up with $50k is far from trivial. Also, IME, places that provide slush funds don't provide other things, so the funds are often needed for other purposes.

  • Dave says:

    Don't bother. Write a letter commenting on the paper, or just let some other mug do the work. What about your student or post-doc that does the work? I'm sure they will be thrilled by this thorough waste of time. For all the talk about replication, journals still could not care less.

  • Grumble says:

    Do a cheap one-off pilot study and if the preliminary results don't replicate the CNS finding, write a whole grant around it. If the politics on the study section are right (e.g., a lot of behaviorists are on it and they all can't stand the crappy behavior in CNS papers, especially from that particular lab), you might stand a good chance.

  • anonymous postdoc says:

    Grumble advocates a reasonable path. I am sick of shitty behavior propping up some glamorously bullshit " this molecule is the key to braining". No wonder we have made so few advancements in mental health, our proofs of behavioral principle have to clear such a low bar that the findings are meaningless.

  • qaz says:

    The bigger issue, of course, is why the paper wasn't reviewed by someone with the appropriate behavioral expertise to say "that's bogus". They make behavioral papers that have some molecular stuff in it go through hoops to prove the molecular stuff is doing what it's supposed to be.

    Instead of you spending $50k on replication, they should have spent their $50k (probably a lot less since they presumably had all the pieces in place) doing the right controls in the first place before they got GlamourPub'd.

  • DrugMonkey says:

    The behavior is crappy because the Emperor is in fact stark nekkid.

  • DrugMonkey says:

    I should qualify that. The behavior is in and of itself not horrible. It just doesn't support the scientific claim being made. Backing off that claim would, presumably, made it less Glam-worthy. It could have been better tested, which is why I'm thinking about a study based on this. A "one-off" pilot that is much cheaper than the range I am contemplating would be nearly worthless...this stuff costs.

  • dsks says:

    "If the politics on the study section are right (e.g., a lot of behaviorists are on it and they all can't stand the crappy behavior in CNS papers, especially from that particular lab), you might stand a good chance."

    Even if the preliminary data simply suggests that the previous study connecting this molecule with that behavior was wrong? Wouldn't this just provoke the boilerplate response that the proposed work does not sufficiently advance the field (even though taking the field a step back from the wrong direction should probably be regarded as an advance, but there...)?

  • drugmonkey says:

    Pretty much, dsks.

  • Travis says:

    In my area of physiology (admittedly quite different from yours) this is where I would consider a journal like PLOS ONE or one of the BMC OA journals as an ideal spot, whether the finding is positive or negative. It means tacking on an extra $2500 or so for publication fees, but that's still a lot cheaper than the $50k option. But, in my area those journals have pretty reasonable IFs, I'm assuming that's not the case for you.

  • toto says:

    Isn't that what Letters to the Editor are there for? Assuming the publication in question is sufficiently recent, of course.

    Otherwise, an email to relevant mailing lists might help test the waters.

  • Isn't that what Letters to the Editor are there for?


  • Grumble says:

    "Wouldn't this just provoke the boilerplate response that the proposed work does not sufficiently advance the field "

    That's where DM's genius as a grantwriter would have to come in. You can't base a grant *entirely* on saying "Dr. BSD and his crap behavior are all wrong." You have to say, "A major hypothesis has recently been proposed by BSD's finding that X. However, evidence consisting of A, B and C suggest an alternative hypothesis, Y. To test this, we will do [repeat of behavioral study in glam journal, among several other solid experiments to test whether DM' hypothesis or BSD's hypothesis is correct]."

    The idea is to insinuate that the behavior in the glam journal didn't support the main hypothesis, but don't make that the whole focus of the grant. Instead you say it raises an intriguing set of possibilities and you're going to find out which one is correct. That's moving the field forward.

  • drugmonkey says:

    True, Grumble. If I decided that I had some interest and the preliminary experiment came out reasonably interpretable....yeah, there are some legs here. I mean, it was a Glamour result after all. If I thought I was on to some necessary preliminary data to support an interesting future grant application, then the calculus for spending the cash gets a lot easier.

    But I'm trying to point out something about the current fervor for "replication". It is not at all cheap to meaningfully and productively replicate a figure or triangulate on a finding to test generality. If NIH-dom is going to call for replication, they need to figure out how to pay for it. These experiments cost real cash and will come out of grant productivity.

    With respect to Dave's comment, I'm trying to account the costs for this replication without assuming "free" trainee labor. Putting some poor schmuck in your lab on something is a good way to delude yourself about the cost of the replication. Particularly if they are paid for by a training grant or fellowship that isn't taking a bite out of your research grant.

  • Trainees who have their own fellowships get "put on" shitte in your labbe???? That's fucken fucked uppe.

  • Ola says:

    Cutting to the bone of this issue, the publication system we have (which is pretty fucked up to begin with) is totally unprepared for an onslaught of replication as envisaged by NIH. This is partly blamed on the journals themselves, by there's also an element of peer reviewer reeducation needed.

  • drugmonkey says:

    Didn't you get the memo? Francis talked with publishers so everything is coolio.

  • Not that this solves the issue of how to pay for the work, or who would do it, but there exist journals like The Journal of Articles in Support of the Null Hypothesis and the Journal of Negative Results in Biomedicine (BMC open access) which would solve *where* to publish the results. Yes, neither are particularly well known venues, granted.

  • Odyssey says:

    There's a R03 deadline coming up...

  • Ola says:

    @Jonathan Badger

    We join our players mid-scene, in the conference room at the Bethesda Marriott Hotel...

    Reviewer 1: He doesn't have a lot of publications recently

    Reviewer 2: There's one just out on PubMed, in the Journal of Articles in Support of the Null Hypothesis

    All: Snicker, guffaw, etc.

    SRO: Well we have been advised to be lenient with investigators who can justify a significant gap in productivity due to devoting resources to reproducibility issues.

    All: Snicker, guffaw, etc.

    Hilarity ensues. We fade to black as reviewer 1 pencils in a score of 7, while joking with his neighbor, who is already posting to Twitter about the amusing incident


  • BioDataSci says:

    Here's a paper that just came out in PNAS. "Calorie restriction does not elicit a robust extension of replicative lifespan in Saccharomyces cerevisiae. Calorie restriction (CR) has been shown to extend the lifespans of various organisms. Consequently, a considerable amount of research has been performed to elucidate its mechanisms, especially in the yeast Saccharomyces cerevisiae. Here, we show that due to small sample sizes, large variation exists between measurements. In addition, the effect of CR on lifespan has been routinely overestimated in yeast due to the use of short-lived experimental controls, which together may explain why contradictory mechanisms were found to mediate CR-induced lifespan extension. Moreover, we did not observe any lifespan-enhancing effect of CR using an alternative measurement technique. The inability of CR to robustly extend lifespan suggests that calories alone do not modulate the lifespan of this important model organism."

    Not saying they could get a grant to fund additional work, but they did get a (glamour) pub out of it.

  • Spiny Norman says:

    Dig into the supplements, BioDataSci… this is not so much a problem with yeast, as with specific yeast labs.

  • BioDataSci says:

    Not saying anything about yeast. Just saying it was a study in a "high impact" journal that focused on other studies' failure to replicate.

  • Davis Sharp says:

    Do a one-off experiment and publish the result on Twitter, thereby changing the conversation from crap in glam journals to the Kardashian Index.

  • lurker says:

    It has been known anecdotally for almost a decade that the budding yeast CR stuff was dodgy at best, but only now is there a European Full Prof not really in the aging field and has the funding structure and cajones to publish this do you get the RARE event of replication rising to a publication. Previously, replication studies and data were at best disclosed during half-drunk chats at the conference bar, although there is much more unreviewed chatter now in the blogosphere, twittersphere, pubpeer.

    If you were a n00b PI in the USA and in the aging field, it would be career suicide to try to "replicate" and try to publish this, if only to clean up shitte science like this which allowed the BSD to get his D really B and S at the time. Money (lack thereof) is the least problematic of the hurdles to get fields to replicate shitte. Its more about the shrouded inner sanctum, I won't expose your faux shitte if you look the other way mentality amongst the BSDs and their enablers, so that they'll be extra nice to you on the next SS review round. The Glamour journals with their professional editors of very limited field experience are amongst these enablers of this inner sanctum.

    Replication costs more in cajones than dineros.

  • Dave says:

    CR and lifespan was disproven in monkeys, and so I have no idea why the worm and yeast crowd keep beating this dead horse.

  • drugmonkey says:

    "Because we have a model and it works!!!!"

  • Dave says:

    Ah yes, next you will be telling me red wine prevents aging through SIRT1.

  • Katy says:

    Incidentally, eNeuro (new open-access spinoff of Journal of Neuroscience) claims to be open to these very types of experiments ("Negative Results" & "Failure to Replicate"):

    ...if you'll put up the $15K + publication fees.

  • Grumble says:

    $15K publishing fees!!!

    Good thing that's a typo. The fees are only ~$1500, which is roughly similar to what other journals charge (it often ends up being that high or higher if they charge extra for color figures).

  • drugmonkey says:

    I think that was a reference to my comment that it was going to cost me $15K for the barest replication...

  • […] (I once blundered into a moose–we saw each other about 30 feet away. Absolutely terrifying) Replication costs money The Fame […]

  • Aaaa says:

    Have done this several times. Not impressed or very surprised by Behavioral
    Results in CNS we tried to replicate, failed, spent a year or two on it to understand why. Maybe 3 times this has yielded a paper for us, each time in a modest journal. These papers are cited well though. Thought about commentary or letter, but really? Put up or shut up was our view.

    In response to your query: funded via bits and pieces, some internal, but mostly part of external existing funding. Have also parlayed those data into new external funding. When your lab people care as much about it, they can also be willing to find a way to help get it done.

    In response to the problem: we all know these journals simply do not have the depth of reviewer expertise in behavior. Never have. Never will.

Leave a Reply