No Way Am I Doing That Experiment

Sep 28 2008 Published by under Peer Review

There has been some interesting discussion over at Blue Lab Coats concerning how to respond to peer reviewer comments on a manuscript submitted to a journal. If a journal editor concludes that a submitted manuscript is potentially suitable for publication, then she will send an e-mail to the corresponding author saying something like this:

Based on the reviews, we will be happy to consider a revised version of your manuscript, in which you should modify the manuscript to address the concerns raised by the reviewers. It is important that you address the reviewers' concerns in your revised manuscript and also in a point-by-point reply to the reviews that indicates how you have responded to each of the reviewers' comments.

Suppose that the reviewers have asked for additional experiments that there is no fucking way you are going to perform before resubmission. What are the circumstances under which you can finesse this, and still get your paper accepted?


Here are some scenarios in which one can usually get away with not performing a suggested experiment. I will illustrate with mock verbiage from a putative point-by-point reply to the reviewer comments.
Scenario #1: Reviewer Asks For Another Control

As the reviewer points out, those experimental conditions were not used as controls for the experiments described. Rather, we used a whole bunch of other fucking controls. These are all better controls than the one suggested by the reviewer because they exclude the much more likely possibility that blah, blah, blah, rather than the highly unlikely possibility that bleh, bleh, bleh ruled out by the control suggested by the reviewer.

The important thing in finessing this scenario is to make a very convincing argument that the possible artifact ruled about the additional suggested control is much less likely to be occurring than the ones ruled out by the controls already performed. There are always an infinite number of possible controls for any experiment, and you obviously can't do all of them. What's important is to rule out the plausible artifacts. Authors need to band together and refuse to indulge reviewer requests for laborious additional controls that rule out exceedingly unlikely artifacts.
Scenario #2: The Experiments Actually Performed Are Interesting, But What Would Really Be Impressive Is Blah

While we appreciate that such a result might be "impressive", it is not clear what it would add to the strength or nature of the conclusions concerning blabbity blabbity blab we have drawn from the experiments already performed. The suggested experiment has two possible outcomes, which would lead to two different conclusions.
However, distinguishing these possible conclusions does not tell us anything further about the strength or nature of the conclusions concerning blabbity blabbity blab we have drawn from the experiments already performed. The relevant point in terms of the conclusions of the maunscript is that--as our experiments already show--the effects we observe are not due to blah, blah, blah. Given the time-consuming nature of the suggested additional experiment--at least three months--we conclude that the cost in time and effort is not worth the potential benefit.

This is another situation where we authors have to band together and refuse to indulge reviewers' attempts to transform our manuscripts from the ones we actually wrote to the ones they wish we had written. If the reviewer already considers the manuscript of sufficient interest and and convicingness to merit publication by the standards of the journal it has been submitted to, then fuck this "what would really be impressive" shit. It's one thing if the reviewer thinks that the manuscript as submitted is not of sufficient interest. But if she thinks it is, then no fucking way am I gonna let her hold up publication for this kind of shit. Those experiments can go in the next fucking paper, thank you very much!
Scenario #3: What Would Really Strengthen The Main Conclusion Of The Manuscript (Which I Have Completely Misunderstood) Is Blah

Blaargh is the only known blingity encoded in the genome, thus making it nearly certain that the effects we see of kapow in our experiments are due to an effect on blaargh. Furthermore, the effect of kapow on blaargh is to inhibit wango, not to decrease the magnitude of wingo, as shown in the experiments of Figure 43. Since neither of the suggested experimental manipulations influence blaargh wango, such experiments would not be germane to the conclusions of the manuscript.

In this case, the reviewer has misunderstood the biological effect of the main experimental manipulation used in the manuscript, and has thus misunderstood the nature of the main conclusion of the manuscript. Based on these misunderstandings, the reviewer has suggested experiments that are irrelevant to the actual main conclusion of the manuscript and are only germane to what the reviewer erroneously thought was the main conclusion.
This is a tricky situation, since you wouldn't have even gotten to this point of invited resubmission if this reviewer hadn't already expressed substantial enthusiasm for the manuscript. So the last thing you want to do is lead the editor to conclude that this reviewer is such a clueless fuckwit that even her substantial enthusiasm for the manuscript must be discounted.
One final point: Regardless of the strength of your arguments against performing one or another suggested experiment, the most important consideration for the editor will always be the consensus of the reviewers. If you have three reviewers and a particular additional experiment was only suggested by one of them, you have a very good chance of finessing it away. If it's made by two of them, you better have a very very strong argument. And if it's made by all three, then forget finesse, and hit the motherfucking bench!

11 responses so far

  • Damn! Why'd you have to post this literally an hour after I uploaded the manuscript, including the response to reviewers, which Advisor and I have been batting back and forth because of a Scenario 2 sort of issue.
    Ended up doing more or less what you said, but now I think we should have emphasized that that experiment wouldn't have actually told us anything further about the interesting part of our findings, as you say.
    Next time, pls be more psychic.

  • jonathan says:

    "as shown in the experiments of Figure 43"
    HAH! Well... at least we know it wasn't submitted to N/S/C! Must have been a JBC paper...

  • I've had a colleague admit to me that, when he's reviewing a manuscript and can't think of anything substantive to say about it (whether for lack of a brain or lack of time, I'm not sure), he always asks for another control to be done -i.e. Scenario #1. What a goddamn waste of time.

  • PhysioProf says:

    Candid, I hope you told that motherfucking asshole to fuck off and die.

  • ---"Furthermore, the effect of kapow on blaargh is to inhibit wango, not to decrease the magnitude of wingo, as shown in the experiments of Figure 43"---
    Damn, got scooped again! There goes my epic WingoWangoTango paper.

  • studyzone says:

    In my case, one of our reviewers didn't like how we interpreted data from one experiment, and asked us to do it a different way, just to be sure. Well, we had already thought of that, and the data was in the next figure (and explained in the paragraph after the objectionable data). After an informal email from my grad advisor to the editor (with page and figure references), our paper was accepted with only minor revisions to text, and one additional experiment. Makes me wonder if the reviewer read the objectionable paragraph, began writing the critique (gleeful at catching us in an obvious lack of insight/judgement), then ignored the rest of the manuscript.

  • CC says:

    Makes me wonder if the reviewer read the objectionable paragraph, began writing the critique (gleeful at catching us in an obvious lack of insight/judgement), then ignored the rest of the manuscript.
    Know what, though? Other readers aren't going to read your paper any more carefully than the reviewers. When a reviewer complains about the absence of something that you have in fact stated, it's a red flag that you might be able to make the presentation clearer.

  • Liam says:

    CC @7: that advice is spot on. You can never really control how carefully other people read your work, but if someone is not catching on to what you have written, it is prudent for you to revisit how you presented the information. Maybe there is no need for a change, but often you might find that a careful rephrasing or addition of a qualifying statement makes all the difference.

  • David Marjanović says:

    When a reviewer complains about the absence of something that you have in fact stated, it's a red flag that you might be able to make the presentation clearer.

    Yep. Happens all the time. I usually end up sprinkling three additional sentences over the manuscript and writing back to the editor: "We have now clarified this in the abstract: [quote], in the Methods section: [quote], and in the conclusions: [quote]."

  • ScienceWoman says:

    I usually end up sprinkling three additional sentences over the manuscript and writing back to the editor: "We have now clarified this in the abstract: [quote], in the Methods section: [quote], and in the conclusions: [quote]."

    I've done exactly this to great success.

  • whimple says:

    Given the time-consuming nature of the suggested additional experiment--at least three months--we conclude that the cost in time and effort is not worth the potential benefit.
    A more polite way to say the same thing goes along the lines of, "the proposed experiments suggested by the reviewer will be very interesting for future studies, but we feel that they lie beyond the scope of our current manuscript."

Leave a Reply