Have all of the data been peer-reviewed?

Oct 22 2010 Published by under Peer Review, Scientific Publication

Imagine this scenario, DearReader. You have submitted your manuscript and the Associate Editor who has managed the review sends you the critiques with a recommendation for "minor revisions".

w00t! In like Flynn, amirite?

Now suppose as you are tidying up responses to the criticisms of the reviewers you find one reasonably substantive criticism that would be best addressed with the addition of another figure or two. This might even be for data that you have already, but left out of the original submission for some reason or other. You include it, package the thing up and re-submit the revised version of the paper.

The AE responds by essentially return-email that the manuscript has been accepted. Clearly, it didn't go back out for review. This is fairly typical, btw, that an AE will make the editorial decision that all criticisms have been addressed adequately.

When it comes to new data, however? That now have not been reviewed by peers...?

Is this a breakdown in the system? Has the peer-review stamp of approval been compromised in this case? Or is this an accepted part and parcel of the peer review process which is no different from the AE accepting the paper after minor stylistic changes in presentation , the addition of a few more caveats or citations to the Discussion or toning down the ELEVENTY language in the Abstract?

9 responses so far

  • Namnezia says:

    Well, the associate editor IS part of the peer-review process, and ideally a scientific peer. So his or her opinion on the extra data also counts, even if it doesn't go back to the original reviewers.

  • anon says:

    I assume the authors have taken responsibility and stated, specifically, in a cover letter that new data were added. I would also assume that the editor is a PhD level scientist who can, in most cases, review the revision on his or her own and determine whether it's publishable. As a reviewer, I've had to look at the same paper 3-4 times to review minor corrections, that in my opinion, should have easily been taken care of by the editor.

  • proflikesubstance says:

    I think this depends largely on the data type and how well the AE knows your field. If you are just adding a figure re-enforcing the findings of the paper and the AE can adequately judge the data, I don't see a problem. To argue that AEs are unqualified for that task seems a slippery slope. OTOH, if you are novel data that require the addition of a new section or two of text, the line blurs a bit.

    Obviously, there are infinity shade of gray here, though,

  • becca says:

    Keeping in mind you are talking to someone who is part of a generation of scientists who grew up with mega-data papers (e.g. genome sequencing, microarrays, proteomics...), I don't think a paper being peer reviewed says anything about whether a particular datum (is that the right word for a quanta of data?) has even been seen with human eyes, let alone 'peer reviewed' in any meaningful sense of the word. And I'm ok with that.
    In any case, there are going to be cases where toning done the ELEVENTY wording of the abstract results in a *more* meaningful change to the scientific record than adding an extra figure (I'm thinking of a case where there's a very speculative claim in the abstract and e.g. a preliminary dose-response curve done to establish conditions to carry out your real experiment with).

  • The AE is a peer, right? I don't think the Peer Review Process stipulates some specific number of peers that must do the reviewing.

    From your account, it sounds like the additional data/graphs were minor, confirmatory analyses that were as expected and didn't change the shape of the argument -- in which case, they were in some sense *already* reviewed. If these analyses resulted in a thorough reinterpretation of the primary data and required changing a lot of "is"es to "is not"s, then that's different.

  • Pinko Punko says:

    I am annoyed when I review the paper and don't see the revised version. Not to hammer it again, but to review whatever has been added.

  • Pinko Punko says:

    Adding, this is less controversial than it sounds. The editor already has the power to ignore your review, even if substantive, and go with dissenting reviewers.

  • Beaker says:

    I recently reviewed a manuscript in which all reviewers gave a favorable review and said it could be published after minor revisions. I also suggested a new experiment that would strengthen the study, but in my comments I made it clear that this was a "it would be nice if they showed..." rather than "they must show in order to be published..." type of experiment. In the revised version, the authors did show that experiment and added a new figure.

    But they botched it! The data were clearly generated using flawed methodology (as they described in their methods), and showing these data in the published work would be unacceptable. All other issues from all reviewers were adequately addressed, but the paper is undergoing another revision because they attempted to go the extra step and ended up losing their footing. I reckon that they'll just take out the flawed figure, but we'll see...

  • In most cases, I think it is reasonable to conclude that, as a (presumably) well-trained scientist, the editor can make a call on less-than-groundbreaking figure additions.

    Sending out a manuscript for re-review upon the addition of any old figure is just going to delay the eyeball-gouging process of peer-review even further. You're also sending the manuscript back to potentially overly fussy, power-hungry reviewers that do little more than waste author's/editor's/other reviewer's time.

Leave a Reply