Nature Neuroscience Joins the Neuroscience Peer Review Consortium

Apr 04 2008 Published by under Peer Review

As we discussed in two prior posts, the Society for Neuroscience has driven forward an experiment in the peer review of neuroscience manuscripts. The Neuroscience Peer Review Consortium seeks to streamline the peer review process by re-using the initial set of reviews of a manuscript rejected at one journal when re-submitted to another journal that is in the Consortium.
Sounds great, doesn't it?

Our initial foray into this was motivated by some pondering Nature Neuroscience was going through in trying to decide whether to join the NPRC. Noah Gray launched a discussion amongst their editorial board and other key scientists over at their Action Potential blog on the topic of "confidential comments to the editor", an obvious consideration in the "transferred reviews" idea that motivated the formation of the Consortium. Well apparently Nature Neuroscience has decided to join up [print editorial]:

The NPRC reduces the overall reviewing workload of the community by allowing authors to continue the initial review process when their paper moves from one consortium journal to another, once the paper has been rejected or withdrawn from the first journal. This arrangement is similar to the manuscript transfer system that has been available within the Nature family of journals for almost a decade.

I do find myself congratulating NN for deciding to discourage confidential comments to editor from the reviewers:

Only comments to the authors are transferred to the receiving journal. Confidential comments to the editors are not passed along. Thus, to ensure transparency in the review process, both at Nature Neuroscience and at other journals after the paper has been transferred, we encourage referees to include all their concerns about the paper in comments to the authors. The small amount of extra time required to word the comments diplomatically for the authors should be more than counterbalanced by the resulting improvement in the peer review process.

This is a good thing. I've never understood why confidential comments to the editor are widespread, myself.
As I pointed out in comments during prior discussion [here, here, here] I do anticipate a few unintended consequences. My analysis rests on the premise that the Consortium contains a hierarchical collection of journals with (now) NN at the pinnacle, J. Neuroscience and to some degree Biological Psychiatry at the next rank and then a bunch of specialty or society journals at the foot. (I have not re-scrutinized the journal list closely so there may be a few more relevant ranks or a few more journals filling out the mid ranks. My points aren't really changed by such details.) My concerns have mostly to do with the fact that I think there will be a lot of one-direction travel of rejected manuscripts, i.e., downward in Impact Factor.
First pitfall I see is the trickle down of manuscripts conceived and formulated for the more GlamorMag style of NN, something that J. Neurosci has been moving toward already. For those of us who think this style of scientific communication is not a GoodThing and are happy that serious journals still exist in which data-heavy, rigorously controlled studies are prioritized, well trickle down of GlamorMag style manuscripts would be a BadThing.
Second pitfall lies in the monopolistic character of the Consortium. This concern depends on the empirical outcome of author decision making. Once rejected at, say, J. Neurosci will authors throw up their hands and dump it to a lower IF journal? When once they might have submitted to a journal more closely competitive with J. Neurosci which is now, unfortunately, not in the Consortium? If the dump option is selected more frequently, well, this is going to distort the status quo of the neuroscience journal rankings. Love or hate the current IF-based ranking system for determining the "quality" of your science, throwing some extra chaos into this system seems detrimental to me. Especially if it is not evaluated in this little "experiment".
Third pitfall is the behavior of the editors of the bottom tier journals. Will they start getting manuscripts submitted that they never before would have seen at their humble journal? Will the chance at a seemingly higher impact paper erode their prior adherence to the strictest standards of data "completeness" and appropriately designed and controlled studies?
As with my opinions on grant review, though it is true that I think my hypotheses are likely to be validated, there is a greater problem. Namely that factors in the process which I think are likely to be highly relevant are simply not discussed. Responses which brush aside concerns such as the above suggest that the relevant data will not be examined during the course of the experiment. This is the crux of the matter.
To go through this experiment focused entirely on the local analysis (by-manuscript) and to not consider all of the possible broader (journal level) ramifications seems an error to me.

8 responses so far

  • bsci says:

    I'm not sure I get your concerns about journal ranking here. If higher ranked journals are losing good papers to lower ranked journals because they don't embrace a more open system, there's an easy solution. Join the system. If lower journals get a ranking boost for participating in a more efficient review system, all the better.
    Probably the big negative of this system is that more people will shoot higher first since the opportunity cost for failure is slightly lower. This means Nat Neuro will probably get some more gems, but it will need to sort through a whole lot more rejections too. In balance, I'm not sure why Nat Neuro agreed to play along.

  • DrugMonkey says:

    It isn't an "open" system bsci, it is a closed system. Agreed, every journal could just join the system. And if a big majority of neuroscience-related journals join the fun, this concern goes away. At present, the list of Consortium journals is small compared to the pool of neuroscience journals.
    Agreed about wondering what NN gets out of this. When JNeurosci was the pinnacle journal, their interest was very clear. I'm not so sure which non-Consortium peer journal NN might be pulling submissions away from with this move.
    One interesting twist that struck me is the scenario when a manuscript gets hammered and the authors want to submit to another consortium journal without using the streamline option. Then when the same reviewer gets the manuscript to review, again, is s/he going to be extra-annoyed that the authors haven't used the streamline procedure?
    I would be...

  • Noah Gray says:

    bsci, you are correct, and as Sandra points out in the comments section of our blog, we do only send a minority of papers out to review, so indeed, there is a chance our workload will increase. The "gems" that you mention may come our way instead of going to our direct competitors (like Neuron), because of our presence in the consortium. That is a good think for the journal. But, frankly, we joined the NPRC because we felt that overall, it would be good for the neuroscience community.
    I'm not sure how annoyed reviewers are when they get asked to re-review. It happens a lot, since editors know the good reviewers in particular fields. If the reviewer does get annoyed, well then that is a risk the authors are taking by not transferring reviews. I don't necessarily see this as a weakness of the consortium. The strength is that all parties involved can flexibly choose when to participate.

  • DrugMonkey says:

    If the reviewer does get annoyed, well then that is a risk the authors are taking by not transferring reviews. I don't necessarily see this as a weakness of the consortium. The strength is that all parties involved can flexibly choose when to participate.
    For the sort of journals I typically review for, manuscripts are rejected for cause and not typically for being insufficiently teh haww3t. Consequently, as a reviewer I am annoyed when a manuscript comes back via another journal without anything that was wrong in the first place fixed.
    My response is usually "why the heck should anyone re-review this if you haven't changed anything?"
    It isn't so much that the consortium changes this. Merely that knowledge of the consortium just makes it extra annoying.

  • bsci says:

    I guess I meant more "open" in the sense that secret comments to the editor disappear and journals have a better idea of the history of the article. As long as the majority of neuroscience journals aren't in the consortium, I'm not sure how often people resubmit and tell them not to pass along the reviews. It would be easier to just send it to a journal that doesn't participate. Of course that also might increase the number of previously rejected articles to non-member journals, which is another push to join up.
    I am very curious what numbers these journals crunch before deciding whether or not to join. I'm not sure if you can or want to answer, but did Nature Neuro do any calculations on the expected increase in # of submissions after joining? Are they planning to keep track of these numbers. It would be great for all journals in the network to keep good track of changes in submissions, acceptance rates, and review times to get a quantitative model of what the network changes. As a random thought, even the removal of comments to the editor could decrease inappropriate negatives and bump up the acceptance rates. That alone could be an interesting study.

  • Noah Gray says:

    I was not involved with the ultimate decision to join, only on the lobbying side. I think that there is a significant chance for increased submission levels (15% increase?), but as I said above, it will do authors no good to simply submit to NN if the paper would not get reviewed there anyway. The thing to remember is that this system only helps the authors (and reviewers and editors), if the paper gets reviewed at another journal. I'm sure we'll crunch the numbers, but I don't think the NPRC will have any effect on our acceptance rate of papers we review. If anything, our overall acceptance rate will decrease due to the presumed increase in submissions.
    I completely agree with you on the point of why should a paper be re-reviewed if nothing has been changed. Well, the editor usually finds out that an author is bouncing a paper around only after s/he asks the reviewer for help. Upon learning that, it is more than reasonable for the reviewer to say exactly what you said, so long as the novelty standards and appropriateness of the subject matter at the new journal are identical to those upheld at the previous journal. In that case, I feel that it does the author well to be explicitly told "DO THE DAMN EXPERIMENTS AND STOP SHOPPING THIS CRAP AROUND!!!" That is usually a more effective statement coming from the reviewer.
    Of course, if the new journal has a more relaxed policy on novelty and that was the main criticism of the paper, the reviewer should re-review the paper and modify his/her opinion accordingly. Having already meticulously combed through the manuscript, it will be vastly faster and easier for the same reviewer to do that job than a brand spanking new one.

  • DrugMonkey says:

    In that case, I feel that it does the author well to be explicitly told "DO THE DAMN EXPERIMENTS AND STOP SHOPPING THIS CRAP AROUND!!!" That is usually a more effective statement coming from the reviewer.
    What seems to happen in most cases* is that the authors shop the same crap** until they luck onto a new set of reviewers and a friendlier editor.
    *that I notice eventually reaching publication.
    **and I should be clear that I am pretty generous about understanding when it is and is not practical to add to the paper. There are some longitudinal and highly expensive studies (i.e., some human work, occasional nonhuman primate studies, super expensive assays or equipment) in which even as a reviewer you think "well it isn't perfect but it is of interest. and this is a one shot deal, there is no way more work is going to get done. Run of the mill rat or bench studies do not fall into this category in my view.

  • Reggie says:

    I'll start by saying I am still on the fence about this one, I can see some potential benefit to authors, and I hope that NNeuro or the other journals will make public how the NPRC "experiment", works out for all parties involved. With that said, I'm not sure that I have seen any comments yet, here or on NN blog about what this expected 15% increase in submissions could mean for the rest of the system. 15% seems like a sizable increase in work load. Is it fair to say that this will further increase the time that papers are in the hands of the journals? Could it lead to rushed decisions by the editors if their individual workloads increase? How about the costs? If journals need to increase their staff to handle the workload, will the financial burden be passed along to authors that are likely already strapped for cash?

Leave a Reply