Review unto others

I think I've touched on this before but I'm still seeking clarity.

How do you review?

For a given journal, let's imagine this time, that you sometimes get manuscripts rejected from and sometimes get acceptances.

Do you review manuscripts for that journal as you would like to be reviewed?

Or as you have perceived yourself to have been reviewed?

Do you review according to your own evolved wisdom or with an eye to what you perceive the Editorial staff of the journal desire?

31 responses so far

  • MoBio says:

    The concerns above are irrelevant for me.

    I simply review them as follows. I don't assume those who review my papers do it this way, btw

    Do the data support the conclusions is most important for me.

    Equally, do I believe the data (are the data believable?)? Are those Western blots doctored? Are photos, photoshopped?

    Next: things like sample sizes, biological replicates, antibodies and other precious reagents.

    For HIF journals where I'm asked to give a sense of 'is this of interest to the general audience?' I ask myself that and give an honest reply.

    Grammar--(as long as I can understand what they meant to say)--assume copy editors will fix up though I will highlight things that are obscure.

    For HIF journal, I assume that if they (Editors) sent it out for review they at least have some thought that there is something interesting here, though I don't take that into account at all in my reviews, other than to occasionally chide them.

  • drugmonkey says:

    So "your own evolved wisdom" then?

  • Busy says:

    I review them for correctness (i.e. sound scientific practices) first. Then I calibrate according to the journal's targeted range. Some aim to be the C/N/S of their field so you apply a higher standard, others just aim for general relevance and you go easier on them. I often look at previous issues to get a feeling of what the journal is aiming for. If this is the reason for rejection I make it very clear in the review: nice paper but seems somewhat below the norm for Learned Journal of Learned Society.

    This leaves the choice on the hands of the editor. If s/he feels I was too tough they are welcome to override my comment. If the paper is incorrect on the other hand then I try to be as constructive as possible, i.e. minimum number of changes that would make me switch my decision to accept and not what ideally one would do, given infinite time and money.

  • namnezia says:

    I don't really alter my review based on the journal. I just try to give as constructive comments as possible. If they specifically ask about impact then I'll list as many novel things I can find. I guess the one difference between my reviews to glam and non-glam journals is that if think a paper is good and it's been sent to a glam/glam-ish journal, then I make sure to be super extra enthusiastic and supportive about its impact.

  • baltogirl says:

    Yes, the Golden Rule applies. There is too much negativity in review these days- don't add to the problem.
    This article on PMC by Drubin (PMC3046051) provides ten useful rules and has an interesting title:
    "Any jackass can trash a manuscript, but it takes good scholarship to create one (how MBoC promotes civil and constructive peer review)"

  • MoBio says:

    @DM: perhaps--or my evolved bias..

    Agree @baltogirl--so easy to trash a paper.

  • drugmonkey says:

    I guess the one difference between my reviews to glam and non-glam journals is that if think a paper is good and it's been sent to a glam/glam-ish journal, then I make sure to be super extra enthusiastic and supportive about its impact.

    I think this is the one thing that I struggle with the most. It's otherwise known as speaking the language that the editors understand, right?

    This leaves the choice on the hands of the editor. If s/he feels I was too tough they are welcome to override my comment.

    There are a couple of journals, with which I spar as an author, that make me reconsider this as a reviewer. Now, maybe the editors are hiding behind some alleged inflexibility in the process when they choose not to override, and they otherwise choose ignore reviews at will, I don't know. But at least on the face of it, for these journals the reviewer behavior can constrain the hands of the editor.

    One example is, a journal that has (supposedly) decided that two reviewer recommendations of "major revisions" in a row is equal to "reject". In the past, ignorant of any such policy, I might as a reviewer return a recommendation of major revisions after a disappointing revised version has been received. I intend it as a statement of "the authors aren't taking these comments seriously and they really need to stop rebutting and make some changes". I might very easily still think that the manuscript should be accepted and that I am not in any way recommending a rejection. Now that I know the editors will treat it as a rejection, I need to speak to them differently by using minor revisions.

  • drugmonkey says:

    Followup question:

    To what extent do you think that Editors for whom you review regularly expect you to follow your average behavior?

    I.e., do you feel that they think they are putting a finger on the review process by selecting you as either a hard case or puff ball reviewer?

  • Juan Lopez says:

    Sometimes I wish I had been nicer to authors that got their paper rejected when I recommended major revisions.
    Sometimes I wish I had not sugar coated the major issues that an Editor let go.

  • I've asked a couple of editors about this recently and they said that it's so hard to get people to agree to review these days that they can't really make anything like that a criterion.

  • Dave says:

    I've always tried to be fair in my reviews and just focus on whether the conclusions are supported by the data, and whether the experiments have actually been performed properly. I don't review for glam, (I'm junior riff raff), but do review for some solid society journals. I try not to worry too much about IF, but admit that sometimes it creeps in.

    In general, I've had a decent time with my own papers......up until a recent paper that is.

  • Dusanbe says:

    I assess whether claims are well supported by data. If the paper is in my immediate sub-field of expertise, look closely at data to see if their interpretation of the data is correct.

    If the paper is outside immediate sub-field, I will not dispute the authors' interpretation. Data in my field is so organism-specific that one can't make heads or tails of the data unless you have 10-15 years' working experience looking at this type of data every day.

    Additonally, I am a stickler for scholarship. I will call out authors for not citing previous papers when needed, especially if they are presenting certain findings as "novel" when they really aren't.

    What I will not do is make a judgment call on whether X is of "broad interest". Meaning, I assume by default that the manuscript is being sent to me because the editor has already made that call. I will not do the editor's job for them. Especially at glam journals, where they are supposedly paid to make those decisions. Therefore, I always say the paper is very interesting and appropriate for the journal (if the claims check out and the authors have properly researched and cited the literature).

  • mH says:

    Are the experiments conducted according to accepted standards, is the data presented appropriately are they interpreted reasonably.

    I don't give a shit which journal publishes what so won't comment on impact or suitability (with an an explanation, so that withholding isn't a tacit "no"). If it's forced, I give maximum scores for everything and explain in comments to the editor.

  • MoBio says:

    @DM: "To what extent do you think that Editors for whom you review regularly expect you to follow your average behavior?"

    As an Editor the only expectation is that if the reviewer is thorough and fair I would anticipate the same high quality in the future.

    Reviewers that send in overly hostile/irrelevant/superficial reviews generally are avoided. Some notorious reviewers (always very late; always extraordinarily negative) I've seen flagged as 'avoid' by senior editors.

  • new PI says:

    The worst are reviewers who *think* they know what's appropriate for a journal and have no clue.

    The only real complaint about my last manuscript was that the biological impact wasn't large enough. (I was showing that a method that has been applied in CNS papers, among others, is in fact completely untrustworthy the vast majority of the time. Our biological impact was thus in fact rather large if you consider all the previously published conclusions now thrown in doubt.) The reviewer said, The lack of biological results makes this a more appropriate study for Other Journal.

    Now it turns out that Original Journal specifically does not require biological impact as a criterion for publication, and it publishes a lot of interdisciplinary methods papers. Other Journal does in fact require some interesting biological result on top of new methods. This is clearly stated on their website.

    The editor obviously wanted to get rid of the paper (five reviewers in the first round, four positive, one clueless). Obviously, negative results and no fun and not cited much. But the reviewer gave him/her an out ("impact") that was misguided and deeply unhelpful.

  • new PI says:

    On coaching biased professional editors:

    One of the glam journals publishes almost nothing from my field at all. It's like they don't recognize that legit scientific conclusions can be reached from certain types of data and analysis--and in fact, sometimes only from this kind of analysis. I try very hard whenever I review for this journal to try to educate the editor about how valuable some types of observational analyses are (and how a mouse model reproducing some sliver of the mechanism really wouldn't add anything, or increase the credibility of statistically stunning and cross-validated conclusions), and she won't buy it. There's always some mouse reviewer saying, But you didn't do an experiment, so can all these other amazing observations really be true? It drives me bonkers. But I guess that's the glam life.

  • odyssey says:

    I review as I hope to be reviewed. And I try not to take into consideration what's "appropriate" for a journal's IF.

  • drugmonkey says:

    New PI- yep, that's glam life. And not-so-glam life. Model based distinctions aren't stupid but very common.

  • Anon says:

    I have a question. What is the ethics of having grad students or postdocs review a manuscript? Obviously the PI has the responsibility to review and approve of the comments but is this appropriate? I found that it is VERY hard to gain peer review experience and this is one way to help that.

  • AcademicLurker says:

    @Anon,

    When I was a grad student/postdoc, I was given papers to review by my PIs. When they submitted the review (with their own additions or changes) they included a line "This review was prepared with the assistance of AcademicLurker".

    Now that I'm a PI, I do the same thing. Not for all papers, of course, but I try to make sure everyone gets some reviewing experience.

  • David says:

    @ Anon
    I see two ethical options, 1) you inform the editor/AE prior to accepting the task and get permission, or 2) while you have the student write a review and you provide feedback, the review you submit is your own (in this case, you could include the students comments with your own).

  • new PI says:

    @Anon, I always ask the editor first.

  • Grumble says:

    "One of the glam journals publishes almost nothing from my field at all. It's like they don't recognize that legit scientific conclusions can be reached from certain types of data and analysis"

    Well, at least you're not alone. Your field didn't win the popularity contest, so you don't get to sit with the prom queen and the cheerleaders and the football players, but with me and the D&D crowd in the back of the cafeteria.

  • Grumpy says:

    Never once had an editor say no I shouldn't involve my postdoc/senior grad student. As long as you sign off on the review, I doubt you even need to ask (though I always have).

  • Justthisguyyouknow says:

    You guys would really check and sign off on a postdoc's review? Why not just have the editor assign it to the trainee?

  • Dusanbe says:

    Gaining "peer review experience" is like gaining jury duty experience. Why would you need to seek it out? Just giving it your honest, fair, best effort is all that can really be expected.

  • Grumble says:

    "Why would you need to seek it out? Just giving it your honest, fair, best effort is all that can really be expected."

    Right, but it's not always obvious how to develop your honest and fair opinion, and how to get your honest and fair opinion on paper.

    Developing an opinion about the papers we read is something we are all supposed to learn how to do in grad school, but the more practice you get, the better you become at it. That is one reason why having the opportunity to review papers is useful for trainees, including post-docs.

    One you have an opinion, you have to write it down it in such a way that it is clear, not derogatory, not ass-kissy if it's favorable, and conveys to both authors and editors what you think the authors will need to do to overcome the shortcoming you identify. This is a skill you learn, not something you are born with.

    Finally, reviewing a paper (if you do it honestly) forces you to read something you may not otherwise have chosen to read, and read it with greater attention than the typical skimming you might do while you're browsing the literature. That, I would argue, is useful not just for faculty types, but for students and post-docs as well.

    All of the above comes from my own anecdotal experience both with learning to review papers from previous PIs' assigning them to me, and interacting with students and postdocs whom I ask to help me to review papers.

  • drugmonkey says:

    At my ripe old age I still wonder if I am doing the best possible job of review. I wonder about what the job is supposed to be and how we, as a community of scientists, need to behave. I wonder about biases.

    See OP.

    Thinking that your unexamined "best effort" is sufficient seems wrong to me.

  • Dusanbe says:

    Sorry, I was being facetious, as I attempted to indicate by comparing peer review service to jury duty. What I meant is, nobody in their right mind goes around volunteering for jury duty just so they can gain experience and become better jurors. Even though we might agree it's an important civic duty yadda yadda, like peer reviewing.

  • Newbie PI says:

    I have a question about the journal Scientific Reports. It is just a pay-for-publication money-maker for Nature? I've reviewed for them twice now. Both times the paper should have been rejected on scientific and ethical grounds (image duplications, etc), and both times the reviewers were overruled and the paper was accepted. Peer review seems like just a formality at that journal. Have others had the same experience?

  • Pinko Punko says:

    Newbs- this is pretty disturbing. Did other reviewers catch the same stuff? Are these papers discussed on PubPeer? Did you report above the handling editor?

Leave a Reply