Triage can be the hardest part of learning to review

Dec 09 2009 Published by under Mentoring, Peer Review

Some of the more consistent characteristics of the scientist are the inclinations to teach, fix, trouble-shoot and generally help out their colleagues, students or research trainees. As with any GoodThing, when it is taken to an extreme it can turn into a BadThing.
A little Twitt action around the toobs recently brought this into focus in the context of reviewing manuscripts submitted for publication in a journal.
In this particular case it was a question about the writing quality in the manuscript.

Science peeps: is it sufficient as a reviewer to state that the manuscript (a lit review) is simply not written in readable English?

One response, from the always perspicacious Dr. Isis, supported the helpful side. She suggested that it is not cool just to shrug off the authors with such a complaint and that one should address what one can as a reviewer. That's my interpretation of the 140-char / time discussion anyway. The other response, from your heroic and very humble narrator, was that it is perfectly acceptable to kick a manuscript back for being written in an unreadable manner.
What I've landed on as a discussion point for today, however, is the notion that we would ever seek to simply triage our efforts on a peer-review assignment. Is it permissible, within the polite society of the Tribe of Science, to essentially refuse to review one component of a paper (say the data) until another (essential, optional, peripheral?) component has been fixed?
I say it is. Indeed, I think that one of the primary functions of a mentor trying to help a graduate student or postdoc learn to review a manuscript is giving them the tools for deciding when NOT to review the manuscript in any depth. When to decide that there is a fatal flaw that simply must be fixed before it is worth going into any additional effort.
This could be be because of the data- a missing condition or control or perhaps a completely inappropriate (or missing) statistical analysis. Not worth going into the arguments in the Discussion until you agree the data actually present some sort of knowledge, right?
And yes, we can acknowledge right off the top that the line of when to say when can be a very slippery thing. Hard to nail down any hard and fast rules making this a very subjective aspect of review. Of course, any scientist should feel free to make any comments they like, even to the extent of trying to rewrite the dang paper themselves. I just think it is bad strategy and a waste of time.
So while I understand that any trainees who are starting to review manuscripts* want to look like they are insightful and pay strict attention to detail, I also work on getting them to understand that it is perfectly okay to triage a complete piece of garbage.
__
*You mentors do have something of a plan in place for starting this process with your postdocs, right?

34 responses so far

  • Dr Becca says:

    Ooh, I can't wait to hear the comments on this! Interestingly, the responses I got on Twitter seemed to be split 50/50:
    "Yes, poor language alone is grounds for recommending rejection."
    "I write that I cannot evaluate the manuscript (or parts of it) and ask that the authors rewrite and resubmit."
    (same twitterer) "But I've never found the English to be so bad that the entire manuscript can't be reviewed. So I'd review what I could as well."
    "Review article has to be easily readable or it's a strikeout in my book.-Writer should have found someone to edit it."
    "No. I think you have to be more substantive than that."
    "This study shows ~5-10% research manuscripts to a journal rejected for language problems, at least in part." link
    "not really. I generally call it 'poorly written' and then provide quite a few examples."
    "absolutely, and especially for a review"
    ...but only you channeled Yoda:
    "Fixing poor manuscript, reviewers' job it is not"

  • Vast Right Wing Conspiracy says:

    The language and the scientific contents are two separate things, and should be reviewed separately. Even if the language is bad, it's seldom bad to the point of making the reviewer unable to understand what the manuscript is about. If a reviewer refuses to review on the sole basis of the language, it's usually an indicator of the reviewer's lack of relevant expertise, or plain laziness.
    I quite regularly review for a journal serving primarily non-native English speakers, so I know how bad the language of manuscripts can be. Still, I firmly adhere to the opinion stated above.

  • Pascale says:

    Was hoping one of your would get around to this post.
    With manuscript review, I rarely get one that is completely unreadable. I often get those from ESL folks that have numerous dysfluencies that make review a challenge. In that case I point it out and, if the data seem interesting, hope I get to review a cleaned-up version.
    I have had grants that were so unclear I couldn't tell what the point was. I have voted to triage these, even from some VIP PIs.

  • Eric Lund says:

    I'm with VRWC on this one. I routinely review papers written by people who are not native English speakers, and while sometimes a specific passage may be unclear due to language issues (I complain about that when it happens), the paper as a whole is generally intelligible.
    The one time I did triage a review, it wasn't because of bad English. That was the paper where the authors copied multiple paragraphs verbatim from one of my papers. I informed the editor, with evidence, and asked for guidance--he said that I had shown him enough to reject the paper. Unfortunately, that meant that I didn't notice that the parts of the paper that were original were (as Pauli would have said) not even wrong until I saw that paper published in a different journal.

  • A Reader says:

    In this blog entry, DrugMonkey expands upon a previous series of twitters by himself and others regarding the peer review of poorly-written manuscripts. DrugMonkey argues that it is perfectly acceptable to avoid substantive scientific review of manuscripts that are not poorly written.
    While a potentially interesting topic for discussion, this blog entry was not well organized and the central argument remains virtually unsupported. I do not think this blog entry adds substantively to the pre-existing twitter discussion and therefore is not suitable for posting here without major revision. Even with revision, it may be more suitable for a blog with a more specialized audience.
    Some of my major concerns include:
    1) The background for the blog entry was poorly described. While DrugMonkey provides a link to more information, readers of a general-interest blog should not be required to click through to twitter discussions or discussions of twitter discussions. Furthermore, the introduction did not set out a clear problem that will be addressed by the blog entry. It remains unclear what the purpose of this blog entry is.
    2) Overall, this blog entry relies too heavily on blogger-specific jargon. 'Tribe of Science', a reference to 'Dr. Isis', etc. may be fine if DrugMonkey intends to write solely for himself and a few other bloggers, but such language leaves out many potential readers. It is difficult to see how this blog topic could be reproduced by others or compared to other discussions, given the overly specialized focus.
    3) It is not clear how DrugMonkey's references to better mentoring in paragraphs 7 and 10 relate to the central thesis of the blog entry. The interrelation between these two topics needs to be much more fully explored (including the addition of several new paragraphs), or the mentoring arguments should be excised from the blog entry.
    4) I have serious concerns regarding the ethical use of advice. In the footnote, it seems that DrugMonkey might be suggesting that mentors 'hand off' reviewing assignments to trainees. Manuscripts for peer-review are privileged communications not to be shared with others except possibly after prior consultation with editors and/or program officers.
    5) Related to my point #4, above, I am concerned that DrugMonkey too easily shirks responsibility for proper scientific peer review. Not all manuscripts are easy to read. It is the responsibility of the editors to ensure that a manuscript is suitably written. It is important that reviewers, on the other hand, critically and carefully review the *scientific* content of manuscripts. This is what editors are asking for help on, not the language. DrugMonkey needs to better support his argument by describing, for example, what the benefits are of 'triaging' a poorly-written manuscript is for the scientific community versus performing a careful and complete review as much as possible, as argued by Dr. Isis. The benefits should be considered from several vantage points, including the editors' and authors' -- not just the reviewers'.
    6) In the penultimate paragraph, DrugMonkey asserts that it is "Hard to nail down any hard and fast rules..." This statement is surprising, given that there is a large body of literature containing suggestions for proper peer review. It is not clear whether DrugMonkey is unaware of this literature or is willfully ignoring it. The sentence "Hard to nail down...", furthermore, is grammatically incomplete. Yet DrugMonkey asserts that poor writing should be grounds for complete dismissal of manuscripts without further consideration!
    Recommendation: Major Revision, re-review required.

  • I think that it is fully appropriate to not review a paper until it is clear that it is in a form that would be "acceptable for publication" if the data and analysis were appropriate. I received an MS for review that had glaring errors (not necessarily in science content). For instance, several references cited were not in the bibliography, Figure 2 was included and not referred to, Figure 4 had the same figure legend as figure 3 etc...
    It sure did make me wonder. If the authors were this flaky with their manuscript, how reliable could their data really be? You can judge a scientist by the care that goes into the generation of an MS. I told the editor that it wasn't worth my time to review it if it wasn't worth their time to properly edit the MS. The editor whole heartedly agreed.

  • BugDoc says:

    Since many scientists have to work with the disadvantage of writing in a non-native language, I generally don't reject manuscripts for bad writing, but have strongly emphasized where needed that the poor quality of the writing and editing (specific examples cited) made it difficult for the reader to appreciate the authors' conclusions. I think this is a fair criticism. Senior authors should make sure that the manuscript is well written and appropriately edited, but I'm not sure they always do.

  • Zen Faulkes says:

    Having said on Twitter, "It's okay to reject because it's badly written," I should clarify that I say that more as a matter of principle than something that should be done routinely. I agree that writing is usually easier to fix than, say, missing data, and so authors should usually get the benefit of the doubt.

  • HA HA HA! A Reader cracks my ass up.

  • El Picador says:

    HA HA HA! A Reader cracks my ass up.
    Mine too. But clearly "A Reader" is conflict with the author because a scathing review like that should have led to a Rejection recommendation...

  • Funky Fresh says:

    I am convinced that there and endless numnber of sockpuppets on the internet. I feel bad for whoever does the internet's laundry.

  • From my perspective, this really should be something that is done at the Editor/Associate Editor level. If I receive an article to review, I assume the Editor/AE has deemed it readable and worthy of a review. As such, I'll perform one, possibly begrudgingly, but a review nonetheless. Another reason to do it at the Editor/AE level is that the more *ahem* impossible manuscripts they send down to be reviewed, the less likely they are to find reviewers the next time around.
    I have, on at least one occasion been unfortunate enough to receive several barely readable manuscripts (in which I strongly suggested someone look over the manuscript for proper English, in addition to offering my own grammatical and typographical corrections) from a particular journal. When they came calling again, I told them to forget it, they had filled their quota of reviews with me for the year and they needed to find some other poor sucker willing to read some horribly written manuscripts.

  • yolio says:

    One problem with stating simply that the "manuscript is poorly written" is that such a brief statement is easily ignored by the editor. Maybe this is just my problem as a lowly, girl post-doc, but I find that if the writing is a problem, I need to make that case forcefully in the review or I will be blown off. And let me tell you, it really pisses me off to see totally unreadable, uninterpretable crap in press AFTER I told the editor that it was unreadable.

  • drdrA says:

    I've never triaged one on non-standard English, although I probably should have. I have, many times, written XYZ manuscript requires extensive editing for standard English (a la Bugdoc #7). I have also written that I can't evaluate the data when proper statistics haven't been done, or apparently even considered by the authors- this mostly in cases where the control and the experimental sample appear close and the authors are claiming an effect, but have offered no statistical evidence. This will cause me to triage or return with minimal comments pretty quick.

  • I agree with what TJ is saying about the editor in chief or AE of the journal tuning their ass up for the poorly written manuscript before passing it off to reviewers. But I think the cop out for writing in a second language is starting to fade away. I know many non-native postdocs that can write better than some native PI's. I'm editing a manuscript for a foreign instructor in our lab and I mean that thing is a beautiful piece of writing. I think I maybe found one grammatical error.

  • Odyssey says:

    Thomas Joseph hit the nail on the head. Unintelligible manuscripts should be triaged at the editorial level and should never be sent out for review. Now by triage I also mean the manuscript can be sent back for rewriting rather than outright rejection. As an editor I have triaged manuscripts for this reason as well as for poor science.
    Many journals require this of their editors and even set target "rejection without review" rates, often around the 50% level (aimed at rejecting the weaker science). This is necessary in order to not burn out reviewers - you'd be amazed how hard it can be to find people willing to review, let alone those who are committed to doing a good job of it.

  • I've rejected several manuscripts without reading most of the discussion because they merely reproduced studies that had long since been published. There have also been several in which the English grammar and/or spelling was so poor that they were almost impossible to understand - in those cases, if the science seemed ok, I recommended a major revision or de novo resubmission and made sure to tell the editor the reason behind the decision.

  • Namnezia says:

    I don't think I've ever outright trashed any paper I've reviewed (well maybe once, one that was trying to pull some serious shenanigans). There's always some merit in even the most piddly and poorly written manuscripts, and I feel it is my job as a reviewer to bring out as much as possible the strengths of the paper I'm reviewing and suggest ways to improve it, including improving the language, no matter how much I dislike it, or how boring I find it. We all know how much effort and hard work it is to produce a paper, and for me to say "oh, they are just reproducing another study" would be self-serving and condescending. And to simply refuse to review something because I don't like the way it is written is a disservice to everyone and inexcusable. As long as the work is honest in the end, then everything should be published, be it from a fancy-ass HHMI lab or from an obscure dude in Belgium. If a paper is truly crappy, it will sink into obscurity on its own.

  • Paul Murray says:

    Let's say you are an english speaker, and someone submits an article written in german to you for review. Now ... are you supposed to struggle through as best you can? Of course not - you just don't speak german, that's all there is to it. English that is so ungrammatical that it cannot be parsed is in a similar basket.

  • Anonymous says:

    What wrong with, please send me this fucking manuscript again for me to fuck it over when it is written in fucking English - The Third Referee.
    OK a bit rough, but if you want to be nice, owing to the poor writing I was unable to review this manuscript. When rewritten I would be pleased to review it.

  • Alex says:

    If you find yourself having to fix most of the sentences for basic issues of grammatical correctness (not just stylistic improvement) then you should probably just make it clear that rejection is merited on those grounds alone.
    It may be that you can slog through it and offer an assessment of the methods, data, and conclusions, but you will do a better job of assessing these things if the manuscript is an easy read. It's only fair to refrain from weighing in on these matters until the authors have presented you with a clear and readable description that doesn't have you instinctively fixing every sentence (which is distracting) and forming (perhaps subconscious) negative impressions of them based on language alone.

  • apallaian says:

    I knew of a prominent Pathologist at the NeuroAcademy of Highest Science, who was extremely careful in reviewing manuscripts for scientific content and yet he was rather condescending with imperfect use of English. His tolerance came from his exhilarating knowledge of languages. His best-known and practiced language was Obscurantism, who he perfected along his Academic Career as a High Rank Administrator. He loved Victor Hugo’s first collection of poems “Odes et Poesies Diverses” that he enjoyed and read during his long hours of biopsy examination at his Electronic Microscope. His Silver and Gold Staining Techniques served him as much as “Odes et Poesies Diverses” served Victor Hugo in gaining a Royal Salary and Pension From Louis XVIII and Dean II respectively.
    His collegial nature and dedicated work to instill his school of thought in his trainees, subordinates and immediate authority was not always successful. Yet, by the end of his pathological career, he was awarded the "Gold-Headed Little Dog", joyfully shared and appreciatively handed over with the help of his faithful followers, in recognition for his Gold Staining Techniques.

  • BB says:

    If you can't understand it, you can't review it fairly. I ask for edit and resubmit.

  • lylebot says:

    Bad writing is one thing; "not written in readable English" is another. I'm guessing that some of you who are saying you review the paper even if it's written badly have never seen a paper that reads like it was written in one language and then put through Google translator before submission to an English-language journal. I got one of those once. I really did try to review it, but it was simply not possible to understand it beyond the 4th page or so. In retrospect I put in way more effort than I should have.
    Some submissions really are literally unreadable.

  • I've struggled with this in the past as well, but I will generally review something if I can get through it, then recommend it be heavily edited. As a grad student and post-doc often went our of my way to comment on and help correct grammar and spelling from non-native speaking authors, but I just don't have that kind of time these days and I have usually have at least one manuscript on my desk to review at all times.

  • Even if the language is bad, it's seldom bad to the point of making the reviewer unable to understand what the manuscript is about. If a reviewer refuses to review on the sole basis of the language, it's usually an indicator of the reviewer's lack of relevant expertise, or plain laziness.
    Disagree. Reviewers (even experts in the field) are not mind readers, and if someone slaps up some shitty graphs along with shitty text, there is often no way to intelligibly review the manuscript. This has nothing to do with lack of expertise or laziness and everything to do with the fact that a reviewer shouldn't have to play detective by spending hours and hours just to decipher what the fucking manuscript is even about. Sometimes language (or lack thereof) is so bad that it prohibits comprehension of the manuscript without an exorbitant amount of legwork.
    From my perspective, this really should be something that is done at the Editor/Associate Editor level. If I receive an article to review, I assume the Editor/AE has deemed it readable and worthy of a review.
    Absolutely, but in my experience, this doesn't happen. Not sure why editors aren't more willing to nip this phenomenon in the bud.

  • First off, I have read submissions where the English was horrid. I slogged through them, but as I said in an earlier post ... when that journal came knocking again, I told them to forget it and that if they wanted me to review for them, they needed to take better care of the quality of the stuff they sent me, because I have work of my own that needs doing and I can ill afford to translate text AND do a review.
    Secondly, as a bit of an aside, I was at the American Society of Agronomy (ASA) meeting this past year, I attended two sessions geared towards early career scientists entitled (I'm paraphrasing) "How to write a manuscript" and "How to get your manuscript published in an ASA journal". They were quite informative, and as a reviewer for ASA journals, I was especially interested in hearing what they had to say in the second session. They spent a fair allotment of time discussing English. They made it quite clear that the paper would not even get past the Editor/AE if the English was not up to par and they said they would gladly refer scientists to professionals who knew science and would edit manuscripts written by people who did not have English as their first language. The editors of the ASA journals made it clear that a 21 day turn-around from submission to notification of acceptance/revision/rejection was their benchmark, and I can confirm that they're spot on. If you're in agriculture and you want a quick return on your manuscript, you send it to an ASA journal. Part of that is because their Editors/AE are very involved in the process.
    And lastly, if editors are not "more willing to nip this phenomenon in the bud" then it's time to get better editors. This is something in which the community needs to be proactive. If a particular editor constantly sends me manuscripts which suck, I'll stop reviewing for them. If more people started doing that, perhaps the editors would take notice and start doing their job.

  • David says:

    My reaction (as a reviewer) to bad writing depends on the other determinants of quality and also where the paper came from.
    A paper on an interesting topic, describing a worthwhile study, or with notable data is deserving of some help if the biggest failing is the quality of the writing. A boring paper describing, say, a review of prior publications, is not deserving of slack. When I see a potentially good paper from a foreign center, I might spend a lot of time making specific suggestions on grammar and phrasing. When I see poorly written pig slop from a U.S. center, with a senior author I know to be fluent in English, I am merciless.
    About 10 years ago I sent in this comment (more or less) for peer review: "Demosthenes practiced oration with stones in his mouth. His intent was to improve his diction so that others could understand him better. The current authors should remove the stones from their typewriters. Unless they write more clearly, they cannot be understood."

  • bsci says:

    I'll note that I've never seen a manuscript that was so bad that I couldn't get the general gist of what they were doing. I'm sure such manuscripts exist, but that's my sampling bias.
    I wanted to add to this discussion that there are two types of bad writing. I often see mistranslation of words and grammatical rearranging of words when someone isn't speaking in their primary language. While those make it harder to read a manuscript, rarely have they gotten in the way of generally deciding if the science is worth a re-review.
    There are other manuscripts where bad grammar is mixed with poor writing and sloppy organization. These manuscripts may or may not be written by a primarily english speaker, but adding second language issues on top of them can make for unpleasant reading.
    Still, I've usually found it possible to distinguish bad grammar from bad writing. I'll try to make a few scientifically relevant comments in a review but if I can't properly review the methods section for scientific accuracy I can't review the paper. Depending on my mood and whether it looks like the work has the potential to reach publication quality will define if I reject or ask for major revisions.

  • Jim Thomerson says:

    Some years ago, I reviewed several papers from a young colleague whose first language was not English. I did considerable editing on the manuscripts. He accepted my editing and his use of English improved rapidly. In later years, he reviewed several of my publications without criticizing my English, just the science.
    In this case, I thought it a benefit to me to help the fellow with his English useage.

  • apallaian says:

    @ David
    You might be right that stones need to be removed from typewriters as to allow to be understood. Well, here you are !
    I have to congratulate the author (s) of “A videographic primer on how to respond “ because is almost the exact representation of a High Rank scientist Administrator, whose primary language is not English. His English should be almost perfect since he has been practicing High Neuroscience at the American Academy of NeuroMoney for almost half a century; unfortunately his English is far from perfect. On the other hand, you don’t need English to practice academic neuroscience; you need Money and the perfect language this scientist-turned money seeker speaks and keeps practicing is obscurantism, written and spoken in “palaces of intrigue”.
    These palaces of intrigue are not laboratories where you breathe curiosity and fascination for brain function and dysfunction. They are “Stock Exchange Dark Rooms” where the High Rank money maker meets with associates to discuss the works of rich followers and make secret plans on destroying the reputation of equal rights supporters and kill peasant protesters without leaving a trace.
    The High Rank neuropathologist, with greek accent and pseudoaristotelian background, has a powerful technique microscopy/histology based, who he has developed in association with the neurocelebrity couple under the auspices of an endocrinology-based Dean of the Academy and his female assistant executive of faculty affairs. The technique has not been patented yet but was designed for high and strong impact. The Technique is called: Connections for Gold Staining. Basically, they look for connections, not only neural connections, but also in the physiological/psychiatric “bodies of knowledge”. Once the connections have been established, the Death gene in a specific human subject is turned on by Gold Staining. This Gold Staining has nothing to do with Ramon y Cajal methodological approaches. It is just an ancient technique, imaginary based, aimed at destroying peasants’s professional future and reputation.

  • PostDoc says:

    A procedural note: some of you are misusing the word "triage." Triage does not mean "deprioritize" or, in this context, "refuse to address." Triage is a more general term that means basically "to sort according to a system of priorities" (www.m-w.com). It doesn't imply sorting out or sorting in - just sorting.
    A specific example from DrugMonkey's text: "I also work on getting them to understand that it is perfectly okay to triage a complete piece of garbage." Yes, and we triage the gems, as well. It all gets sorted.
    I suggest as an alternative the term "triage-out" or the more conventional "reject."

  • A Reader says:

    PostDoc -
    NIH panels generally sort proposals into two groups at the start of the meeting -- the better ones, which will be discussed and given a priority score, and the worst ones, which will not be discussed at panel meeting, and not scored. This saves everyone wasting time on proposals that have no chance of getting funded. This process used to be called 'triage', per general use of the term. However, a grant not discussed and not scored eventually came to be called 'triaged', and thus among NIHers, triage became synonymous with 'removed from further consideration', or as you suggest, 'sorted out', or 'summarily rejected'. Nowadays, the official term is 'streamlined', or simply 'unscored'. But use of 'triage' per DM persists.

Leave a Reply