Representation in manuscript review

Feb 25 2016 Published by under Diversity in Science, Science Publication

One of the things that determines success in science careers is the opinion ~three peer reviewers have about your manuscript as offered up for publication in a given journal.

Hopefully I do not have to rehash the way that journal identify of a scientist's published work affects career success.

Hopefully I do not have to rehash the way that bias creeps into what otherwise is supposed to be objective analysis.

And let us leave your well-intentioned, but hopelessly naive calls for blinded peer review aside until that nirvana is reached.

Do you think about reviewer diversity at all? Many journals publish a year-end list of all reviewers (these don't say how many each reviewer wrote, of course). Have you ever scanned them for, say, gender balance? If you are an AE or EIC....does diversity* concern you?

On the author side, would you work to ensure your suggestions for potential reviewers are not biased? Do you ask for about as many women as men? Does ethnic or other minority characteristic of your suggestions play a role?

I'm guessing the answer is no?

I have taken to trying to suggest equal numbers of male and female reviewers when I submit a manuscript. This is pretty simple in my fields of work, so long as you think about it.

Other forms of representation? Not really possible, is my first thought. But....now I'm thinking about it. Maybe I'll put a few people on my usual lists that I do not typically consider.

And when I get a chance I'm going to go through those published reviewer lists. I'm curious how the journals I think of as being in my field are doing.


___
*Editorial boards are another place to look, those are published.

39 responses so far

  • AcademicLurker says:

    For our last PNAS paper we revised our list of suggested reviewers when we realized that the initial list was all men.

    And it got in, so yay gender diversity.

  • drugmonkey says:

    I hope more people come to this realization...or don't need to. I realized at one point that it wasn't on my radar and that it would be simple to go to at least 50/50 as my standard.

  • Newbie PI says:

    If there are fewer female professors overall (particularly at the level of full professor), why would you expect there to be a 50/50 gender balance of reviewers? Wouldn't that mean that the women have to take on more work than the men?

  • odyssey says:

    I haven't done this nearly as much as I should as an AE (except with regard to career level) or author. I will now.

  • drugmonkey says:

    NPI- I request 50/50 for manuscript review. If I were trying to assess a journals' record of course we should make allowances for representation in the relevant field.

  • whizbang says:

    My suggseted reviewers are usually overwhelmingly female, just because of the nature of my work.

    I do scan editorial boards for gender imbalance, I have informed some editors they needed work.

  • Newbie-Ish says:

    DM - in what fields aren't women underrepresented? I think the extra work burden [NPI] is a serious consideration. This is a particular concern with manuscript review, where the reviewer is basically donating their time. Also, I agree conceptually with equal representation, but we need to keep in mind that gender bias is not fixed when women are the ones doing the evaluation; we possess bias against our gender, as well. So it is an important part of the problem but not necessarily a complete solution.

  • drugmonkey says:

    Review is opt-in.

  • Newbie-Ish says:

    Opt-in does not occur at equal frequency between genders. It is *very* well acknowledged that female academics are both overburdened with service requests, which reduces their availability for tasks that actually matter. It has also been shown that female academics bear a higher fraction of professional/laboratory housekeeping duties.

  • drugmonkey says:

    It is disturbingly paternalistic to suggest we should not extend the *opportunity* on this basis.

  • drugmonkey says:

    You are also very mistaken about the role participating in manuscript review plays in the arc of an individual scientist's impact and influence.

  • poke says:

    I definitely think about this.

    There was an editorial or other such article in science or nature a few years ago that had a helpful suggestion: any time you're making a list of scientists for any purpose (conference/panel presenters, paper reviewers, etc), make your list, and then take a moment to consciously think about your candidates with an eye towards diversity. I've found that often the initial list of names I come up with is biased in all the ways you'd expect. But taking just a few moments to think about the field as a whole usually bring to mind a number of underrepresented people whose work and expertise I have equal respect for, but who just didn't pop into my mind as easily as people on the first list.

    It sort of sounds like a dumb heuristic on its face, but I've made it a habit and found it really useful. ymmv.

  • Dave says:

    I think about it primarily in the form of career stage representation, as always. I like to get reviewed by people who understand what it means to me to request multiple additional experiments, for example.

    But definitely food for thought here...

  • drugmonkey says:

    people who understand what it means to me to request multiple additional experiments,

    Are you implying that differential capability* of the laboratory to complete the additional experiments should affect paper review comments and/or acceptance at a particular journal?

    *perceived/assumed

  • Dave says:

    Errrr, yeh maybe I am. What I'm saying is that I like the idea that a reviewer who is at a similar career stage may think twice about certain review comments/requests. Perhaps they might be more understanding that certain things may put a younger lab under a lot more pressure financially. I dont know....I know this might be more fantasy than reality as it is fairly well recognized that post-docs can be the worst people to review your manuscript!

    I'm sure most of us have also seen editorial comments to senior PIs saying something like:

    "As you will see from the comments below, the three reviewers request that a number of key experiments and data be added to the manuscript before it can be acceptable for publication in journal X. I am aware that you likely have most of these data on hand already and so these requests shouldn't be too time consuming for you. As such, I would like to invite to submit a revised version of your manuscript, incorporating these changes......"

    Just seems like bias can creep in.....

  • Insect Biologist says:

    Newbie-Ish has some really good points. As a woman in a male-dominated field, I seriously hope that authors in my field do not start sending in reviewer suggestion lists with a 50/50 gender balance. I am pretty good at protecting my time, but there's a psychological cost to saying no all the time.

  • drugmonkey says:

    Errrr, yeh maybe I am.

    I think that is outrageous.

  • Dave says:

    LOL!

    Why?

  • Jonathan Badger says:

    Of course looking for female representation in names is somewhat Eurocentric. I doubt many Westerners can easily figure out what is a "male" or "female" Chinese name. And Chinese are forming a growing number of scientists publishing papers and need to be considered as reviewers as well.

  • Newbie-Ish says:

    DM - if women are represented at 10% in a field, the way to get parity is NOT to pelt them with 80% of service requests. The misery. You're hearing from women here (loudly) that we are sick of being tokens because it is a burden on our time. Not all service is created equal, and, yes, you do get something out of manuscript review, but it is not uniformly an honor, particularly if you're talking about low impact journals. I'm early in my career and reviewing for lots of low impact journals and OMG OMG the time suck. It is an absolute waste of time, but I do it anyway. Maybe I know the handling editor. Maybe they cited one of our papers and I want to help make the paper better. Most often I go in with a glimmer of hope that I'll gain something out of the experience (techniques, how to talk about science, whatever), and I am disappointed. When you're new and making a name for yourself, nobody's sending the good stuff your way. Either way, I never in a million years would want the task and I am only doing it because I feel obligated.

    Enforcing gender parity in review requests is kind of like cutting of your nose to spite your face. The issue is important, but the proposed solution only creates more burden on the already burdened gender.

    Now conference talks? Invited reviews? Editorial boards? Totally. Those are visible service tasks that go far to promote an individual's career and to change culture/perception in the ranks. Publish those stats, damnit, and make sure the numbers improve.

  • drugmonkey says:

    Why?

    Because I think that the assessment of whether a paper is 1) of good quality and 2) of sufficient impact/importance/pizzazz/interest/etc for the journal at hand should depend on what is in the manuscript. Acceptance should depend on the work presented.

    The further tactical problem with trying to determine what is or is not possible/easy/quick/cheap/reasonable/etc for one lab versus another lab is that you are inevitably going to be wrong. A lot. How do you know what financial pressures are on a given lab? How do you know, by extension, what career pressures are on various participants on that paper? Why do you, as an external peer reviewer, get to navigate those issues and again, what bearing does it have on the data?

  • drugmonkey says:

    JB- you are making idiotic excuses. Datahound uses the web when he can't figure it out and that adds additional people into the dataset. you can exclude if necessary if you are trying to retrospectively assess performance. Stop trying to make perfection the enemy of better.

  • Newbie-Ish says:

    This is university service, not society service, but you get the point:

    https://www.insidehighered.com/news/2011/01/12/new_study_finds_unequal_distribution_by_gender_in_academic_service_work

    A good example more pertinent to the link above: faculty search committees. Ideally, serving on a search committee is to my professional benefit. In practice, I get picked for this task when a woman is interviewing and they want to look good. And when that scientist's research has nothing to do with my research, it takes away from my research hours. It makes me feel devalued by my institution. And, without fail, it takes me away from my kids for a whole night. It's the trifecta of exhausting and turning off the very demographic you are trying to promote.

  • drugmonkey says:

    I'm early in my career and reviewing for lots of low impact journals and OMG OMG the time suck.

    say no. I don't know why it is so hard for noobs to listen to this advice which every fricken body gives them. You have to strategically harbor your time. I am not seeing how encouraging journals to seek better gender parity through their *requests* is some huge problem here. YOU. CAN. DECLINE. ANY. REQUEST.

  • scientific editor says:

    Reviewing papers is definitely a burden, but also an opportunity. An opportunity to see the latest pre-publication science, to influence the field, and to become known by a journal editor (hopefully in a good way). If any scientist is offered too many such opportunities, then he or she can become choosier about which ones to accept. A female senior faculty member I work with often mentions how she "trades up" obligations: If asked to serve on a committee that she feels will be a good use of her time and might benefit her career (e.g., being on the search committee for a new dean), she may take that opportunity and remove herself from another lower-level obligation. I think the same could be true of reviewing papers. More requests to review = more opportunities to "trade up".

  • drugmonkey says:

    This is university service, not society service, but you get the point:

    No, I don't. apples and oranges.

  • Dave says:

    Fair enough. I'll accept that and will concede that my thoughts are not well formed on this point. But often points (1) and (2) are bundled in with how the paper might look with this really cool piece of additional data in revision. Reviewers frequently review the paper they want to see, not the one they are reading. I just think somewhat nefarious requests for more data hit younger investigators harder. When it's critical for the paper, that's one thing. But I'm in no way saying standards should be lowered for young investigators.

  • AcademicLurker says:

    DM, don't you think that a BSD with an army of postdocs at her/his command might be more likely to request a bunch of additional experiments to "increase the impact" of the manuscript (i.e., not strictly necessary to make the case the manuscript is making) without thinking about what sort of burden that will impose on groups that don't have BSD level resources to draw on?

  • drugmonkey says:

    Reviewers frequently review the paper they want to see, not the one they are reading.

    and it is my view that this is to be stamped down upon. This is GlamHumping talking and has all sorts of bad effects.

    I just think somewhat nefarious requests for more data hit younger investigators harder.

    Define "nefarious". The whole escalation of demanding additional experiments is the result of the powerful and well-heeled getting an advantage from excluding those less fortunate. Also see "the latest and greatest technology". This, as always, is not always due to personal moustache-twirling ill-will but is an emerging property.

    Sure, younger investigators are, on average, going to have fewer resources, less risk-tolerance, a higher opportunity-cost of spending precious grant dollars on the extra bit that ends up in Supplemental materials and take a harder hit for delays. But you can apply this rationale to some very well funded labs who have postdocs seeking jobs, promotion steps, grants under review...etc. Lots of people are negatively affected by letting reviewers demand endless additional experiments.

    I just don't see how letting reviewers and AEs decide who deserves a break and who does not is anything other than an opportunity for more bias and nefariousness.

  • drugmonkey says:

    AL- Sure, it is possible. But in my view, the personal intent for making such demands for additional work is irrelevant when we are talking about how a system should be structured.

  • Newbie-Ish says:

    How is it apples and oranges? Someone is asking me to do something. If I say no, I lose something professionally. I therefore feel obligated to say yes.

    Look, I'm not saying we should ask journals to decrease review requests to women. I'm saying that increasing review requests to women merely creates additional problems for junior/early career women. It's a silly way to try to promote gender equity because - whether we refuse the requests or not - you are asking us to do more grunt work and women are *tired of being asked to do grunt work*

    I also think you are forgetting what's it like to be junior, regarding manuscript reviews. When I was training in my giant-name-PI lab, the papers I reviewed were high quality and cutting edge. I learned a hell of a lot. What I review now is crap. 2-3 out of every 4 papers I review is the kind of work that I would actually fire someone over (controls, people; posthoc testing; not difficult). 1 or 2 might be decent papers. 1 out of the last 20 manuscripts I reviewed was by a leader in my field. 20.

    And you won't know if it's a good manuscript until you agree to review it.

    Plus, if we have tier 4 = paper dump journal, tier 3 = mixed bag, tier 2 = high quality field specific journal, and tier 1 = CNS... I still want to publish in tier 3! I hope to publish in tier 3.

    For all of the reasons above, I accept review requests from any journal that is at or above the tier I am trying to publish in (and otherwise related to my expertise).

    To be clear: I'm OK doing the reviewing. I am knowingly agreeing to do the reviewing. This is what I should be doing at my career stage. But this is why I do *not* want to see enforced gender ratios in review requests.

  • drugmonkey says:

    I also think you are forgetting what's it like to be junior, regarding manuscript reviews. When I was training in my giant-name-PI lab, the papers I reviewed were high quality and cutting edge. I learned a hell of a lot. What I review now is crap. 2-3 out of every 4 papers I review is the kind of work that I would actually fire someone over (controls, people; posthoc testing; not difficult). 1 or 2 might be decent papers. 1 out of the last 20 manuscripts I reviewed was by a leader in my field. 20.

    I can't really assist you with your dismay over your fall from grace. What I can say is that I have always, right up through this week, accepted or declined invitations to review manuscripts based on the same principles. Do I have the time? Am I interested? Are there implications for my career in saying yes/no to this particular journal or Editor?

    And you won't know if it's a good manuscript until you agree to review it.
    well....yeah, that is kind of the point. but in your case it seems like if you are bummed about no longer getting the Glams and everything else is shit...there are larger issues at work here.

    But this is why I do *not* want to see enforced gender ratios in review requests.

    I am not seeing where it is such a huge burden on you to refuse review requests when you do not have the time or interest. Here's a hint. AEs get refused all the time and it would be silly for them to take it personally if someone they selected as a potential reviewer declined. We're all busy. AEs get that.

    This differs considerably from the expectations of Chairs, Fulls and Deans of your University in their expectations of your Service leg on your way to tenure.

  • odyssey says:

    AEs get refused all the time and it would be silly for them to take it personally if someone they selected as a potential reviewer declined.

    This. I'm an AE. Way less than a quarter of the people I ask to review accept. So what? I say no to way more than three quarters of the review requests I receive. Why should I expect the people I ask to do more than I would?

  • Newbie-Ish says:

    My meaning was clearly not communicated properly. I am not "bummed" about anything. Getting cutting edge stuff would be fun. One day maybe I'll review work like that, but I am not expecting it. I am saying that you have forgotten that the *value* of manuscript review changes with career stage. For those of us just beginning to make a name for ourselves? The benefits of manuscript review are minimal at best. That's just how it is. I spend most of my time politely explaining about writing hypotheses, positive and negative controls, appropriate statistical methods, etc. This does not benefit me. I do it because I love science and I love helping make it better - quite literally the definition of service. I am not astonished about the 20 because I think I deserve better but because 19 reviews were giving and only 1 was getting.

    The reason it is a burden to refuse requests is that I basically have to begin the work of reviewing a manuscript if I am going to discern between no-name-lab-quality-paper and no-name-lab-bad-paper. There is no easy way to do this other than digging into the science. And that takes time. It's a burden.

    I appreciate the comments that AEs don't take it personally. Actually, that's really helpful. Because wanting to be in good AE favor is generally why I say yes. However, that still doesn't solve my problem. I reviewed 20 manuscripts: 1 was cutting edge, maybe 4 were reasonable scientific studies, and 10 were just a lot of teaching other people some of the basics of hypothesis testing.

    This started with the idea that publishing disparity could be reduced by enforcing equity in reviewer demographic. I stand with the statement: programs that assign additional service work to disenfranchised academics at a junior stage does not benefit them. You can keep beating your head against a wall that we should be "free to reject requests" but the statistics tell the real story: we're not really free, no matter the underlying reason, we bear more grunt work than the majority. There are better ways to go about solving the problem.

  • David says:

    As an AE for a journal that doesn't accept author suggestions for reviewers, I am only looking for reviewers I think are 1) capable and 2) willing. If I had a long list of potential reviewers (which I hope to have one day), then I might start trying to have a more diverse set of reviewers for a given paper, but that sure sounds like a luxury.

  • Newbie-Ish says:

    Oops, apparently I can't do math. 1+4+*15*, not 10.

  • qaz says:

    Newbie-ish - set a limit to the number of reviews you are going to do. (Some say one a month or one a week or one at a time. Whatever will not interfere with your work. You are going to be graded on your science, not your reviewing. Your reviewing is a favor and will only get you favor points back.). When you reach your limit say no.

    I actually do this with everything, including giving talks (may you have the problem of being so popular that you have to turn down talk opportunities). I even do this with grant getting. Once you have the money you need to do science, turn down grant opportunities and do the science. Otherwise, you fall out of balance.

    Of course, there are always exceptions. Being invited as a junior prof to share the stage with a BSD in some big-name comference is probably worth stretching for. Similraly, having a glam-mag editor beg you to do a review is probably worth the favor points.

    But set your limits and remember that you need to do what's best for you.

  • bethann mclaughlin says:

    Hi Dr. DrugMonkey,

    I just moved up with JNeuroscience from Ass'c Editor to Reviewing Editor, so I'm just now getting access to more data and the meetings where we talk about these things. I would likely say that my reviews skew towards having more females. Again, my experience is unique to me, and I can try to get numbers for you. I know @marinap63 is interested in this as well.

    I put a lot of work into reviews because they gave me a sense of service and science awareness beyond my university at a time when I couldn't travel much (wee ones), so I would wager that's why the group of women I invite accept reviews and turn in quality work.

    Another informal thing I've been beta testing is pulling in third reviewers where they don't have a long history of reviewing for us. Rather than the traditional 'tie breaker' role of the third reviewer, this job this as an opportunity to step into a new role with a bit of a safety net, get feedback and be mentored by other reviewers (if only by seeing their comments) as well as by direct feedback from me. My understanding is some other Reviewing Editors are doing this as well

    We have also gotten very good feedback using the Twitter account @jneuroscience irreverantly but earnestly run by editor-types. We've had postdocs reach out and offer to help with reviews and their names were forwarded to editors.

    I'm not sure how we could address the people of color diversity issue as we don't have metrics on this that I know of, but I will ask.

    I think I can speak for Marina and myself when I say we thrive on trainees coming to us at national meetings and introducing themselves and talking about their passions. The new trainee-only Journal Club format gives these folks a chance to practice writing and synthesizing a bit of literature in a structured and productive way.

    When you see me or any other editor type at SfN or another national meeting, we have ribbons on so you know who we are and can say hi, show us your poster and connect as mentors and colleagues. (There was a rumor we had all won 'best in show' competitions to get ribbons, so I wanted to clarify this is not why we are given ribbons)

    There's too much pressure on women and people of color already, but we hope that we can work harder to ensure they are part of our future in the journal. They are a hugely untapped talent pool ,and science needs talent. And better boots. And maybe kittens.

    Hope this is helpful and thanks for blogging!
    BethAnn

  • drugmonkey says:

    Thanks for your thoughts, and your efforts on behalf of JNeuro. Sounds like you've been thinking these issues over which is likely more than most journal staff do.

Leave a Reply