You came so close, Nature Editorial staff.

Aug 20 2014 Published by under Fixing the NIH, NIH Careerism, NIH funding

A news bit in Nature overviews Richard Nakamura's plans to investigate the disparity in grant funding that was identified in the Ginther report in 2011. See here, here, here for blog comment on Gither et al. Nakamura is, of course, the current director (pdf) of the Center for Scientific Review, the entity at NIH that conducts the peer review of most grant applications that are submitted.

It is promising-ish. Nakamura's plans are summarized.

One basic issue that the NIH will address is whether grant reviewers are thinking about an applicant’s race at all, even unconsciously. A team will strip names, racial identification and other identifying information from some proposals before reviewers see them, and look at what happens to grant scores.

Hope they check on the degree to which the blinding works, of course. As you know, Dear Reader, I am always concerned that blinding of academic works cannot always be assumed to have functionally worked to prevent the reviewer from identifying the author or lab group.

The NIH will also study reviewers’ work in finer detail, by analysing successful applications for R01 grants, the NIH’s largest funding programme for individual investigators. The goal is to see whether researchers can spot trends in the language used by reviewers to describe proposals put forward by applicants of different races. There is precedent for detectable differences: in a paper to be published in Academic Medicine, a team led by Molly Carnes, a physician at the University of Wisconsin-Madison, used automated text analysis to show that reviewers’ critiques of R01 grant applications by women tended to include more words denoting praise, as though the writer is surprised at the quality of the work.

Very intriguing contribution to the analysis. Nice.

The NIH will also analyse text in samples of reviewers’ unedited critiques. The Center for Scientific Review typically edits the wording and grammar of these reviews before grant proposals are returned to applicants, but even the subtlest details of such raw comments might hold clues about bias. Nakamura says that reviewers will not be told whether their comments will be analysed, because that in itself would bias the sample. “We want them to be sloppy,” he says.

Hmmm. I guess this is just human factors checking on the automated analysis. Together they are stronger.

The NIH’s Study Sections, in which review groups discuss the top 50% of grant applications, might also harbour bias: the 2011 Science paper found that submissions authored by African Americans are less likely to be discussed in the meetings. But when they are, a negative comment arising from even one person’s unconscious bias could have a major impact in such a group setting, says John Dovidio, a psychologist at Yale University in New Haven, Connecticut, and a member of the NIH’s Diversity Working Group. “That one person can poison the environment,” he says.

This is not presented in a context that suggests the NIH plans to investigate this directly. Not sure how this could be done without putting a severe finger on the balance. I mean sure, most meetings have call-in lines open to Program staff and Nakamura could just record and transcribe meetings but.....reviewers won't like it and if you warn them you scare off the fishies. So to speak.

Nakamura expects that the NIH’s effort to identify and root out prejudice, which he says could cost up to $5 million over three years, might prove controversial. “People resent the implication they might be biased,” he says — an idea borne out by some responses to his 29 May blogpost on the initiative. One commenter wrote, “It is absolutely insulting to be accused of review bigotry. Please tell me why I should continue to give up my time to perform peer review?”

But Nakamura believes that the NIH — and reviewers — need to keep open minds.

He, and media covering this, need to focus on the opportunity to communicate that institutional racism does not hinge on whether individual actors are overtly biased. The piece leads with the comment that Nakamura got his butt over to the Implicit Association website and identified his own biases to himself. This is the sort of introspection that needs to happen. Heck, I'd love to see a trial where study section reviewers were told to go over there and complete a few tests prior to receiving their grant assignments*.

The Nature editorial is, for the most part, on the side of goodness and light on this.

The idea that scientists who volunteer time and energy to review NIH grants could be biased against qualified minority researchers is a tough pill to swallow. The NIH is to be commended for not sweeping this possibility under the rug: it has turned to the scientific method to investigate the suggestion.

good, good.....

It is a topic that the NIH will need to broach delicately. Few academics consciously hold any such inclinations, and fewer still would deliberately allow them to affect their grant evaluations. Some are likely to bristle at what might be seen as an accusation of racism, and the NIH plans to conduct at least some of its studies of grant reviews without the reviewers’ knowledge or consent.

But better for the NIH to offend a few people than to make snap judgements and institute blunt policies to address the problem. Fixes such as increasing scholarships and training for minority groups would no doubt be a good thing, but they could be an unhelpful use of money if they do not address the root cause of the disparity.

yes, yes, excellent.....

And policies such as grant-allocation quotas could come at the expense of other researchers.

BZZZTTTTTTT!!!!

No. Bad Nature.

Right back to victim blaming. Right back to ignoring what it means to have a BIAS identified. Right back to ignoring what the nature of privilege means.

Those "other researchers" at present enjoy a disparate benefit at the expense of AfricanAmerican PIs. That's what Ginther means. Period. The onus shifted, upon identification of the disparity, to proving that non-AfricanAmerican PIs actually deserve their awards.

Ginther, btw, went a long ways toward rejecting some of the more obvious reasons why the disparity was not in fact an unfair bias. Read it, including the supplementary materials before you start commenting with stupid. Also, review this.

But there is also this. The low numbers of AfricanAmerican scientists submitting applications to the NIH for funding means that any possible hit to the success rate of non-AfricanAmerican PIs would be well nigh undetectable. A miniscule effect size relative to all other sources of variance in the funding process.

Another way to look at this issue is to take Berg's triage numbers from above. To move to 40% triage rate for the African-AmericanPI applications, we need to shift 20% (230 applications) into the discussed pile. This represents a whopping 0.4% of the White PI apps being shifted onto the triage pile to keep the numbers discussed the same.

These are entirely trivial numbers in terms of the "hit" to the chances of White PIs and yet you could easily equalize the success rate or award probability for African-American PIs.

It is even more astounding that this could be done by picking up African-American PI applications that scored better than the White PI applications that would go unfunded to make up the difference.

And of course "grant-allocation quotas" are precisely what the special paylines and other assists for ESI investigators consist of. Affirmative Action for the young and untried.

Did we get this sort of handwringing, call for long-duration "study" of the "true causes" of the disparity?

Hell no.

The NIH just started picking up ESI grants to balance the odds of funding, even when study sections responded to news of this affirmative action by further punishing ESI scores!

So yeah, my call is for the NIH to balance the funding rates first, and then do all their fancy studies to root out the "real cause" later.

Also for editorial teams like the one at Nature to stop repeating this whinging about those who already enjoy disparate privilege who might lose (some of) it.
__
*yeah, it might backfire. that would itself be interesting.

14 responses so far

  • Comradde PhysioProffe says:

    Your analysis makes sense. Just one quibble:

    I mean sure, most meetings have call-in lines open to Program staff and Nakamura could just record and transcribe meetings but.....reviewers won't like it and if you warn them you scare off the fishies. So to speak.

    Reviewers are never told who from NIH is listening on the teleconf. At least not on any of the numerous panels I've served on.

  • drugmonkey says:

    I would swear we're always told "Program staff are on the line". Point being that if CSR put in a line for transcription it would not cause any notable difference for reviewers to cue in on.

  • Joe says:

    Getting discussed makes a big difference. If the chair of the SS could be told which were the applications from minority applicants, then those applications could be included in the discussed group.

  • drugmonkey says:

    I am skeptical. I've been through review where "the burden of triage cannot fall on.." rules meant discussing ESI applications, R21 and R03 applications for which the preliminary scores put them over the triage line (based on the R01 apps' preliminary scores). The major input has already been applied at that point. Forcing the reviewers to discuss the app doesn't really help much.

    Now true *post review* outcome of discussed/triaged application can mean a great deal. It permits Program to pick it up a lot more easily. It may possibly come in to play when the revision comes back for review, (particularly if assigned to new reviewers who don't know it was pulled up for discussion out of the triage stack for affirmative action reasons).

  • pentahedron says:

    my call is for the NIH to balance the funding rates first, and then do all their fancy studies to root out the "real cause" later.

    Balancing needs to happen, fancy studies need to happen, but the focus seems to be on the panels. What about the POs and SROs handling these situations now. They're humans. Doesn't the failure to discuss potentially meritorious applications from African-American PIs imply the possibility of bias by the SROs who are making the call on whether to discuss borderline cases or to follow up with reviewers when reviews strongly conflict? Or even at the level of making sure there is an appropriate time for panel members to pull up ND apps (ie NOT when everyone is tired and pissed off and ready to head out for dinner or whatever)... etc?

  • drugmonkey says:

    the possibility of bias by the SROs who are making the call on whether to discuss borderline cases or to follow up with reviewers when reviews strongly conflict?

    The latter is a possibility. The former is not, IME, a direct responsibility of the SRO.

    Or even at the level of making sure there is an appropriate time for panel members to pull up ND apps

    My view is that the decision years ago to start reviewing apps in order of initial priority score instead of whatever random order was caused by trying to put apps assigned to a single PO together (or whatever) is what seriously squelched pulling up apps for discussion. I absolutely detest this difference.

  • Joe says:

    My experience is that there are 2 chances to decide what is and what is not discussed. Once at the beginning of the first day (to set the line) and another near the end of the second (to give another chance for people to pull up favorites or to move some applications out of the discussion set). At either time you can just ask that an application be discussed. No reason needed. I would not think that any reason need be given in these situations either.
    I think getting the applications discussed will help because the applicants will get a full review including a summary of the full discussion, thus helping with the plans for revision. Also, having your application get triaged is terribly discouraging, not what we want in these cases. And as mentioned, having the application discussed can make it easier for program to pick it up.

  • becca says:

    I agree with your identification of the problem and would fully support your implementation of a solution. Yes, we can compensate for biased reviewers and we should do so.

    That said, there are various reasons we *also* need good strategies to reduce implicit bias. Googling about this, I found a document from the National Center for State Courts about how to reduce implicit bias (I'm frankly surprised and pleased to find out someone is actually thinking about how these issues play out in the context of our justice system). My best guess from the summaries they cite is that requiring reviewers to do an implicit bias awareness exercise could actually help. But there are a lot of other things- time pressure brings out bias in decision making. That's a practically useful thing to know. Actually, it seems like many of the things you want to avoid to reduce bias (negative emotional states like anger and disgust) could actually result in warmer fuzzier review panel experiences. If you can demonstrate that you get a *reduction* in bias if you treat reviewers well, that may change the tone of discussions about how review panels should be run. It may or may not help the politics of "new requirements to reduce bias" if they subjectively make sitting on review panels more appealing.
    http://www.ncsc.org/~/media/Files/PDF/Topics/Gender%20and%20Racial%20Fairness/IB_Strategies_033012.ashx

  • toto says:

    Nakamura et al. are trying their best to be pedagogical about it.

    Their entire discourse is about making it clear that they are trying to eliminate the effects of discrimination, [i]not[/i] trying to enforce equality of outcomes. "It's not about making it easier for Those People, it's about stopping making things harder for them."

    Trying to quantify the impact of discrimination is both a necessary step and an important PR signal in this regard.

    Which leads us to this:

    [i]So yeah, my call is for the NIH to balance the funding rates first, and then do all their fancy studies to root out the "real cause" later.[/i]

    Yeah. Well, no. Thank god.

  • Joe says:

    But Toto, "We'll do a study" doesn't sound like it will lead to any real change and is something that will take years. Surely something can also be done in the meantime.

  • Morgan Price says:

    "my call is for the NIH to balance the funding rates first" -- there's all these complicated discrimination statues and I'm not sure which (if any) apply. Do you know if this would be legal? Has anyone at NIH discussed it?

  • DrugMonkey says:

    Why is it legal to give affirmative action quota boosts to ESIs?

  • qaz says:

    For the same reason that it's legal to give boosts to legacies in college admissions. (Because it supports the current class structure.)

  • drugmonkey says:

    Yep.

Leave a Reply