Apr 09 2016

## Thought of the Day

"Uppity" is a fascinating concept when it comes to NIH Grant award.

We know the sentiment applies to newer and younger investigators. I've heard countless review and Program Officer comments which amount to "let's not get too big for your britches, young un!" in my day.

I wonder how much of the Ginther effect is related to sentiments similar to "damn uppity [insert subvocalization]"?

Mar 10 2016

Feb 02 2016

## CSR Head Nakamura Makes Bizarre Pronouncement

An email from the CSR of the NIH hit late yesterday a few days ago, pointing to a number of their Peer Review Notes including one on the budget bump that we are about to enjoy.
Actually that should be "some selected few of us will enjoy" because

“While $2 billion is a big increase, it is less than a 10 percent increase, and a large portion of it is set aside for specific areas and initiatives,” said Dr. Nakamura. “Competition for funding is still going to be intense, and paylines will not return to historic averages . . . Yeah, as suspected, that money is already accounted for. The part that has me fired up is the continuation after that ellipsis and a continuing header item. So make sure you put your best effort into your application before you apply.” Counterproductive Efforts “We know some research deans have quotas and force their PIs to submit applications regularly,” said Dr. Nakamura. “It’s important for them to know that university submission rates are not correlated with grant funding. Therefore, PIs should be encouraged to develop and submit applications as their research and ideas justify the effort to write them and have other scientists review them.” As usual I do not know if this is coming from ignorance or calculated strategy to make their numbers look better. I fear both possibilities. I'm going from memory here because I can't seem to rapidly find the related blog post or data analysis but I think I recall an illustration that University-total grant submission rates did not predict University-total success rates. At a very basic level Nakamura is using the lie of the truncated distribution. If you don't submit any grant applications, your success rate is going to be zero. I'm sure he's excluding those because seemingly that would make a nice correlation. But more importantly, he is trying to use university-wide measures to convince the individual PI what is best for her to do. Wrong. Wrong. Wrong. Not everyone's chances at that institution are the same. The more established investigators will probably, on average, enjoy a higher success rate. They can therefore submit fewer applications. Lesser folk enjoy lower success rates so therefore they have to keep pounding out the apps to get their grants. By extension, it takes very little imagination to understand that depending on your ratio of big important established scientists to noobs, and based somewhat on subfields, the apparent University-wide numbers are going to swamp out the information that is needed for each individual PI. In short, this is just another version of the advice to young faculty to "write better grants, just like the greybeards do". The trick is, the greybeards DO NOT WRITE BETTER GRANTS! I mean sure, yes, there is a small experience factor there. But the major driver is not the objective quality but rather the established track record of the big-deal scientist. This gives them little benefits of the doubt all over the place as we have discussed on this blog endlessly. I believe I have yet to hear from a new-comer to NIH grant review that has not had the experience within 1-2 rounds of a reviewer ending his/her review of a clearly lower-quality grant proposal with "....but it's Dr. BigShot and we know she does great work and can pull this off". Or similar. I have been on a study section round or two in my day and I am here to tell you. My experience is not at all consistent with the idea that the "best" grants win out. Merit scores are not a perfect function of objective grant quality at all. Imperfectly prepared or boring grants get funded all the time. Really exciting and nearly-perfect grants get unfundable scores or triaged. Frequently. This is because grant review hinges on the excitement of the assigned reviewers for the essence of the project. All else is detail. You cannot beat this system by writing a "perfect" grant. Because it may not be perfect for all three reviewers no matter how well it has been prepared and how well vetted by whatever colleagues you have rounded up to advise you. Nakamura should know this. He probably does. Which makes his "advice" a cynical ploy to decrease submissions so that his success rate will look better. One caveat: I could simply be out of touch with all of these alleged Dean-motivated crap apps. It is true that I have occasionally seen people throw up grant applications that really aren't very credible from my perspective. They are very rare. And it has occasionally been the case that at least one other reviewer liked something about an application I thought was embarrassingly crappy. So go figure. I also understand that there are indeed Deans or Chairs that encourage high submission rates and maybe this leads to PIs writing garbage now and again. But this does not account for the dismal success rates we are enjoying. I bet that magically disappearing all apps that a PI submitted to meet institutional vigor requirements (but didn't really mean to make a serious play for an award) would have no perceptible effect on success rates for the rest of us. I just haven't ever seen enough non-credible apps for this to make a difference. Perhaps you have another experience on study section, DearReaders? Finally, I really hate this blame-the-victim attitude on the part of the CSR and indeed many POs. There are readily apparent and demonstrable problems with how some categories of PIs' grants are reviewed. Newer and less experienced applicants. African-American PIs. Women. Perhaps, although this is less well-explicated lately, those from the wrong Universities. For the NIH to avoid fixing their own problems with review (for example the vicious cycle of study sections punishing ESI apps with ever-worsening scores when the NIH used special paylines to boost success rates) and then blame victims of these problems by suggesting they must be writing bad grants takes chutzpah. But it is wrong. And demoralizing to so many who are taking it on the chin in the grant review game. And it makes the problems worse. How so? Well, as you know, Dear Reader I am firmly convinced that the only way to succeed in the long term is to keep rolling the reviewer dice, hoping to get three individuals who really get what you are proposing. And to take advantage of the various little features of the system that respond to frequent submissions (reviewer sympathy, PO interest, extra end of year money, ARRA, sudden IC initiatives/directions, etc). Always, always you have to send in credible proposals. But perfect vs really good guarantees you nothing. And when perfect keeps you from submitting another really good grant? You are not helping your chances. So for Nakamura to tell people to sacrifice the really good for the perfect he is worsening their chances. Particularly when the people are in those groups who are already at a disadvantage and need to work even harder* to make up for it. __ *Remember, Ginther showed that African-American PIs had to submit more revisions to get funded. Nov 23 2015 ## Scenes In the past few weeks I have been present for the following conversation topics. 1) A tech professional working for the military complaining about some failure on the part of TSA to appropriately respect his SuperNotATerrorist pass that was supposed to let him board aircraft unmolested...unlike the the rest of us riff raff. I believe having his luggage searched in secondary was mentioned, and some other delays of minor note. This guy is maybe early thirties, very white, very distinct regional American accent, good looking, clean cut... your basic All-American dude. 2) A young guy, fresh out of the military looking to get on with one of the uniformed regional service squad types of jobs. This conversation involved his assertions that you had to be either a woman or an ethnic minority to have a shot at the limited number of jobs available in any given cycle. Much of the usual complaining about how this was unfair and it should be about "merit" and the like. Naturally this guy is white, clean cut, relatively well spoken.... perhaps not all that bright, I guess. 3) A pair of essentially the most privileged people I know- mid-adult, very smart, blonde, well educated, upper middle class, attractive, assertive, parents, rock of community type of women. Literally *everything* goes in these women's direction and has for most of their lives. They had the nerve to engage in a long running conversation about their respective minor traffic stops and tickets and how unfair it was. How the cops should have been stopping the "real" dangers to society at some other location instead of nailing them for running a stop sign a little too much or right on red-ing or whatever their minor ticket was for. One of the great things about modern social media is that, done right, it is a relatively non-confrontational way to start to see how other people view things. For me the days of reading science blogs and the women-in-academics blogs were a more personal version of some of the coursework I enjoyed in my liberal arts undergraduate education. It put me in touch with much of the thinking and experiences of women in my approximate career. It occasionally allowed me to view life events with a different lens than I had previously. It is my belief that social media has also been important for driving the falling dominoes of public opinion on gay marriage over the past decade or so. Facebook connections to friends, family and friends of the same provides a weekly? daily? reminder that each of us know a lot of gay folks that are important to us or at the very least are important to people that are important to us. The relentless circulation of memes and Bingo cards, of snark and hilarity alike, remind each of us that there is a viewpoint other than our own. And the decent people listen. Occasionally they start to see things the way other people do. At least now and again. The so-called Black Twitter is similar in the way it penetrated the Facebook and especially Twitter timelines and daily RTs of so many non-AfricanAmerican folks . I have watched this develop during Ferguson and through BlackLivesMatter and after shooting after shooting after shooting of young black people that has occurred in the past two years. During the three incidents that I mention, all I could think was "Wow, do you have any idea that this is the daily reality for many of your fellow citizens? And that it would hardly ever occur to non-white people to be so blindly outraged that the world should dare to treat them this way?" And "Wait, so are you saying it sucks to have a less-assured chance of gaining the career benefits you want due to the color of your skin or the nature of your dangly bits....it'll come to you in a minute". This brings me to today's topic in academic science. Nature News has an editorial on racial disparity in NIH grant awards. As a reminder the Ginther report was published in 2011. There are slightly new data out, generated from a FOIA request: Pulmonologist Esteban Burchard and epidemiologist Sam Oh of the University of California, San Francisco, shared the data with Nature after obtaining them from the NIH through a request under the Freedom of Information Act. The figures show that under-represented minorities have been awarded NIH grants at 78–90% the rate of white and mixed-race applicants every year from 1985 to 2013 I will note that Burchard and Oh seem to be very interested in how the failure to include a diverse population in scientific studies may limit health care equality. So this isn't just about career disparity for these scientists, it is about their discipline and the health outcomes that result. Nevertheless, the point of these data are that under-represented minority PIs have less funding success than do white PIs. The gap has been a consistent feature of the NIH landscape through thick and thin budgets. Most importantly, it has not budged one bit in the wake of the Ginther report in 2011. With that said, I'm not entirely sure what we have learned here. The power of Ginther was that it went into tremendous analytic detail trying to rebut or explain the gross disparity with all of the usual suspect rationales. Trying....and failing. The end result of Ginther was that it was very difficult to make the basic disparate finding go away by considering other mediating variables. After controlling for the applicant's educational background, country of origin, training, previous research awards, publication record, and employer characteristics, we find that black applicants remain 10 percentage points less likely than whites to be awarded NIH research funding. The Ginther report used NIH grant data between FY 2000 and FY 2006. This new data set appears to run from 1985 to 2013, but of course only gives the aggregate funding success rate (i.e. the per-investigator rate), without looking at sub-groups within the under-represented minority pool. This leaves a big old door open for comments like this one: Is it that the NIH requires people to state their race on their applications or could it be that the black applications were just not as good? Maybe if they just keep the applicant race off the paperwork they would be able to figure this out. and this one: I have served on many NIH study sections (peer review panels) and, with the exception of applicants with asian names, have never been aware of the race of the applicants whose grants I've reviewed. So, it is possible that I could have been biased for or against asian applicants, but not black applicants. Do other people have a different experience? This one received an immediate smackdown with which I concur entirely: That is strange. Usually a reviewer is at least somewhat familiar with applicants whose proposals he is reviewing, working in the same field and having attended the same conferences. Are you saying that you did not personally know any of the applicants? Black PIs are such a rarity that I find it hard to believe that a black scientist could remain anonymous among his or her peers for too long. Back to social media. One of the tweeps who is, I think, pretty out as an underrepresented minority of science had this to say: Not entirely sure it was in response to this Nature editorial but the sentiment fits. If AfricanAmerican PIs who are submitting grants to the NIH after the Ginther report was published in the late summer of 2011 (approximately 13 funding rounds ago, by my calendar) were expecting the kind of relief provided immediately to ESI PIs.....well, they are still looking in the mailbox. The editorial The big task now is to determine why racial funding disparities arise, and how to erase them. ...The NIH is working on some aspects of the issue — for instance, its National Research Mentoring Network aims to foster diversity through mentoring. and the News piece: in response to Kington’s 2011 paper, the NIH has allocated more than$500 million to programmes to evaluate how to attract, mentor and retain minority researchers. The agency is also studying biases that might affect peer review, and is interested in gathering data on whether a diverse workforce improves science.

remind us of the entirely toothless NIH response to Ginther.

It is part and parcel of the vignettes I related at the top. People of privilege simply cannot see the privileges they enjoy for what they are. Unless they are listening. Listening to the people who do not share the set of privileges under discussion.

I think social media helps with that. It helps me to see things through the eyes of people who are not like me and do not have my particular constellations of privileges. I hope even certain Twitter-refuseniks will come to see this one day.

Jun 09 2015

## Really NIDA National Advisory Council? REALLY????

I was reading over the minutes from the 119th meeting of the National Advisory Council for NIDA. As you may imagine, Dear Reader, I am more than usually interested in the doings of this particular IC.

I ran across the part where Hannah A. Valantine, M.D., MRCP, Chief Officer for Scientific Workforce Diversity, National Institutes of Health came to report on diversity issues, explicitly linked to the wake of the Ginther report.

She began her focus as the COSWD by reviewing evidence from the NIH Intramural Research Program as a Laboratory for Testing Interventions to Diversify the Biomedical Workforce evaluation. The results, as of October 2014 showed that among intramural tenure track investigators 38 % are female and 1.4% are African American, 10% are Hispanic, and 0.5% are Native American compared to approximately 61% being both males and white.

PhotoCredit: NIDA IRP

Wow, NIDA IRP, wow. Only 1.4%...okay that's just Jean Lud Cadet, correct? I mean, you only list 28 "PIs" so how many tenure track investigators could there be?

Not something that is on our radar that much out here in extramural researchistan but.... yeah. Get on that NIH ICs. I would be fascinated to see the representation numbers for the various NIH Intramural Research Programs at the various levels of lab heads and non-head so-called tenure-track investigators. I imagine this is not going to look good.

There's a lot more blah de blah in the NIDA Council notes from Valantine about the NIH efforts, particularly the "pipeline" solutions that so irritate me.

But that isn't the hilarious part. The hilarious part comes after Valentine described

...in detail the strategy and essential components, such as a strategic partnership with research intensive institutions, and tracking and evaluation. Program deliverables, would include: a national network to support career transitions; evidence-based literature to eliminate/reduce barriers to key career transition points; Individual access to the network in support of career development success; organizational infrastructure to support career development and transitions; and tools and resources to catalyze and sustain career transition success.

Here was the Council response.

NIDA Council members thanked Dr. Valantine for her dedication to this issue and for implementing so many diverse and potentially transformative programs. Council members encouraged, as applicable by the research, efforts to measure success with endpoints other than “tenure”.

Jesus.

Oh hey, let's not create too high of a barrier for ourselves. Let's not set it at sustained tenured research careers THAT WERE THE VERY POINT OF THE GINTHER REPORT! Differential success of PIs at getting grant funding. How is this not intimately tied to the tenure success of African-American academic scientists?

Feb 23 2015

## Patricia Arquette, Privilege and the Oppression Olympics

I was watching the Oscars last night when Patricia Arquette busted out some equal-pay feminism in her acceptance speech.

"To every woman who gave birth, to every taxpayer and citizen of this nation, we have fought for everybody else's equal rights," Arquette said, her voice intensifying. "It is our time to have wage equality once and for all and equal rights for women in the United States of America!"

HECK YEAH!!!!!

I was hooting and hollering too much, I assume, because I got shushed. Apparently some other people in the room wanted to hear what else was being said or whatever.

So it was with some confusion that I saw backlash later on the Twitts about her. It seemed to be of the intersectionality sort of criticism. Also known as the Oppression Olympics. Not to make light of it but look, we all come with various attributes that confer privileges upon us in this society we inhabit. Most of us have one or two attributes that confer the opposite. Some unlucky folks have a pretty tough menu of biases slanting against them. So yeah, there seemed to be a drumbeat of Twitterage against Patricia Arquette's immense privilege of wealth, whiteness and heteronomativity. I thought at first that this was undeserved, based on what she said from the stage...it's the Oscars for goodness sake, of course they are all white and perfect and immensely rich.

Then today I finally happened upon her expanded backstage comments. From this account:

"The truth is: even though we sort of feel like we have equal rights in America, right under the surface, there are huge issues that are applied that really do affect women," she mused. "And it's time for all the women in America and all the men that love women, and all the gay people, and all the people of color that we've all fought for to fight for us now."

Breathtakingly tone-deaf.

Look, I've spent a lot of time in my life feeling sorry for myself. I get it. It is really, really easy to focus narrowly on that one aspect, attribute, experience, factor or misfortune that leaves the self at apparent disadvantage. And it is correspondingly easy to forget all about all the other factors and attributes that have conveyed immense privileges upon our lives.

This is not solved by the data, of course. Firstly, because we can all pick and choose which truthy stat we want to brandish. Is it equal pay? Very easy to brandish the generally accepted, broad brush stats for men versus women. And very easy to ignore that women of color are even more screwed than woman not of color. Easy to have no idea whatsoever how well minority men are paid relative to women not of color. Or what being gay confers in terms of salary.

And it is incredibly seductive to argue the anecdote. Well, Oprah! And J.Lo. And Eddie Murphy! And FFS Neil Patrick Harris is the Master of Ceremonies for goodness sake! They are sitting right there, so therefore why would anyone think of how their respective skin tones and desired life-partner would have anything to do with equal pay for women, eh?

Academic science is no different my friends. If this highly public case makes the intersectionality issue clearer to you than it has ever been, do try to turn that inwards.

We run across these examples on the blog all the time, of course. Whether we're discussing the struggles of women in science, the Ginther report, outing yourself to search committees or thesis advisors, the Baby Boomer hegemony of NIH Grant funding, postdoctoral pay rates or the evils of PIs with too many grants, the issues are the same.

"Sure, sure, there are these other biases in careers. But what is REALLY important is that I, the speaker, haven't experienced* any of those advantages that adhere to my classes and characteristics. And let me tell you about my specific set of life events that prove that really, I personally have been at huge disadvantage. So it is totally misplaced to talk about the general advantages of my characteristic X because the anecdote of me proves that X is much less important than this totally other thing that I happen to suffer from."

At this point one or the other of you, DearReader, may suspect I am talking about you in particular. Naturally, I am not. This is a common theme. Very common.

It is something that I have suffered from in my life and continue to do so. I have felt immensely sorry for myself a lot over the years.

Like many of you, I can claim one or two disadvantages within a context of immense privileges when it comes to pursuing the career of academic science. Like many of you, I CANNOT HELP BUT IGNORE MY PRIVILEGES AND PITY MYSELF ABOUT MY HARDSHIPS. Like many of you, I feel compelled to speak out about perceived injustices in the world. Like many of you, some of those injustices I speak about happen to be ones that I think affect me. Like many of you, some of those injustices I speak about do not happen to affect me in any direct way.

And, like many of you and Patricia Arquette, I often speak about injustices in a way that appears to ignore the fact that other people have it a lot worse.

Social media has a way of helping us to remember that other people have it even worse. And that trying to recruit others to help you in your fights, without ever appearing** to be that concerned about their fights comes across as selfish and tone deaf.

__
*of course you have, you just think that this is totally normal and average and deserved, and thus not worthy of inclusion in any discussion.

**For all I know Patricia Arquette is a huge fighter for underrepresented groups, including ethnic minorities and LGBT folks. But her comments certainly didn't convey that.

The tldr; version of this post:

Feb 11 2015

## A tiny bias goes a long way when it comes to grant review

From ScienceInsider:

Now, a new computer simulation explores just how sensitive the process might be to bias and randomness. Its answer: very. Small biases can have big consequences, concludes Eugene Day, a health care systems engineer at the Children's Hospital of Philadelphia, in Research Policy. He found that bias that skews scores by just 3% can result in noticeable disparities in funding rates.

T. E. Day, The big consequences of small biases: A simulation of peer review, 2015, Research Policy [epub ahead of print 28 Jan] [Publisher Site]

from the paper Abstract:

When total review bias exceeds 1.9% of grant score, statistically significant variation in scores between PC and NPC investigators is discernable in a pool of 2000 grant applications. When total review bias exceeds 2.8% of total grant score, statistically significant discrepancies in funding rates between PC and NPC investigators are detectable in a simulation of grant review.

Day generated a Preferred Class of applications and a NonPreferred Class of applications and ran a bunch of 3-reviewer scenarios with and without reviewer bias against the NPC applications. As far as I can tell the takeaway conclusion about funding here refers to a situation in which the effective payline is 10%. You will immediately grasp that NIH grant review was a strong contributor to the model parameters.

I will admit I am only able to grasp the main points here and I am in no way able to evaluate the nitty gritty.

But it appears to have a very strong message. Namely, that our introspections that "well, if there is bias it is very tiny so we don't have to be worried about it" needs to change.

There is something even scarier in this paper. From the Discussion:

The threshold level of bias in this environment seems to be 2.8% of the total possible score of the grant; this is the level at which the 95% CI of the odds ratio “kisses” 1.00. This represents a single reviewer with a bias of 0.75 points (or three reviewers each with biases of 0.25 points), which is less than half (44.4%) of the standard deviation in a single reviewer’s score. What this suggests is that levels of bias which are sub-noise – that is, that are dramatically less detectable than normal variation in reviewer scores – are sufficient to substantially bias the number of funded applications in favor of preferred investigators.

RIGHT???? The bias can be of smaller effect size than many "normal" sources of variability in scoring that we accept as the resolution of the system. And it still leads to a statistically significant bias in funding outcome.

We are talking in recent days about bias in favor of highly established, older scientists. It has been longer but the Ginther report indicating disparity of grant review outcome for African-American PIs is clearly relevant here.

What this simulation cannot do, of course, is to model the cumulative, iterative effects of review bias. Namely, the way that selection of PC applications for funding has a tendency to increase the bias in the reviewer pool, since those beneficiaries become the next reviewers. Also, the way that over the long haul, disparity of the first award can lead to actual quality differences in the subsequent applications because PI #1 had the money to pursue her science and PI #2 did not have as easy of a time generating data, publishing papers and recruiting postdoc talent.

Feb 09 2015

## Perspectives from senior scientists on the Emeritus award discussion

Tthe comments just keep coming over at RockTalking.

In 2012-13 my NIH renewal proposal with 4 specific aims was turned down 2X by the GM, NCSD Panel, with 35%+ priority scores. ...I appealed the grant reviews to the GM Council and they awarded the grant to me for 3 years at somewhat reduced funding. This funding will finance my lab until Sept 2016, after which I will close down. I am NOT closing down because I have lost my energy for, or interest in, research: [blah, blah we published and showed them!] I am closing down my lab because I can no longer put up with the aggravation of having my grants turned down

Cry me a river.

another senior investigator is on fire:

So far, all the comments fully support age discrimination. How sad! -

Age limits are silly and discriminatory. Merit worked well UNTIL there was no more money in the NIH bank.

There should be COMPETITIVE opportunities for scientists at all stages of their careers (notice I said competitive) . So it’s not a handout at all and it would be merit-based….just like those for newly trained scientists.

There are wonderful examples of some amazing senior scientists. ONE special initiative is not unreasonable for this group!

Finally, I don’t believe I said that I “deserve” anything… except not to be discriminated against for age, gender or whatever characteristic you wish to select.

updated: omg, the old guy again!

At the last American Society for Cell Biology meeting in Philadelphia (December, 2014) I stood up at the membership meeting ( only ca. 30 people) attended by some of the senior wheels in cell biology. I asked that the society appoint a new standing committee that did
nothing but try to re-design the NIH extramural grant system in its entirety, eg. the grant applications, who can apply, how review panels are chosen, requirements for NIH supported investigators to serve on panels, and how key personnel of the NIH bureaucracy are chosen, and what type of grants should be awarded. My suggestion was not greeted with great enthusiasm and, in fact, elicited some negative comments.

Bashir points out something that is also important about this proposal.

After all that determination to do something after Ginther Report, with mentoring groups and other round-a-bout approaches aimed at eventually addressing racial disparities in grant awards, suddenly we may have a very direct new mechanism that, regardless of the underlying logic, is essentially un-diversity.

Aug 20 2014

## You came so close, Nature Editorial staff.

A news bit in Nature overviews Richard Nakamura's plans to investigate the disparity in grant funding that was identified in the Ginther report in 2011. See here, here, here for blog comment on Gither et al. Nakamura is, of course, the current director (pdf) of the Center for Scientific Review, the entity at NIH that conducts the peer review of most grant applications that are submitted.

It is promising-ish. Nakamura's plans are summarized.

One basic issue that the NIH will address is whether grant reviewers are thinking about an applicant’s race at all, even unconsciously. A team will strip names, racial identification and other identifying information from some proposals before reviewers see them, and look at what happens to grant scores.

Hope they check on the degree to which the blinding works, of course. As you know, Dear Reader, I am always concerned that blinding of academic works cannot always be assumed to have functionally worked to prevent the reviewer from identifying the author or lab group.

The NIH will also study reviewers’ work in finer detail, by analysing successful applications for R01 grants, the NIH’s largest funding programme for individual investigators. The goal is to see whether researchers can spot trends in the language used by reviewers to describe proposals put forward by applicants of different races. There is precedent for detectable differences: in a paper to be published in Academic Medicine, a team led by Molly Carnes, a physician at the University of Wisconsin-Madison, used automated text analysis to show that reviewers’ critiques of R01 grant applications by women tended to include more words denoting praise, as though the writer is surprised at the quality of the work.

Very intriguing contribution to the analysis. Nice.

The NIH will also analyse text in samples of reviewers’ unedited critiques. The Center for Scientific Review typically edits the wording and grammar of these reviews before grant proposals are returned to applicants, but even the subtlest details of such raw comments might hold clues about bias. Nakamura says that reviewers will not be told whether their comments will be analysed, because that in itself would bias the sample. “We want them to be sloppy,” he says.

Hmmm. I guess this is just human factors checking on the automated analysis. Together they are stronger.

The NIH’s Study Sections, in which review groups discuss the top 50% of grant applications, might also harbour bias: the 2011 Science paper found that submissions authored by African Americans are less likely to be discussed in the meetings. But when they are, a negative comment arising from even one person’s unconscious bias could have a major impact in such a group setting, says John Dovidio, a psychologist at Yale University in New Haven, Connecticut, and a member of the NIH’s Diversity Working Group. “That one person can poison the environment,” he says.

This is not presented in a context that suggests the NIH plans to investigate this directly. Not sure how this could be done without putting a severe finger on the balance. I mean sure, most meetings have call-in lines open to Program staff and Nakamura could just record and transcribe meetings but.....reviewers won't like it and if you warn them you scare off the fishies. So to speak.

Nakamura expects that the NIH’s effort to identify and root out prejudice, which he says could cost up to \$5 million over three years, might prove controversial. “People resent the implication they might be biased,” he says — an idea borne out by some responses to his 29 May blogpost on the initiative. One commenter wrote, “It is absolutely insulting to be accused of review bigotry. Please tell me why I should continue to give up my time to perform peer review?”

But Nakamura believes that the NIH — and reviewers — need to keep open minds.

He, and media covering this, need to focus on the opportunity to communicate that institutional racism does not hinge on whether individual actors are overtly biased. The piece leads with the comment that Nakamura got his butt over to the Implicit Association website and identified his own biases to himself. This is the sort of introspection that needs to happen. Heck, I'd love to see a trial where study section reviewers were told to go over there and complete a few tests prior to receiving their grant assignments*.

The Nature editorial is, for the most part, on the side of goodness and light on this.

The idea that scientists who volunteer time and energy to review NIH grants could be biased against qualified minority researchers is a tough pill to swallow. The NIH is to be commended for not sweeping this possibility under the rug: it has turned to the scientific method to investigate the suggestion.

good, good.....

It is a topic that the NIH will need to broach delicately. Few academics consciously hold any such inclinations, and fewer still would deliberately allow them to affect their grant evaluations. Some are likely to bristle at what might be seen as an accusation of racism, and the NIH plans to conduct at least some of its studies of grant reviews without the reviewers’ knowledge or consent.

But better for the NIH to offend a few people than to make snap judgements and institute blunt policies to address the problem. Fixes such as increasing scholarships and training for minority groups would no doubt be a good thing, but they could be an unhelpful use of money if they do not address the root cause of the disparity.

yes, yes, excellent.....

And policies such as grant-allocation quotas could come at the expense of other researchers.

BZZZTTTTTTT!!!!

Right back to victim blaming. Right back to ignoring what it means to have a BIAS identified. Right back to ignoring what the nature of privilege means.

Those "other researchers" at present enjoy a disparate benefit at the expense of AfricanAmerican PIs. That's what Ginther means. Period. The onus shifted, upon identification of the disparity, to proving that non-AfricanAmerican PIs actually deserve their awards.

Ginther, btw, went a long ways toward rejecting some of the more obvious reasons why the disparity was not in fact an unfair bias. Read it, including the supplementary materials before you start commenting with stupid. Also, review this.

But there is also this. The low numbers of AfricanAmerican scientists submitting applications to the NIH for funding means that any possible hit to the success rate of non-AfricanAmerican PIs would be well nigh undetectable. A miniscule effect size relative to all other sources of variance in the funding process.

Another way to look at this issue is to take Berg's triage numbers from above. To move to 40% triage rate for the African-AmericanPI applications, we need to shift 20% (230 applications) into the discussed pile. This represents a whopping 0.4% of the White PI apps being shifted onto the triage pile to keep the numbers discussed the same.

These are entirely trivial numbers in terms of the "hit" to the chances of White PIs and yet you could easily equalize the success rate or award probability for African-American PIs.

It is even more astounding that this could be done by picking up African-American PI applications that scored better than the White PI applications that would go unfunded to make up the difference.

And of course "grant-allocation quotas" are precisely what the special paylines and other assists for ESI investigators consist of. Affirmative Action for the young and untried.

Did we get this sort of handwringing, call for long-duration "study" of the "true causes" of the disparity?

Hell no.

The NIH just started picking up ESI grants to balance the odds of funding, even when study sections responded to news of this affirmative action by further punishing ESI scores!

So yeah, my call is for the NIH to balance the funding rates first, and then do all their fancy studies to root out the "real cause" later.

Also for editorial teams like the one at Nature to stop repeating this whinging about those who already enjoy disparate privilege who might lose (some of) it.
__
*yeah, it might backfire. that would itself be interesting.

Mar 13 2014

## Guest Post: BUILDing Diversity with Dollars: Can Grants Change Culture

This guest post is from @iGrrrl, a grant writing consultant. I think I first ran across her in the comments over at writedit's place, you may have as well. She brings a slightly different, and highly valuable, perspective to the table.

There was an old New Yorker cartoon of two people at a party, and one tells the other, "I'm a fiction writer in the grant-proposal genre." I hate putting fiction in grant applications, especially the type that will be due shortly in response to the Ginther report.

For those who have been worrying about their own grant applications, the Ginther report detailed the relationship of race and ethnicity to NIH grant funding at the R01 level, and NIH has created a few initiatives to try to change the pattern of lower success rates for African American applicants. In early April, the applications are due for the Building Infrastructure Leading to Diversity (BUILD) Initiative, which would fund large-scale projects within individual institutions, and the NIH National Research Mentoring Network (NRMN), which is designed to build a network of mentors. In other words, diversity interventions writ large, with millions of dollars behind them.

The NSF ADVANCE program started in a similar way, and some of the early ADVANCE projects included programs with a limited evidence base. The successes of components of these programs were variable, and the recent RFA included a social science research component to complement the required evaluation of ADVANCE-funded activity. It started with good intentions, and eventually became clear that more than good intentions were needed. Part of the NRMN announcement calls for pilot programs, but I would argue that in the first year, the network should do the social science work to consolidate the anecdotes African Americans tell about their mentoring experiences into hard data, so that the pilot programs can be based upon addressing the needs identified by African Americans who have been through, or training right now in, the current system.

At the recent AAAS meeting in Chicago there were sessions on building diversity in science. At one I learned that explicit bias has reduced in the last 30 years, but implicit bias hasn't. We think we have made progress, and that our conscious intentions are enough. But they clearly are not. Expecting trainees to overcome biased behaviors (to which the actors are blind) places an undue burden on those who are discriminated against. There are studies showing that education about implicit bias helps to reduce such biased behavior, but education attempts can also be done badly and backfire. As pointed out in a recent piece in Science by Moss-Racusin, et al., there is an evidence base now for doing intervention well. If NIH is putting money into large-scale intervention, I hope the existing science will be part of the applications, and expected by the reviewers.

I've spent a lot of my professional life working on exactly the kind of large, infrastructure-based grant application represented by the BUILD and NRMN programs. It is easy for PIs to make assumptions that interventions that sound good on paper will actually have any impact. My concern is that what will be proposed by the applicants to BUILD and NRMN may miss the strong social science work that exists, and that still needs to be done. In fact, some of the best research on effective mentoring is the business literature, a place where few biomedical scientists would think to look.

Grant applications shouldn't be pure fiction, but based on solid evidence. Every grant application represents a possibility, a reality, that could come to pass if the funds are awarded. In the mentoring literature, practices that improve the success for African Americans are often shown to improve the climate for everyone. There is an opportunity here for those in biomedicine to learn from other fields, to consider an evidence base that is outside their usual ken, and to improve the entire biomedical enterprise by improving the overall environment. I hope that those applying for BUILD and the NRMN include the social sciences, and even more importantly, include the voices and ideas of the very people these programs are meant to serve.

NIH has a long history of using dollars to encourage cultural change, with mixed results, because applicants can have varying levels of commitment to the NIH vision while being happy to take NIH dollars. The ADVANCE program at NSF had some hiccups as they worked out what worked to improve the climate for women in STEM. The leadership teams for BUILD and NRMN should include people with a deep knowledge of the research and scholarship on bias and on mentoring, and who can do the rigorous analysis of the current state of affairs for African Americans in biomedical science. I hope I'm wrong here in worrying that such people won't be included, but I've seen fiction in grant applications a few too many times.

• Scientopia Blogs

• DrugMonkey is an NIH-funded researcher who blogs about careerism in science. And occasionally about the science of drug use.

• Your donation helps to support the operation of Scientopia - thanks for your consideration.