Search Results for "ginther"

Aug 15 2018

GAO report shows the continued NIH grant funding disparity for underrepresented PIs

A comment from pielcanelaphd on a prior post tips us off to a new report (PDF) from the General Accountability Office, described as a report to Congressional Committees.

The part of the report that deals with racial and ethnic disparities is mostly recitation of the supposed steps NIH has been taking in the wake of the Ginther report in 2011. But what is most important is the inclusion of Figure 2, an updated depiction of the funding rate disparity.
GAO-18-545:NIH RESEARCH Action Needed to Ensure Workforce Diversity Strategic Goals Are Achieved

These data are described mostly as the applicant funding rate or similar. The Ginther data focused on the success rate of applications from PIs of various groups. So if these data are by applicant PI and not by applications, there will be some small differences. Nevertheless, the point remains that things have not improved and PIs from underrepresented ethnic and racial groups experience a disparity relative to white PIs.

No responses yet

Jul 20 2018

When NIH uses affirmative action to fix a bias

We have just learned that in addition to the bias against black PIs when they try to get research funding (Ginther et al., 2011), Asian-American and African-American K99 applicants are also at a disadvantage. These issues trigger my usual remarks about how NIH has handled observed disparities in the past. In the spirit of pictures being worth more than words we can look up the latest update on success rates for RPG (a laundry list of research grant support mechanisms) broken down by two key factors.
First up is the success rate by the gender of the PI. As you can see very clearly, something changed in 2003. All of a sudden a sustained advantage for men disappeared. Actually two things happened. This disparity was "fixed" and the year after success rates went in the tank for everyone. There are a couple of important observations. The NIH didn't suddenly fix whatever was going on in study section, I guaranfrickentee it. I guarantee there were not also any magic changes in the pipeline or female PI pool or anything else. I guarantee you that the NIH decided to equalize success rates by heavy handed top-down affirmative action policies in the nature of "make it so" and "fix this". I do not recall ever seeing anything formal so, hey, I could be way off base. If so, I look forward to any citation of information showing change in the way they do business that coincided directly with the grants submitted for the FY2003 rounds.
The second thing to notice here is that women's success rates never exceeded that for men. Not for fifteen straight Fiscal Years. This further supports my hypothesis that the bias hasn't been fixed in some fundamental way. If it had been fixed, this would be random from year to year, correct? Sometimes the women's rates would sneak above the men's rates. That never happens. Because of course when we redress a bias, it can only ever just barely reach statistically indistinguishable parity and if god forbid the previously privileged class suffers even the tiniest little bit of disadvantage it is an outrage.
Finally, the fact that success rates went in the tanker in 2004 should remind you that men enjoyed the advantage all during the great NIH doubling! The salad days. Lots of money available and STILL it was being disproportionately sucked up by the advantaged group. You might think that when there is an interval of largesse that systems would be more generous. Good time to slip a little extra to women, underrepresented individuals or the youth, right? Ha.

Which brings me to the fate of first-time investigators versus established investigators. Oh look, the never-funded were instantly brought up to parity in 2007. In this case a few years after the post-doubling success rates went in the toilet but more or less the same pattern. Including the failure of the statistically indistiguishable success rates for the first timers to ever, in 11 straight years of funding, to exceed the rates for established investigators. Because of affirmative action instead of fixing the bias. As you will recall, the head of the NIH at that time made it very clear that he was using "make it so" top-down heavy handed quota based affirmative action to accomplish this goal.

Zerhouni created special awards for young scientists but concluded that wasn't enough. In 2007, he set a target of funding 1500 new-investigator R01s, based on the previous 5 years' average.

Some program directors grumbled at first, NIH officials say, but came on board when NIH noticed a change in behavior by peer reviewers. Told about the quotas, study sections began “punishing the young investigators with bad scores,” says Zerhouni.

"quotas".

I do not recall much in the way of discussing the "pipelines" and how we couldn't possible do anything to change the bias of study sections until a new, larger and/or better class of female or not-previously-funded investigators could be trained up. The NIH just fixed it. ish. permanently.

For FY2017 there were 16,954 applications with women PIs. 3,186 awards. If you take the ~3% gap from the interval prior to 2003, this means that the NIH is picking up some 508 research project grants from women PIs via their affirmative action process. Per year. If you apply the ~6% deficit enjoyed by first time investigators in the salad days you end up with 586 research project grants picked up by affirmative action. Now there will be some overlap of these populations. Women are PI of about 31% of applications in the data for the first graph and first timers are about 35% for the second. So very roughly women might be 181 of the affirmative action newbie apps and newbies might be 178 of the affirmative action women's apps. The estimates are close. So let's say something like 913 unique grants are picked up by the NIH just for these two overt affirmative action purposes. Each and every Fiscal Year.

Because of the fact that, for example, African-American PIs of research grants or K99 apps represent such tiny percentages of the total (2% in both cases), the number of pickups that would be necessary to equalize success rate disparities is tiny. In the K99 analysis, it was a mere 23 applications across a decade. Two per year. I don't have research grant numbers handy but if we use the data underlying the first graph, this means there were about 1,080 applications with African-American PIs in FY2017. If they hit the 19% success rate this would be about 205 applications. Ginther reported about a 13% success rate deficit, working out to 55% of the success rate enjoyed by white applicants at the time. This would correspond to a 10.5% success rate for black applicants now, or about 113 application. So 92 would be needed to make up the difference for African-American PIs assuming the Ginther disparity still holds. This would be less than one percent of the awards made.

Less than one percent. And keep in mind these are not gifts. These are making up for a screwjob. These are making up for the bias. If any applicants from male, established or white populations go unfunded to redress the bias, they are only losing their unearned advantage. Not being disadvantaged.

28 responses so far

Jul 19 2018

Racial Disparity in K99 Awards and R00 Transitions

Oh, what a shocker.

In the wake of the 2011 Ginther finding [see archives on Ginther if you have been living under a rock] that there was a significant racial bias in NIH grant review, the concrete response of the NIH was to blame the pipeline. Their only real dollar, funded initiatives were to attempt to get more African-American trainees into the science pipeline. The obvious subtext here was that the current PIs, against whom the grant review bias was defined, must be the problem, not the victim. Right? If you spend all your time insisting that since there were not red-fanged, white-hooded peer reviewers overtly proclaiming their hate for black people that peer review can't be the problem, and you put your tepid money initiatives into scraping up more trainees of color, you are saying the current black PIs deserve their fate. Current example: NIGMS trying to transition more underrepresented individuals into faculty ranks, rather than funding the ones that already exist.

Well, we have some news. The Rescuing Biomedical Research blog has a new post up on Examining the distribution of K99/R00 awards by race authored by Chris Pickett.

It reviews success rates of K99 applicants from 2007-2017. Application PI demographics broke down to nearly 2/3 White, ~1/3 Asian, 2% multiracial and 2% black. Success rates: White, 31%, Multiracial, 30.7%, Asian, 26.7%, Black, 16.2%. Conversion to R00 phase rates: White, 80%, Multiracial, 77%, Asian, 76%, Black, 60%.

In terms of Hispanic ethnicity, 26.9% success for K99 and 77% conversion rate, neither significantly different from the nonHispanic rates.

Of course, seeing as how the RBR people are the VerySeriousPeople considering the future of biomedical careers (sorry Jeremy Berg but you hang with these people), the Discussion is the usual throwing up of hands and excuse making.

"The source of this bias is not clear...". " an analysis ...could address". "There are several potential explanations for these data".

and of course
"put the onus on universities"

No. Heeeeeeyyyyyuuullll no. The onus is on the NIH. They are the ones with the problem.

And, as per usual, the fix is extraordinarily simple. As I repeatedly observe in the context of the Ginther finding, the NIH responded to a perception of a disparity in the funding of new investigators with immediate heavy handed top-down quota based affirmative action for many applications from ESI investigators. And now we have Round2 where they are inventing up new quota based affirmative action policies for the second round of funding for these self-same applicants. Note well: the statistical beneficiaries of ESI affirmative action polices are white investigators.

The number of K99 applications from black candidates was 154 over 10 years. 25 of these were funded. To bring this up to the success rate enjoyed by white applicants, the NIH need only have funded 23 more K99s. Across 28 Institutes and Centers. Across 10 years, aka 30 funding cycles. One more per IC per decade to fix the disparity. Fixing the Asian bias would be a little steeper, they'd need to fund another 97, let's round that to 10 per year. Across all 28 ICs.

Now that they know about this, just as with Ginther, the fix is duck soup. The Director pulls each IC Director aside in quiet moment and says 'fix this'. That's it. That's all that would be required. And the Directors just commit to pick up one more Asian application every year or so and one more black application every, checks notes, decade and this is fixed.

This is what makes the NIH response to all of this so damn disturbing. It's rounding error. They pick up grants all the time for reasons way more biased and disturbing than this. Saving a BSD lab that allegedly ran out of funding. Handing out under the table Administrative Supplements for gawd knows what random purpose. Prioritizing the F32 applications from some labs over others. Ditto the K99 apps.

They just need to apply their usual set of glad handing biases to redress this systematic problem with the review and funding of people of color.

And they steadfastly refuse to do so.

For this one specific area of declared Programmatic interest.

When they pick up many, many more grants out of order of review for all their other varied Programmatic interests.

You* have to wonder why.
__
h/t @biochembelle

*and those people you are trying to lure into the pipeline, NIH? They are also wondering why they should join a rigged game like this one.

13 responses so far

May 07 2018

Addressing the Insomnia of Francis Collins and Mike Lauer

The Director of the NIH and the Deputy Director in charge of the office of extramural research have posted a blog post about The Issue that Keeps Us Awake at Night. It is the plight of the young investigator, going from what they have written.


The Working Group is also wrestling with the issue that keeps us awake at night – considering how to make well-informed strategic investment decisions to nurture and further diversify the biomedical research workforce in an environment filled with high-stakes opportunity costs. If we are going to support more promising early career investigators, and if we are going to nurture meritorious, productive mid-career investigators by stabilizing their funding streams, monies will have to come from somewhere. That will likely mean some belt-tightening in other quarters, which is rarely welcomed by the those whose belts are being taken in by a notch or two.

They plan to address this by relying on data and reports that are currently being generated. I suspect this will not be enough to address their goal.

I recently posted a link to the NIH summary of their history of trying to address the smooth transition of newly minted PIs into NIH-grant funded laboratories, without much comment. Most of my Readers are probably aware by now that handwringing from the NIH about the fate of new investigators has been an occasional feature since at least the Johnson Administration. The historical website details the most well known attempts to fix the problem. From the R23 to the R29 FIRST to the New Investigator check box, to the "sudden realization"* they needed to invent a true Noob New Investigator (ESI) category, to the latest designation of the aforementioned ESIs as Early Established Investigators for continued breaks and affirmative action. It should be obvious from the ongoing reinvention of the wheel that the NIH periodically recognizes that the most recent fix isn't working (and may have unintended detrimental consequences).

One of the reasons these attempts never truly work and have to be adjusted or scrapped and replaced by the next fun new attempt was identified by Zerhouni (a prior NIH Director) in about 2007. This was right after the "sudden realization" and the invention of the ESI. Zerhouni was quoted in a Science news bit as saying that study sections were responding to the ESI special payline boost by handing out ever worsening scores to the ESI applications.

Told about the quotas, study sections began “punishing the young investigators with bad scores,” says Zerhouni.

Now, I would argue that viewing this trend of worsening scores as "punishing" is at best only partially correct. We can broaden this to incorporate a simple appreciation that study sections adapt their biases, preferences and evolved cultural ideas about grant review to the extant rules. One way to view worsening ESI scores may have to do with the pronounced tendency reviewers have to think in terms of fund it / don't fund it, despite the fact that SROs regularly exhort them not to do this. When I was on study section regularly, the scores tended to pile up around the perceived payline. I've seen the data for one section across multiple rounds. Reviewers were pretty sensitive to the scuttlebutt about what sort of score was going to be a fundable one. So it would be no surprise whatsoever to me if there was a bias driven by this tendency, once it was announced that ESI applications would get a special (higher) payline for funding.

This tendency might also be driven in part by a "Get in line, youngun, don't get too big for your britches" phenomenon. I've written about this tendency a time or two. I came up as a postdoc towards the end of the R29 / FIRST award era and got a very explicit understanding that some established PIs thought that newbies had to get the R29 award as their first award. Presumably there was a worsening bias against giving out an R01 to a newly minted assistant professor as their first award**, because hey, the R29 was literally the FIRST award, amirite?

sigh.

Then we come to hazing, which is the even nastier relative of the "Don't get to big for your britches". Oh, nobody will admit that it is hazing, but there is definitely a subcurrent of this in the review behavior of some people that think that noob PIs have to prove their worth by battling the system. If they sustain the effort to keep coming back with improved versions, then hey, join the club kiddo! (Here's an ice pack for the bruising). If the PI can't sustain the effort to submit a bunch of revisions and new attempts, hey, she doesn't really have what it takes, right? Ugh.

Scientific gate-keeping. This tends to cover a multitude of sins of various severity but there are definitely reviewers that want newcomers to their field to prove that they belong. Is this person really an alcohol researcher? Or is she just going to take our*** money and run away to do whatever basic science amazeballs sounded super innovative to the panel?

Career gate-keeping. We've gone many rounds on this one within the science blog- and twittospheres. Who "deserves" a grant? Well, reviewers have opinions and biases and despite their best intentions and wounded protestations...these attitudes affect review. In no particular order we can run down the favorite targets of the "Do it to Julia, not me, JULIA!" sentiment. Soft money job categories. High overhead Universities. Well funded labs. Translational research taking all the money away from good honest basic researchers***. Elite coastal Universities. Big Universities. R1s. The post-normative-retirement crowd. Riff-raff plodders.

Layered over the top of this is favoritism. It interacts with all of the above, of course. If some category of PI is to be discriminated against, there is very likely someone getting the benefit. The category of which people approve. Our club. Our kind. People who we like who must be allowed to keep their funding first, before we let some newbie get any sniff of a grant.

This, btw, is a place where the focus must land squarely on Program Officers as well. The POs have all the same biases mentioned above, of course. And their versions of the biases have meaningful impact. But when it comes to thought of "we must save our long term investigators" they have a very special role to play in this debacle. If they are not on board with the ESI worries that keep Collins and Lauer awake at night, well, they are ideally situated to sabotage the effort. Consciously or not.

So, Director Collins and Deputy Director Lauer, you have to fix study section and you have to fix Program if you expect to have any sort of lasting change.

I have only a few suggestions and none of this is a silver bullet.

I remain convinced that the only tried and true method to minimize the effects of biases (covert and overt) is the competition of opposing biases. I've remarked frequently that study sections would be improved and fairer if less-experienced investigators had more power. I think the purge of Assistant Professors effected by the last head of the CSR (Scarpa) was a mistake. I note that CSR is charged with balancing study sections on geography, sex, ethnicity, university type and even scientific subdomains...while explicitly discriminating against younger investigators. Is it any wonder if there is a problem getting the newcomers funded?

I suggest you also pay attention to fairness. I know you won't, because administrators invariably respond to a situation of perceived past injustice with "ok, that was the past and we can't do anything about it, moving forward please!". But this is going to limit your ability to shift the needle. People may not agree on what represents fair treatment but they sure as heck are motivated by fairness. Their perception of whether a new initiative is fair or unfair will tend to shape their behavior when reviewing. This can get in the way of NIH's new agenda if reviewers perceive themselves as being mistreated by it.

Many of the above mentioned reviewer quirks are hardened by acculturation. PIs who are asked to serve on study section have been through the study section wringer as newbies. They are susceptible to the idea that it is fair if the next generation has it just about as hard as they did and that it is unfair if newbies these days are given a cake walk. Particularly, if said established investigators feel like they are still struggling. Ahem. It may not seem logical but it is simple psychology. I anticipate that the "Early Established Investigator" category is going to suffer the same fate as the ESI category. Scores will worsen, compared to pre-EEI days. Some of this will be the previously mentioned tracking of scores to the perceived payline. But some of this will be people**** who missed the ESI assistance who feel that it is unfair that the generation behind them gets yet another handout to go along with the K99/R00 and ESI plums. The intent to stabilize the careers of established investigators is a good one. But limiting this to "early" established investigators, i.e., those who already enjoyed the ESI era, is a serious mistake.

I think Lauer is either aware, or verging on awareness, of something that I've mentioned repeatedly on this blog. I.e. that a lot of the pressure on the grant system- increasing numbers of applications, PIs seemingly applying greedily for grants when already well funded, they revision queuing traffic pattern hold - comes from a vicious cycle of the attempt to maintain stable funding. When, as a VeryEstablished colleague put it to me suprisingly recently "I just put in a grant when I need another one and it gets funded" is the expected value, PIs can be efficient with their grant behavior. If they need to put in eight proposals to have a decent chance of one landing, they do that. And if they need to start submitting apps 2 years before they "need" one, the randomness is going to mean they seem overfunded now and again. This applies to everyone all across the NIH system. Thinking that it is only those on their second round of funding that have this stability problem is a huge mistake for Lauer and Collins to be making. And if you stabilize some at the expense of others, this will not be viewed as fair. It will not be viewed as shared pain.

If you can't get more people on board with a mission of shared sacrifice, or unshared sacrifice for that matter, then I believe NIH will continue to wring its hands about the fate of new investigators for another forty years. There are too many applicants for too few funds. It amps up the desperation and amps up the biases for and against. It decreases the resistance of peer reviewers to do anything to Julia that they expect might give a tiny boost to the applications of them and theirs. You cannot say "do better" and expect reviewers to change, when the power of the grant game contingencies is so overwhelming for most of us. You cannot expect program officers who still to this day appear entirely clueless about they way things really work in extramural grant-funded careers to suddenly do better because you are losing sleep. You need to delve into these psychologies and biases and cultures and actually address them.

I'll leave you with an exhortation to walk the earth, like Caine. I've had the opportunity to watch some administrative frustration, inability and nervousness verging on panic in the past couple of years that has brought me to a realization. Management needs to talk to the humblest of their workforce instead of the upper crust. In the case of the NIH, you need to stop convening preening symposia from the usual suspects, taking the calls of your GlamHound buddies and responding only to reps of learn-ed societies. Walk the earth. Talk to real applicants. Get CSR to identify some of your most frustrated applicants and see what is making them fail. Find out which of the apparently well-funded applicants have to work their tails off to maintain funding. Compare and contrast to prior eras. Ask everyone what it would take to Fix the NIH.

Of course this will make things harder for you in the short term. Everyone perceives the RealProblem as that guy, over there. And the solutions that will FixTheNIH are whatever makes their own situation easier.

But I think you need to hear this. You need to hear the desperation and the desire most of us have simply to do our jobs. You need to hear just how deeply broken the NIH award system is for everyone, not just the ESI and EEI category.

PS. How's it going solving the problem identified by Ginther? We haven't seen any data lately but at last check everything was as bad as ever so...

PPS. Are you just not approving comments on your blog? Or is this a third rail issue nobody wants to comment on?
__
*I make fun of the "sudden realization" because it took me about 2 h of my very first study section meeting ever to realize that "New Investigator" checkbox applicants from genuine newbies did very poorly and all of these were being scooped up by very well established and accomplished investigators who simply hadn't been NIH funded. Perhaps they were from foreign institutions, now hired in the US. Or perhaps lived on NSF or CDC or DOD awards. The idea that it took NIH something like 8-10 years to realize this is difficult to stomach.

**The R29 was crippled in terms of budget, btw. and had other interesting features.

***lolsob

****Yep, that would be my demographic.

12 responses so far

Apr 30 2018

Diversity statements from faculty candidates are an unfair test of ideology?

Someone on the twitts posted an objection:

to UCSD's policy of requiring applicants for faculty positions to supply a Statement of Contribution to Diversity with their application.

Mark J Perry linked to his own blog piece posted at the American Enterprise Institute* with the following observation:

All applicants for faculty positions at UCSD now required to submit a Contribution to Diversity Statement (aka Ideological Conformity Statements/Pledge of Allegiance to Left-Liberal Orthodoxy Statements)

Then some other twitter person chimed in with opinion on how this policy was unfair because it was so difficult for him to help his postdocs students with it.


Huh? A simple google search lands us on UCSD's page on this topic.

The Contributions to Diversity Statement should describe your past efforts, as well as future plans to advance diversity, equity and inclusion. It should demonstrate an understanding of the barriers facing women and underrepresented minorities and of UC San Diego’s mission to meet the educational needs of our diverse student population.

The page has links to a full set of guidelines [PDF] as well as specific examples in Biology, Engineering and Physical Sciences (hmm, I wonder if these are the disciplines they find need the most help?). I took a look at the guidelines and examples. It's pretty easy sailing. Sorry, but any PI who is complaining that they cannot help their postdocs figure out how to write the required statement are lying being disingenuous. What they really mean is that they disagree with having to prepare such a statement at all.

Like this guy Bieniasz, points for honesty:


I am particularly perplexed with this assertion that "The UCSD statement instructions (Part A) read like a test of opinions/ideology. Not appropriate for a faculty application".

Ok, so is it a test of opinion/ideology? Let's go to the guidelines provided by UCSD.

Describe your understanding of the barriers that exist for historically under-represented groups in higher education and/or your field. This may be evidenced by personal experience and educational background. For purposes of evaluating contributions to diversity, under-represented groups (URGs) includes under-represented ethnic or racial minorities (URM), women, LGBTQ, first-generation college, people with disabilities, and people from underprivileged backgrounds.

Pretty simple. Are you able to understand facts that have been well established in academia? This only asks you to describe your understanding. That's it. If you are not aware of any of these barriers *cough*Ginther*cough*cough*, you are deficient as a candidate for a position as a University professor.

So the first part of this is merely asking if the candidate is aware of things about academia that are incredibly well documented. Facts. These are sort of important for Professors and any University is well within it's rights to probe factual knowledge. This part does not ask anything about causes or solutions.

Now the other parts do ask you about your past activities and future plans to contribute to diversity and equity. Significantly, it starts with this friendly acceptance: "Some faculty candidates may not have substantial past activities. If such cases, we recommend focusing on future plans in your statement.". See? It isn't a rule-out type of thing, it allows for candidates to realize their deficits right now and to make a statement about what they might do in the future.

Let's stop right there. This is not different in any way to the other major components of a professorial hire application package. For most of my audience, the "evidence of teaching experience and philosophy" is probably the more understandable example. Many postdocs with excellent science chops have pretty minimal teaching experience. Is it somehow unfair to ask them about their experience and philosophy? To give credit for those with experience and to ask those without to have at least thought about what they might do as a future professor?

Is it "liberal orthodoxy" if a person who insists that teaching is a waste of time and gets in the way of their real purpose (research) gets pushed downward on the priority list for the job?

What about service? Is it rude to ask a candidate for evidence of service to their Institutions and academic societies?

Is it unfair to prioritize candidates with a more complete record of accomplishment than those without? Of course it is fair.

What about scientific discipline, subfield, research orientations and theoretical underpinnings? Totally okay to ask candidates about these things.

Are those somehow "loyalty pledges"? or a requirement to "conform to orthodoxy"?

If they are, then we've been doing that in the academy a fair bit with nary a peep from these right wing think tank types.

__
*"Mark J. Perry is concurrently a scholar at AEI and a professor of economics and finance at the University of Michigan's Flint campus." This is a political opinion-making "think-tank" so take that into consideration.

84 responses so far

Feb 07 2018

Undue influence of frequent NIH grant reviewers

A quotation

Currently 20% of researchers perform 75-90% of reviews, which is an unreasonable and unsustainable burden.

referencing this paper on peer review appeared in a blog post by Gary McDowell. It caught my eye when referenced on the twitts.

The stat is referencing manuscript / journal peer review and not the NIH grant review system but I started thinking about NIH grant review anyway. Part of this is because I recently had to re-explain one of my key beliefs about a major limitation of the NIH grant review system to someone who should know better.

NIH Grant review is an inherently conservative process.

The reason is that the vast majority of reviews of the merit of grant applications are provided by individuals who already have been chosen to serve as Principal Investigators of one or more NIH grant awards. They have had grant proposals selected as meritorious by the prior bunch of reviewers and are now are contributing strongly to the decision about the next set of proposals that will be funded.

The system is biased to select for grant applications written in a way that looks promising to people who have either been selected for writing grants in the same old way or who have been beaten into writing grants that look the same old way.

Like tends to beget like in this system. What is seen as meritorious today is likely to be very similar to what has been viewed as meritorious in the past.

This is further amplified by the social dynamics of a person who is newly asked to review grants. Most of us are very sensitive to being inexperienced, very sensitive to wanting to do a good job and feel almost entirely at sea about the process when first asked to review NIH grants. Even if we have managed to stack up 5 or 10 reviews of our proposals from that exact same study section prior to being asked to serve. This means that new reviewers are shaped even more by the culture, expectations and processes of the existing panel, which is staffed with many experienced reviewers.

So what about those experienced reviewers? And what about the number of grant applications that they review during their assigned term of 4 (3 cycles per year, please) or 6 (2 of 3 cycles per year) years of service? With about 6-10 applications to review per round this could easily be highly influential (read: one of the three primary assigned reviewers) review of 100 applications. The person has additional general influence in the panel as well, both through direct input on grants under discussion and on the general tenor and tone of the panel.

When I was placed on a study section panel for a term of service I thought the SRO told us that empaneled reviewers were not supposed to be asked for extra review duties on SEPs or as ad hoc on other panels by the rest of the SRO pool. My colleagues over the years have disabused me of the idea that this was anything more than aspirational talk from this SRO. So many empaneled reviewers are also contributing to review beyond their home review panel.

My question of the day is whether this is a good idea and whether there are ethical implications for those of us who are asked* to review NIH grants.

We all think we are great evaluators of science proposals, of course. We know best. So of course it is all right, fair and good when we choose to accept a request to review. We are virtuously helping out the system!

At what point are we contributing unduly to the inherent conservativeness of the system? We all have biases. Some are about irrelevant characteristics like the ethnicity** of the PI. Some are considered more acceptable and are about our preferences for certain areas of research, models, approaches, styles, etc. Regardless these biases are influencing our review. Our review. And one of the best ways to counter bias is the competition of competing biases. I.e., let someone else's bias into the mix for a change, eh buddy?

I don't have a real position on this yet. After my term of empaneled service, I accepted or rejected requests to review based on my willingness to do the work and my interest in a topic or mechanism (read: SEPs FTW). I've mostly kept it pretty minimal. However, I recently messed up because I had a cascade of requests last fall that sucked me in- a "normal" panel (ok, ok, I haven't done my duty in a while), followed by a topic SEP (ok, ok I am one of a limited pool of experts I'll do it) and then a RequestThatYouDon'tRefuse. So I've been doing more grant review lately than I have usually done in recent years. And I'm thinking about scope of influence on the grants that get funded.

At some point is it even ethical to keep reviewing so damn much***? Should anyone agree to serve successive 4 or 6 year terms as an empaneled reviewer? Should one say yes to every SRO request that comes along? They are going to keep asking so it is up to us to say no. And maybe to recommend the SRO ask some other person who is not on their radar?

___
*There are factors which enhance the SRO pool picking on the same old reviewers, btw. There's a sort of expectation that if you have review experience you might be okay at it. I don't know how much SROs talk to each other about prospective reviewers and their experience with the same but there must be some chit chat. "Hey, try Dr. Schmoo, she's a great reviewer" versus "Oh, no, do not ever ask Dr. Schnortwax, he's toxic". There are the diversity rules that they have to follow as well- There must be diversity with respect to the geographic distribution, gender, race and ethnicity of the membership. So people that help the SROs diversity stats might be picked more often than some other people who are straight white males from the most densely packed research areas in the country working on the most common research topics using the most usual models and approaches.

**[cough]Ginther[cough, cough]

***No idea what this threshold should be, btw. But I think there is one.

18 responses so far

Jun 18 2016

Power in the NIH review trenches

That extensive quote from a black PI who had participated in the ECR program is sticking with me.

Insider status isn't binary, of course. It is very fluid within the grant-funded science game. There are various spectra along multiple dimensions.

But make no mistake it is real. And Insider status is advantageous. It can be make-or-break crucial to a career at many stages.

I'm thinking about the benefits of being a full reviewer with occasional/repeated ad hoc status or full membership.

One of those benefits is that other reviewers in SEPs or closely related panels are less likely to mess with you.

Less likely.

It isn't any sort of quid pro quo guarantee. Of course not. But I guarantee that a reviewer who thinks this PI might be reviewing her own proposal in the near future has a bias. A review cant. An alerting response. Whatever.

It is different. And, I would submit, generally to the favor of the applicant that possesses this Mutually Assured Destruction power.

The Ginther finding arose from a thousand cuts, I argue. This is possibly one of them.

3 responses so far

Jun 17 2016

Nakamura reports on the ECR program

If I stroke out today it is all the fault of MorganPhD.

Jeffery Mervis continues with coverage of the NIH review situation as it pertains to the disparity for African-American PIs identified in 2011 (that's five years and fifteen funding rounds ago, folks) by the Ginther report.

The main focus for this week is on the Early Career Reviewer program. As you will recall, this blog has advocated continually and consistently for the participation of more junior PIs on grant review panels.

The ECR program was created explicitly to deal with underrepresented groups. However, what happened is that there was immediate opposition which insisted that the ECR program had to be open to all junior faculty/applicants, regardless of representation in the NIH game.

One-quarter of researchers in ECR's first cohort were from minority groups, he notes. “But as we've gone along, there are fewer underrepresented minorities coming into the pool.”
...
Minorities comprise only 13% of the roughly 5100 researchers accepted into the program (6% African-American and 7% Hispanic), a percentage that roughly matches their current representation on study sections.

Ok, but how have the ECR participants fared?

[Nakamura] said ECR alumni have been more than twice as successful as the typical new investigator in winning an R01 grant.

NIIIIIICE. Except they didn't flog the data as hard as one might hope. This is against the entire NI (or ESI?) population.

The pool of successful ECR alumni includes those who revised their application, sometimes more than once, after getting feedback on a declined proposal. That extra step greatly improves the odds of winning a grant. In contrast, the researchers in the comparison group hadn't gone through the resubmission process.

Not sure if this really means "hadn't" or "hadn't necessarily". The latter makes more sense if they are just comparing to aggregate stats. CSR data miners would have had to work harder to get this isolated to those who hadn't revised yet, and I suspect if they had gone to that effort, they could have presented the ESIs who had at least one revision under their belt. But what about the underrepresented group of PIs that are the focus of all this effort?

It's also hard to interpret the fact that 18% of the successful ECRs were underrepresented minorities because NIH did not report the fraction of minorities among ECR alumni applicants. So it is not clear whether African-Americans participating in the program did any better than the cohort as a whole—suggesting that the program might begin to close the racial gap—or better than a comparable group of minority scientists who were not ECR alumni.

SERIOUSLY Richard Nakamura? You just didn't happen to request your data miners do the most important analysis? How is this even possible?

How on earth can you not be keeping track of applicants to ECR, direct requests from SROs, response rate and subsequent grant and reviewing behavior? It is almost as if you want to look like you are doing something but have no interest in it being informative or in generating actionable intelligence.

Moving along, we get a further insight into Richard Nakamura and his position in this situation.

Nakamura worries that asking minority scientists to play a bigger role in NIH's grantsmaking process could distract them from building up their lab, finding stable funding, and earning tenure. Serving on a study section, he says, means that “those individuals will have less time to write applications. So we need to strike the right balance.”

Paternalistic nonsense. The same thing that Scarpa tried to use to justify his purge of Assistant Professors from study sections. My answer is the same. Let them decide. For themselves. Assistant Professors and underrepresented PIs can decide for themselves if they are ready and able to take up a review opportunity when asked. Don't decide, paternalistically, that you know best and will refrain from asking for their own good, Director Nakamura!

Fascinatingly, Mervis secured an opinion that echoes this. So Nakamura will surely be reading it:

Riggs, the only African-American in his department, thinks the program is too brief to help minority scientists truly become part of the mainstream, and may even exacerbate their sense of being marginalized.

“After I sat on the panel, I realized there was a real network that exists, and I wasn't part of that network,” he says. “My comments as a reviewer weren't taken as seriously. And the people who serve on these panels get really nervous about having people … that they don't know, or who they think are not qualified, or who are not part of the establishment.”

If NIH “wants this to be real,” Riggs suggests having early-career researchers “serve as an ECR and then call them back in 2 years and have them serve a full cycle. I would have loved to do that.”

The person in the best position to decide what is good or bad for his or her career is the investigator themself.

This comment also speaks to my objection to the ECR as a baby-intro version of peer review. It isn't necessary. I first participated on study section in my Asst Prof years as a regular ad hoc with a load of about six grants, iirc. Might have been 2 less than the experienced folks had but it was not a baby-trainee experience in the least. I was treated as a new reviewer, but that was about the extent of it. I thought I was taken seriously and did not feel patronized.

__
Related Reading:
Toni Scarpa to leave CSR

More on one Scientific Society’s Response to the Scarpa Solicitation

Your Grant In Review: Junior Reviewers Are Too Focused on Details

The problem is not with review…

Peer Review: Opinions from our Elders

23 responses so far

Jun 11 2016

NIH Director Collins and CSR Director Nakamura continue to kick the funding disparity can down the road

A News piece in Science by Jeffrey Mervis details the latest attempt of the NIH to kick the Ginther can down the road.

Armed with new data showing black applicants suffer a 35% lower chance of having a grant proposal funded than their white counterparts, NIH officials are gearing up to test whether reviewers in its study sections give lower scores to proposals from African-American applicants. They say it’s one of several possible explanations for a disparity in success rates first documented in a 2011 report by a team led by economist Donna Ginther of the University of Kansas, Lawrence.

Huh. 35%? I thought Ginther estimated more like a 13% difference? Oh wait. That's the award probability difference. About 16% versus 29% for white applicants which would be about a 45% lower chance. And this shows "78-90% the rate of white...applicants". And there was Nakamura quoted in another piece in Science:

At NIH, African-American researchers “receive awards at “55% to 60% the rate of white applicants,” Nakamura said. “That's a huge disparity that we have not yet been able to seriously budge,” despite special mentoring and networking programs, as well as an effort to boost the number of scientists from underrepresented minorities who evaluate proposals.

Difference vs rate vs lower chance.... Ugh. My head hurts. Anyway you spin it, African-American applicants are screwed. Substantially so.

Back to the Mervis piece for some factoids.

Ginther..noted...black researchers are more likely to have their applications for an R01 grant—the bread-and-butter NIH award that sustains academic labs—thrown out without any discussion...black scientists are less likely to resubmit a revised proposal ...whites submit at a higher rate than blacks...

So, what is CSR doing about it now? OK HOLD UP. LET ME REMIND YOU IT IS FIVE YEARS LATER. FIFTEEN FUNDING ROUNDS POST-GINTHER. Ahem.

The bias study would draw from a pool of recently rejected grant applications that have been anonymized to remove any hint of the applicant’s race, home institution, and training. Reviewers would be asked to score them on a one-to-nine scale using NIH’s normal rating system.

It's a start. Of course, this is unlikely to find anything. Why? Because the bias at grant review is a bias of identity. It isn't that reviewers are biased against black applicants, necessarily. It is that they are biased for white applicants. Or at the very least they are biased in favor of a category of PI ("established, very important") that just so happens to be disproportionately white. Also, there was this interesting simulation by Eugene Day that showed a bias that is smaller than the non-biased variability in a measurement can have large effects on something like a grant funding system [JournalLink].

Ok, so what else are they doing?

NIH continues to wrestle with the implications of the Ginther report. In 2014, in the first round of what NIH Director Francis Collins touted as a 10-year, $500 million initiative to increase the diversity of the scientific workforce, NIH gave out 5-year, $25 million awards to 10 institutions that enroll large numbers of minority students and created a national research mentoring network.

As you know, I am not a fan of these pipeline-enhancing responses. They say, in essence, that the current population of black applicant PIs is the problem. That they are inferior and deserve to get worse scores at peer review. Because what else does it mean to say the big money response of the NIH is to drum up more black PIs in the future by loading up the trainee cannon now?

This is Exhibit A of the case that the NIH officialdom simply cannot admit that there might be unfair biases at play that caused the disparity identified in Ginther and reinforced by the other mentioned analyses. The are bound and determined to prove that their system is working fine, nothing to see here.

So....what else ?

A second intervention starting later this year will tap that fledgling mentoring network to tutor two dozen minority scientists whose R01 applications were recently rejected. The goal of the intervention, which will last several months, is to prepare the scientists to have greater success on their next application. A third intervention will educate minority scientists on the importance of resubmitting a rejected proposal, because resubmitted proposals are three times more likely to be funded than a de novo application from a researcher who has never been funded by NIH.

Oh ff..... More of the same. Fix the victims.

Ah, here we go. Mervis finally gets around to explaining that 35% number

NIH officials recently updated the Ginther study, which examined a 2000–2006 cohort of applicants, and found that the racial disparity persists. The 35% lower chance of being funded comes from tracking the success rates of 1054 matched pairs of white and black applicants from 2008 to 2014. Black applicants continue to do less well at each stage of the process.

I wonder if they will be publishing that anywhere we can see it?

But here's the kicker. Even faced with the clear evidence from their own studies, the highest honchos still can't see it.

One issue that hung in the air was whether any of the disparity was self-inflicted. Specifically, council members and NIH officials pondered the tendency of African-American researchers to favor certain research areas, such as health disparities, women’s health, or hypertension and diabetes among minority populations, and wondered whether study sections might view the research questions in those areas as less compelling. Valantine called it a propensity “to work on issues that resonate with their core values.” At the same time, she said the data show minorities also do less well in competition with their white peers in those fields.

Collins offered another possibility. “I’ve heard stories that they might have been mentored to go into those areas as a better way to win funding,” he said. “The question is, to what extent is it their intrinsic interest in a topic, and to what extent have they been encouraged to go in that direction?”

Look, Ginther included a huge host of covariate analyses that they conducted to try to make the disparity go away. Now they've done a study with matched pairs of investigators. Valantine's quote may refer to this or to some other analysis I don't know but obviously the data are there. And Collins is STILL throwing up blame-the-victim chaff.

Dude, I have to say, this kind of denialist / crank behavior has a certain stench to it. The data are very clear and very consistent. There is a funding disparity.

This is a great time to remind everyone that the last time a major funding disparity came to the attention of the NIH it was the fate of the early career investigators. The NIH invented up the ESI designation, to distinguish it from the well established New Investigator population, and immediately started picking up grants out of the order of review. Establishing special quotas and paylines to redress the disparity. There was no talk of "real causes". There was not talk of strengthening the pipeline with better trainees so that one day, far off, they magically could better compete with the established. Oh no. They just picked up grants. And a LOT of them.

I wonder what it would take to fix the African-American PI disparity...

Ironically, because the pool of black applicants is so small, it wouldn’t take much to eliminate the disparity: Only 23 more R01 applications from black researchers would need to be funded each year to bring them to parity.

Are you KIDDING me? That's it?????

Oh right. I already figured this one out for them. And I didn't even have the real numbers.

In that 175 bin we'd need 3 more African-American PI apps funded to get to 100%. In the next higher (worse) scoring bin (200 score), about 56% of White PI apps were funded. Taking three from this bin and awarding three more AA PI awards in the next better scoring bin would plunge the White PI award probability from 56% to 55.7%. Whoa, belt up cowboy.

Moving down the curve with the same logic, we find in the 200 score bin that there are about 9 AA PI applications needed to put the 200 score bin to 100%. Looking down to the next worse scoring bin (225) and pulling these 9 apps from white PIs we end up changing the award probability for these apps from 22% to ..wait for it..... 20.8%.

Mere handfuls. I had probably overestimated how many black PIs were seeking funding. If this Mervis piece is to be trusted and it would only take 23 pickups across the entire NIH to fix the problem....

I DON'T UNDERSTAND WHAT FRANCIS COLLINS' PROBLEM IS.

Twenty three grants is practically rounding error. This is going to shake out to one or maybe three grants per year for the ICs, depending on size and what not.

Heck, I bet they fund this many grants every year by mistake. It's a big system. You think they don't have a few whoopsies sneak by every now and again? Of course they do.

But god forbid they should pick up 23 measly R01s to fix the funding disparity.

37 responses so far

Apr 12 2016

Bias at work

A piece in Vox summarizes a study from Nextions showing that lawyers are more critical of a brief written by an African-American. 

I immediately thought of scientific manuscript review and the not-unusual request to have a revision "thoroughly edited by a native English speaker". My confirmation bias suggests that this is way more common when the first author has an apparently Asian surname.

It would be interesting to see a similar balanced test for scientific writing and review, wouldn't it?

My second thought was.... Ginther. Is this not another one of the thousand cuts contributing to African-American PIs' lower success rates and need to revise the proposal extra times? Seems as though it might be. 

22 responses so far

« Newer posts Older posts »

:)