NIH always jukes the stats in their favor

Oct 04 2016 Published by under Gender, NIH, NIH Careerism, NIH funding

DataHound requested information on submissions and awards for the baby MIRA program from NIGMS. His first post noted what he considered to be a surprising number of applications rejected prior to review. The second post identifies what appears to be a disparity in success for applicants who identify as Asian* compared with those who identify white.

The differences between the White and Asian results are striking. The difference between the success rates (33.8% versus. 18.4%) is statistically significant with a p value of 0.006. The difference between the the all applications success rate (29.4% versus 13.2%) is also statistically significant with a p value of 0.0008. Finally, the difference between the probabilities of administrative rejection (15.4% versus 28.1%) is statistically significant with p = 0.007.

There was also a potential sign of a disparity for applicants that identify as female versus male.

Male: Success rate = 28.9%, Probability of administrative rejection = 21.0%, All applications success rate = 22.8%

Female: Success rate = 23.2%, Probability of administrative rejection = 21.1%, All applications success rate = 18.3%

Although these results are not statistically significant, the first two parameters trend in favor of males over females. If these percentages persisted in larger sample sizes, they could become significant.

Same old, same old. Right? No matter what aspect of the NIH grant award we are talking about, men and white people always do better than women and non-white people.

The man-bites-dog part of the tale involves what NIGMS published on their blog about this.

Basson, Preuss and Lorsch report on the Feedback Loop blog entry dated 9/30/2016 that:

One step in this effort is to make sure that existing skews in the system are not exacerbated during the MIRA selection process. To assess this, we compared the gender, race/ethnicity and age of those MIRA applicants who received an award with those of the applicants who did not receive an award
We did not observe any significant differences in the gender or race/ethnicity distributions of the MIRA grantees as compared to the MIRA applicants who did not receive an award. Both groups were roughly 25% female and included ≤10% of underrepresented racial/ethnic groups. These proportions were also not significantly different from those of the new and early stage R01 grantees. Thus although the MIRA selection process did not yet enhance these aspects of the diversity of the awardee pool relative to the other groups of grantees, it also did not exacerbate the existing skewed distribution.

Hard to reconcile with DataHound's report which comes from data requested under FOIA, so I presume it is accurate. Oh, and despite small numbers of "Others"* DataHound also noted:

The differences between the White and Other category results are less pronounced but also favored White applicants. The difference between the success rates (33.8% versus. 21.1%) is not statistically significant although it is close with a p value of 0.066. The difference between the the all applications success rate (29.4% versus 16.2%) is statistically significant with a p value of 0.004. Finally, the difference between the probabilities of administrative rejection (15.4% versus 28.1%) not statistically significant with p = 0.14 although the trend favors White applicants.

Not sure how NIGMS will choose to weasel out of being caught in a functional falsehood. Perhaps "did not observe" means "we took a cursory look and decided it was close enough for government work". Perhaps they are relying on the fact that the gender effects were not statistically significant, as DataHound noted. Women PIs were 19 out of 82 (23.2%) of the funded and 63/218 (28.9%) of the reviewed-but-rejected apps. This is not the way DataHound calculated success rate, I believe, but because by chance there were 63 female apps reviewed-but-rejected and 63 male apps awarded funding the math works out the same.

There appears to be no excuse whatever for the NIGMS team missing the disparity for Asian PIs.

The probability of administrative rejection really requires some investigation on the part of NIGMS. Because this would appear to be a huge miscommunication, even if we do not know where to place the blame for the breakdown. If I were NIGMS honchodom, I'd be moving mountains to make sure that POs were communicating the goals of various FOA fairly and equivalently to every PI who contacted them.

Related Reading.
*A small number of applications for this program (403 were submitted, per DataHound's first post) means that there were insufficient numbers of applicants from other racial/ethnic categories to get much in the way of specific numbers. The NIH has rules (or possibly these are general FOIA rules) about reporting on cells that contain too few PIs...something about being able to identify them too directly.

19 responses so far

  • HRC_academic says:

    My comment on Datahound's blog is awaiting moderation, but I am infuriated by John's comment and Datahound's response that 'Language issues, grantsmanship,' could be a reasonable hypothesis to account for the disparity for Asian PI's not winning as many MIRAs by percentage compared to Whites. Why is it okay for them to raise these "rationales" to explain the Asian PI disparity, as a way to deflect potential implicit bias (aka racism)?

    Would they get away with using the same rationale for non-Asian minorities? Would one of them say, hey maybe Hispanic and Black PI's grants aren't 'articulate' enough to win a grant? It's their poorer grantsmanship, not their race, which by the way is an easy give-away for many Asians by their last name.

  • drugmonkey says:

    Why is it okay for them to raise these "rationales" to explain the Asian PI disparity, as a way to deflect potential implicit bias (aka racism)?

    It's okay to raise all sorts of issues for potential investigation of the source of the disparity. If you don't think it up, how are you going to figure out if it is a contributing factor or not?

    Now I agree, if NIGMS simply assumes language is the problem and walks away, that is a big problem.

    Would they get away with using the same rationale for non-Asian minorities?
    The NIH has definitely raised "grantsmanship" to explain and deflect just about every disparity that has ever been detected. Women, young investigators, Ginther finding on African-American PI disparity, ZIP code name it, they default to this.

    Would one of them say, hey maybe Hispanic and Black PI's grants aren't 'articulate' enough to win a grant?

    There have been things said about the Ginther finding that come as close to this as makes no difference, yes.

  • gzw says:

    Whew thank god someone's looking out for the Asian labs

    don't know how they'll ever survive let alone prosper at this rate

    US Asian population = 3%

    US Asian scientist = 18%

  • SidVic says:

    Yeah HRC i'm sympathetic to your view but.. I would bet you a million that the english-as-a-first language Asians (american born) have equivalent or better success rates than whites. Higher especially in those 1st generation raised by tiger moms. Go try to submit a grant in japanese.

    What the cali and ivy schools were doing with the gross subtraction of test scores from asians for the purposes of admission was indefensible and utter BS.

  • A. Tasso says:

    but... but... there is no such thing as anti-Asian discrimination in this country! the only minority groups worth helping with formal programs are blacks, Latinos, and Native American Indians.

  • Dave says:

    There's bias in the NIH review system??? That can't be right.

  • jojo says:

    DH's "white vs other" comparison:
    "Finally, the difference between the probabilities of administrative rejection (15.4% versus 28.1%) not statistically significant with p = 0.14 although the trend favors White applicants."

    DH's "white vs asian" comparison:
    "Finally, the difference between the probabilities of administrative rejection (15.4% versus 28.1%) is statistically significant with p = 0.007."

    WTH? These are the same numbers? Typo?

  • Ola says:

    4 groups....

    A. White Funded / B. White UnFunded

    C. URM Funded / D. URM UnFunded

    What DataHound did, was to ask whether A/(A+B) was different to C/(C+D).

    What NIGMS did, was ask whether C/(A+C) was different to D/(B+D).

    They're different questions, so the answers may be different. Let's say for example: A=30, B=70, C=15, D=45 (these are not the actual numbers but someone could figure it out if necessary).

    DataHound's question: A/(A+B)=30/(30+70)=30% vs. C/(C+D)=15/(15+45)=25%.
    Maybe these numbers are significantly different? The larger of the two is 20% higher

    NIGMS' question: C/(A+C)=15/(15+3o)=33.3% vs. D/(B+D)=45/(45+70)=39.1%
    Maybe these numbers are not significantly different? The larger of the two is 17% higher

    Yes, it sucks that someone in NIGMS chose to use the obfuscatory math way of looking at these numbers rather than the more infrmative one, but it wouldn't be the first time the government has fudged the numbers right?

  • jojo says:

    I calculated the percentages myself from raw data in DH OP and I get:

    % of white applications that are admin reject: 15.4%
    % of asian applications that are admin reject: 28.1%
    % of other applications that are admin reject: 23.5%
    % of asian + other (nonwhite) apps that are admin reject: 26.5%

    Looking at the OP now, I think DH may have accidentally written "28.1%" for the white vs other comparison paragraph - because he has it correct (23.5%) in an earlier part of the post (which didn't make it into drugmonkey's coverage).

    What test was used by NIH or DH to assess significance? I would think the appropriate test would be a contingency test but I don't see any reference to it (in the 5min I actually spent on this).

  • Elephant says:

    It's heartwarming to se, gzw, that racism is alive and well. Since there are too many PIs of Asian background, e.g. me, our grants should be downgraded. Otherwise, we might (gasp) prosper!

    I'm honestly puzzled at what you would tell an Asian-background PI whose grant is rejected. "You deserve it; there are too many of you?" That's exactly how your comment reads.

  • A. Tasso says:

    @Elephant, what is more telling is the fact that Asian PI's have achieved relative prosperity ***despite*** all of the biases and pervasive racism embedded in the system. The problem is that too many observers look at the relative prosperity and think "oh I guess we don't need to do anything about anti-Asian racism". WRONG. Just because Asian PI's demonstrate resiliency in the face of pervasive anti-Asian racism doesn't mean that the racism shouldn't be corrected with formal policies and programs. (Clearly I am emphasizing a *process* oriented approach to racism intervention rather than an *outcome* oriented approach.)

  • datahound says:

    jojo: I corrected a couple of typos that you noted on my blog. I determined p values using two-sided Fisher's exact test (2 x 2 contingency tables).

  • jmz4 says:

    So does this difference not persist in regular study section? Do we think this is the PO's biases creeping in when evaluating MIRA eligibility?

  • SidVic says:

    Ataso- Maybe there is just not that much racism. Why do you assume this is the case?
    What kind of process changes are you talking about?

  • socsciencegal says:

    @HRC_academic. I was watching the NIA council meeting live stream last week and one of the presenters, who was talking about accounting for lack of diversity in funded applicants, mentioned some analysis of grammar in past applications. I guess the hypothesis being tested was whether race/ethnic disadvantage found in the Ginther report could be due to worse grammar of certain groups (honestly, who thought this was a viable factor?). Anyway, they found that non-white applications had *fewer* grammatical errors. Also the most common grammatical error found, was use of commas.

  • Jojo says:

    @datahound - cool, thanks!

    Any thoughts /guesses on what test or tests NIH was doing?

  • A. Tasso says:

    @SidVic: anyone who says there is "not much racism" directed toward Asian PI's has little understanding of academia.

    I am referring to process changes like affirmative action programs, special grant opportunities, etc available to *all* groups who are systematically discriminated against, not just URM. (Asian PI's are not underrepresented but they are systematically discriminated against. Native American Indian PI's are underrepresented and they are systematically discriminated against.)

  • jmz4 says:

    What's interesting, and worth posting here, are the possible reasons, according to DH, that administrative rejections were handed down:
    According to a response to my question on the NIGMS Feedback Loop, the possible reasons are:

    The proposed research was outside the NIGMS mission.
    The applicant was not a New or Early Stage Investigator.
    The applicant received other R01-equivelent NIH support after submission of the MIRA application that changed their eligibility.
    The applicant received support from another funder (e.g., NSF) and that funder deemed the MIRA application to be overlapping with their grant, requiring withdrawal of the MIRA application.

    These seem like they should be pretty hard and fast rules (except maybe the first one). DH is apparently requesting data on the reasons the applications were declined. If it really is a racial bias on the part of the POs, I would expect most of the administrative rejections to fall under the first category.

  • HRC_academic says:

    @socsciencegal Thanks for noting that /bold/ [non-white applications had *fewer* grammatical errors] /bold/. I will highlight your comment in my response to 'John' on Datahound's blog. He is still standing by his dismissive claim.

    This is more evidence of racism thinly veiled by attempts to argue against implicit bias by claiming it's a merit-based determination of deficiency on the other groups.

Leave a Reply