A question was recently posted to my prior post that pointed out the disingenuous arguments made by the NIH in the wake of Hoppe et al. (2019). There are data in the supplement to Hoppe et alia that show clearly that applications with Black PIs fare worse across all five topic-cluster quintiles ranked on success rate. So the implication that the disparity of funding to applications with Black PIs would disappear if they just applied in equal proportion within the more successful topics is falsified by their own data (see Table S6).

aspirational skunk asks:

Fig S6 of Hoppe et al. (2019) seems to show that WH PIs experience funding rate advantages across all topics, be they “AA/B preferred” or not. But then the “re-analysis” by the Lauer et al. (2021) topic choice paper writes that, well aschully, “peer review outcomes were similar” and that presumably Fig S6 can be explained away by ICs with lower award rates.

I do not understand Lauer et al (2021)’s argument. Doesn’t Fig S6 indicate that AA/B PIs experience funding rate disadvantages *within ICs* (if topic cluster is a good enough proxy for ICs)? So the fact that ICs with more AA/B applicants have lower funding rates is kinda besides the point? I’ve been looking, but isn’t it the case that Lauer et al. (2021) doesn’t show within IC funding rates for AA/B and WH PIs? My generous takeaway from the Lauer et al. (2021) paper is that yes, disaggregating statistics is important for isolating the largest IC contributions to disparities. Am I missing something here?

My answer was that is really hard for me to work through the various defenses and excuses of the NIH. As an example, they used some inferential statistics to show there was “no difference” in exception pay behavior in Hoppe et al. My calculation suggests that about half again as many applications were funded to white PIs at 35%ile and worse scores, as were funded to Black PIs, all at 34%ile or better. They say “look, exception pay decisions are not the issue” based on the lack of significance in their inferential analysis. To me it is overwhelmingly obvious that they are doing a serious red herring with stats to arive at this conclusion.

I agree with the comment that the S6 data show very clearly that Black PI apps are at a disadvantage no matter *which* topic-success quintile they fall into.

Lauer et al 2021 is also a little funny because it focuses on all of the applications in “AAB preferred topics”. Most of the PIs of those applications are not AAB, of course. It’s just that of the AAB PI apps, there is a preference for certain topics. By “preference” they mean that half of AAB PI applications map to 10% of the topic clusters. Which of course also means that half do not. The analysis also focuses on AAB “preferred” Institutes or Centers, which really means ICs that NIH prefers to assign those applications to. Remember, PIs can ask for assignment, and in some cases the FOA makes it inevitable, but on the whole the NIH itself is the one deciding which I or C to assign, or not assign, as a funder. If you look at Table 1, the AAB (%) column is telling. NINR gets 4.66% of their apps with AAB PIs and NIMHD gets 14.84%. After that, their cutoff where to decide on AAB “preference” ICs is entirely arbitrary.

To make a point, I’ve taken the liberty of graphing the data from Table 1. The red squares are the ICs used in the Lauer analysis as “AAB preferred” ICs. The blue triangles are a selection of ICs which I’ve arbitrarily identified for reasons discussed below.

Beyond these two, the next three included in their analysis (Fogarty is excluded) are NIDCR 2.03%, NIAID 2.08% and NICHD 3.08% of applications with AAB PIs. The next most-preferred are NIEHS 2.01%, NIDA 1.83%, NIMH 1.72%, NIDDK 1.57%, NHLBI 1.51%. It is clear that NIMHD and NINHR are very far outliers and that the remaining included ICs group much better with a host of other ones in the middle of the distribution. Particularly NIEHS (asthma? lead paint? smog?), NIDA (substance use by race?) and NIMH (see Harnett, 2020), and possibly NIDDK (hello metabolic disorders in Black folks) and NIHLB (heart attack, ditto).

So why did they draw the cutoff where they did? “Table 2 shows review and funding outcomes for applications according to whether the assignment was to an IC in the top quartile of proportion of applications with AAB PIs“. An analysis choice which is simply stated, unexplained and undefended. As a reminder this is a catchment that accounts for more than a quarter of the applications with Black PIs, something around 28% by my rough count. Yet the topic analysis differs slightly “We designted (sic) a topic as ’AAB Preferred’ if it was among the 15 topics that accounted for 50% of AAB applications.” Why? That comes from 10% of the topics, whereas if they’d used 25% of the most-preferred AAB topics they would have captured 70% of the applications with AAB PIs. Why did they make these decisions? What impact would the use of other arbitrary cutoffs have for the overall conclusions? who knows???

It sure smells like p-hacking to me.

After that, Lauer et al 2021 report that for the total sample of apps (white and AAB PIS) in the “AAB preferred topics”, the merit rankings are not different from other topics, but that they are less likely to be funded. From there it is another jump to show a disproportionate assignment of the total sample of apps in these topic domains to the poorly-funded ICs. Although of course Lauer talks about success rates at those ICs, chalks it up to “funding ecology” and does not address the fact that NIMHD and NINR get low amounts of the NIH allocation and that is the real problem. For grins, I’ve also graphed the IC %AAB apps against the IC’s percent of the extramural allocation from Fiscal Year 2015 (application sample is 2011-2015), as provided in Table 1 by Lauer et al. NIMHD gets 0.99% and NINR gets 0.52% of the budget, NCI gets 18.3% and NIAID gets 16.1% by these numbers.

After that the paper gets into the weeds of PI race by referring to a change in the value of the regression coefficient in their probit model in Table 5. I mean…come on. There is nowhere that just shows us success rates for AAB and white PI apps within IC. Or scores for each category within topics. It all hides behind fancy statistics. Take Tables 3 and 4. They would be the ideal place to give aggregate statistics by PI race. They do not.