NINDS joins the grant-funding-outcome party

Nov 14 2012 Published by under NIH, NIH Budgets and Economics

First it was NIAMS which joined NIGMS in publishing funding outcome data that is of high interest to applicants. I'm referring to the number of grants funded/not funded by priority score. NIGMS has presented these data for ages and the National Institute of Arthritis and Musculoskeletal and Skin Diseasesput theirs up for the first time just this summer.

The new data from NINDS for Fiscal Year 2011 are remarkably consistent with the datasets posted by NIGMS and by NIAMS.

This graph presents the R01 data, inclusive of new and competing continuation applications and of both experienced and New Investigators/Early Stage Investigators. You can visit their page for further graphical breakdowns.

What is readily apparent from this graph, as with all similar graphs that I've seen to date, is that there is a readily perceptible percentile-rank payline under which essentially all grants are funded. Above this apparent payline, funding is possible but comparatively rare. Furthermore, in this zone of exception funding (aka grey zone, aka "pickups") the chances of getting funded are best for apps with scores closest to the apparent payline. The probability of the exception funding decreases as a function of this distance.

This tells you, as always, that the major input to the system continues to be the score ranks (and therefore percentiles) decided by the study section review. The relative impact of Program decisions to fund grants out of order is comparatively small. In the broader scheme.

But yes, if you are one of those with an application in the 15-18%ile that didn't get funded by NINDS while they were picking up 25+%ile scores this is a HUGE impact. I've been on both sides of this fence (I assume) in the past so....yeah.

The rather interesting bits in all of this have to do with my comment about "remarkably consistent" and "apparent payline". As you know, some of the NIH ICs publish their paylines formally and some do not. In fact, many of the latter insist that they do not in fact have a payline. As it happens, most ICs from whom I tend to seek money fall into the latter category. What you realize over the years of calling Program Officers to beg for some small indication of your grant's chances of funding is that they all have paylines. There is always some internal sense of what kind of score is going to be a near-certainty, what is a "no-way" and what might be arguable depending on the final details. What you also gain a vague picture of, through this and through talking to your peers about what funded and what did not, a distribution much like the above figure. So, as we start to see more and more ICs (with different alleged payline policies) post their data, we can see that they all behave more or less the same when it comes to funding grants by initial review scores/percentiles.

There are also certain ICs that bruit it about that they stick strictly to the order of peer review. I.e., that they never skip a grant under the payline and that, more importantly, that they never fund differentially above the payline. They imply quite strongly, or state outright, that grants with equal ranks above the payline will all be funded or not funded.

I have been immensely skeptical of such claims and insisted that I wouldn't believe it until I see the funding data.

NINDS is one of these ICs that claims/is claimed to fund in "strict order".

These data show that is totally false.

I LOVE being right.

[h/t: PhysioProf]

5 responses so far

  • Eli Rabett says:

    The interesting thing is the ratio of funded to total proposals over the pay line. There appear to be two things, first, that a relatively high number (50%) of those near the pay line are funded. Second that there is a Maxwell Boltzmann distribution of those out to 30% or so that are funded, and the question is why the second maximum, are these high risk/high payoff, newbies or what.

  • Virgil says:

    As ER alludes to above, I think your line "Above this apparent payline, funding is possible but comparatively rare" is wrong. It is rare at the far edge yes, but close in to the payline there's a 50%+ chance that you might get funded.

    Presumably, since this data includes ESIs/NIs, then the stuff close in to the payline is the grants from such individuals, but still getting funded even though they're beyond the nominal cut off? It's not until we get out beyond 20% that a real "tail" starts to develop.

  • DrugMonkey says:

    You are underlining my point, Virgil. The vast majority of the input remains the study section. This is all responding to a sense you get from people complaining about Program's pickups that they think it is random, or uninfluenced by score.

    Ps. If you follow the links, they have more breakdowns by ESI, etc. )

  • DrugMonkey says:

    In fairness, if you use the apparent Paylines for experienced, NI and ESI apps differentially, the exceptions seem less likely than in the NIGMS datasets. But there are still *some* being picked up our of order.

  • [...] NINDS joins the grant-funding-outcome party Great whites ‘not evolved from megashark’ Have your soda and lose weight too?: A new Pepsi product in Japan claims to do just that. DNA sequencing of MRSA used to stop outbreak A Few Things to Consider Before Eating Wild Mushrooms [...]

Leave a Reply