Repost: Program Interferes with The Funding Line

A comment from anony and an observation from YFS

At one institute ... they wouldn't even report what NRSA scores were fundable, just who got the grants and who didn't...Yeah, that's scientific all right.

remind me that one of the apparently less-obvious things about the selection of NIH grants for funding is that the study section score is only ONE part of the puzzle. I wrote about one of the other main influences on selection for funding in an entry posted to the old blog March 5, 2007. It seemed worth discussing again.


Discussion of the abysmal "funding line" among researchers is common these days. There are many related topics worthy of discussion but one issue that seems to be universal is a suspicion over the behavior of Program in funding grants that scored outside of the funding line. First, some definitions. "Program" here means the decision apparatus within individual Institutes (NIH is plural) such as NIDA, NIMH, NCI, NIAID, etc, which ranges from the professional administrative staff (Program Official) through the Institute Director advised by a peer group of senior scientists called the Advisory Council ("Council"). There are many ways to define the "funding line" but in most cases people refer to what I call the "hard" line meaning a proportion of grants that Institutes state they intend to fund each round. [UPDATE 8/5/08: Ok, on complaint from the PP, I need to revise this and the next bit because it is wrong, even if the idea is still valid.


We should really refer to the payline which is the percentile which ICs plan to use as their most-conservative cutoff for definitely-fund in each round. This is based on the pool of applications submitted to a given study section (in most cases, R01s only, etc) which are usually assigned to more than one IC. So an IC saying that they can fund everything that hits a 9%ile score does not necessarily mean the top 9% of grants assigned to their Institute. This is what PP was objecting to, I believe.]

There are three general submission dates and three Council meetings per year which make up a "round" for funding. At present the line is somewhere around 7-10%ile (Institutes vary, there are changes each submission round), in recent history during the budget "doubling" this number has trended more toward 15%ile or higher. Yet the NIH publishes grant award data which indicate that success rates are much higher, 20% now, around 30% during the "doubling". So what gives?
[Here is where I stand by the intent and underlying meaning of my message with this post. Even if the study-section based percentile is not identical to the same percentile of grants assigned to a given IC, I bet it is a decent proxy. It is certainly a qualitative proxy since the whole point here is that one hardly ever hears of grants not getting funded that score within a published (or even PO bandied-about) payline but one frequently hears of grants getting funded with percentiles that are worse than those of grants one hears of not getting funded (in the same round, at the same Institute). ]
Some of this, of course, is that the NIH stats refer to per-year success meaning that a project which is submitted and revised two rounds later counts as one application. But the real reason is the "soft line" behavior of Program in "reaching down" to "pick up" grants that did not make the hard funding percentile. WHAAAATTTT?? How can a grant which scores worse than mine get funded while mine is passed over?

Programmatic priorities dictate something other than the "best possible science" gets funded all the time. An Institute may decide that any of a whole host of issues are underrepresented in their portfolio for various reasons both internally scientific (i.e., Council recommendations, meeting or symposium discussions (Program attends meetings!), influential reviews, etc) or external (i.e., big media splash on some issue, Congressional "interest" via inquiry, Congressional mandate, etc). The Institute may decide that their portfolio is underrepresented with PIs of various gender, ethnic and geographic descriptions, under/overrepresented with grant mechanisms, New Investigators, etc. The Institute may decide that they "have an investment" in a given research program or resource and choose to keep it running. This really outrages people who fall just off the funding line and don't get their applications "picked up" as you can imagine.

Grow up. This is why the Institutes exist. The notion of pure investigator-initiated science is a good one, but much like "democracy" can't be carried to the extreme. Scientists, and the scientific enterprise, exhibit well discussed conservatism in many ways, see Nature editorial about Nobel-Prize-destined work being passed over. This is unsurprising given that we are human. We have a tendency to understand scientific models and domains that relate to our own work the best. We have a tendency to stick to these models and domains, particularly as our scientific careers mature. This is natural. But it means that the funding of science by the priorities of those doing the science leads to a suppression of innovation and novelty. Not to mention health domain coverage, the interest of the National Institutes of Health.

With that said, there is a problem with Program's behavior in that it is almost perfectly opaque. There is very little way to determine how many grants have been "picked up" at all. Imprecision in the budgeting/prediction/scoreoutcome process means that the number of grants funded in perfect line with the priority scores can vary due to unexpectedly low numbers of high scoring grants per round (percentiling is across three rounds), high scoring grants that meet Program priorities, etc. In any case, Program is very loathe to explain their "pick up" reasoning in specific terms no doubt hoping to avoid lengthy debates, Congressional inquiry and even lawsuits from someone who didn't get funded. On the balance this seems silly. If Program is going to assert a priority, do so honestly and forthrightly. Just say, we picked up X number of women PIs and Y number of New Investigators and Z grants between the Mississippi and the Sierra Nevada! And then explain why. If the reason is good enough to use, it is good enough to defend, no?
UPDATE: Apparently NASA funded scientists have similar issues.


One addendum is that I talked very broadly about "Program" interests. One reason is because, only seeing this from the outside, I have little understanding of how it works. One can pick up things here and there, though. Fairly open, is the fact that pickups can originate with lowly line POs, at Advisory Council or with the Director herself. I've heard mutterings from at least one Council member that they have little actual influence, shrug. Also mutterings (and some fairly convincing evidence from colleagues) that some Directors are very active in the sense of putting general personal priorities into practice by fingering grants.

Less open is the fairly obvious conclusion that ICs are political entities with varying and shifting influences. Some line POs (or more likely the whole Division/Branch unit of several POs) are probably more likely than others to get their pickup proposals approved. Some are more or less likely to engage in active behavior. I have little doubt that there is a calculus for how much they can go to bat for a pickup based on original score- harder to argue for a 240 than a 180, that sort of thing. Rules of thumb trickle out at times, such as "no way we're going to pick up over 200, if you'd only snuck in at a 195...". These are no doubt highly variable over time.

The take home for me on all of this is my usual pleading for PIs to establish and maintain good relationships with Program staff- if they are going to take a little extra effort on a proposal, putting your winning personality to a name on an app cannot possibly hurt. Also why I advise paying close attention to what Program (and particularly the Director!) are saying about funding priorities. Is it mercenary to follow funding priorities, to decide what applications to submit based on what ICs want? Sure, but do you want a career or not?

24 responses so far

  • PhysioProf says:

    There are many ways to define the "funding line" but in most cases people refer to what I call the "hard" line meaning a proportion of grants that Institutes state they intend to fund each round.

    This is not correct. You are confusing payline with success rate. Payline is a study section priority score %ile cutoff. Success rate is (roughly) the proportion of grants that an Institute intends to fund.

  • msphd says:

    Gotta agree with you. What is needed is transparency. Here's an idea: why don't they ADVERTISE which topics they want to fund AHEAD OF TIME, instead of using that as an excuse after the fact to decide who gets money regardless of score?
    Oh, right, because it actually could have nothing to do with the topic, that's just a good excuse.
    I know of too many examples where it's because of politics (e.g. the grant author has buddies on the council or the study section who owe him - usually it's a him - favors).
    Transparency would ruin this whole game for them, that's why they don't want it.

  • DrugMonkey says:

    This is not correct. You are confusing payline with success rate. Payline is a study section priority score %ile cutoff. Success rate is (roughly) the proportion of grants that an Institute intends to fund.
    Yeah, you always complain when I say stuff like this. Anyone else confused?
    payline is in fact an IC term and yes, the percentile is not based on the grants assigned to the given IC but the study section (or in some cases the 'CSR base') itself. Success rate at the IC level is, I assume, based on total grants assigned for possible funding (how are dual assignments counted ? :-)) and the number actually funded.
    we are talking relatively large numbers and relatively fixed review/submission behavior so that the ICs can say yes, we anticipate anything coming in this round with a 9%ile (or whatever payline they select) we can fund. They know they can do this because they know in fact they have enough money for many more proposals and the expected variance will not bust their budget.
    So where are the rest coming from? That was the point here. That there is a "the rest" above the grants that hit the payline and that the selection of "the rest" may not be in strict line of priority score.
    Do we really need to quibble over semantics?

  • bill says:

    Here's an idea: why don't they ADVERTISE which topics they want to fund AHEAD OF TIME
    You mean something like this? Or perhaps something more like this?

  • bill says:

    Do we really need to quibble over semantics?
    You don't *have* to, but it's fun to watch.

  • PhysioProf says:

    Dude, this is like fucking deja vu, but it's not "quibbling over semantics". It's getting the shit right and using NIH's own terminology correctly.
    http://www2.niaid.nih.gov/ncn/budget/balance.htm#benefit4
    Dillweed.

  • bsci says:

    Perhaps another way to look at this is that there aren't binary cutoff points depending on score. There is a distribution. (I'm making up numbers here) If you are in the top 3% you have a 100% chance of getting funded. If you are 3-5% then you have a 95% chance of getting funded, etc. Somewhere down the line there is a score where only 50% the people get funded and lower scores where they pluck out a few people for funding. I suspect the entire distribution is rather tight with most scores in the highly likely to get funded or not ranges.
    msphd, The institutes DO list priorities for R01s, but do a terrible job describing what they want for postdoc & grad student NRSAs. I don't know exactly how they can provide much more detail since if you want 5% of the awarded grants to go to developing widget painters, the breakdown of funding depends on whether 3% or 10% of the fundable range grants are in this topic.

  • CC says:

    The institutes DO list priorities for R01s, but do a terrible job describing what they want for postdoc & grad student NRSAs.
    It's probably just my cluelessness (and having been in a megalab where I I got thrown in the deep end on the first morning I walked in) but until MsPhD's recent post, I didn't even know there were scores and a general review-like process for NRSAs. I'd thought you just got a yes/no letter like from foundation fellowships, except with committee comments. I've been meaning to dig out the offer letter from my files and see if it had a score attached.
    Certainly all my institution ever told me was that my benefits were being terminated...

  • PhysioProf says:

    Perhaps another way to look at this is that there aren't binary cutoff points depending on score. There is a distribution. (I'm making up numbers here)

    You're also making up reality. Some Institutes have absolutely hard paylines. This means that every single fucking grant within the payline %ile gets paid (assuming there aren't administrative problems with it, like you propose to develop a human-animal chimera or something). Institutes with hard paylines also reserve funds to pay some grants outside the payline.
    Other Institutes have soft paylines, and don't necessarily pay all grants within that soft payline.

  • DrugMonkey says:

    Ok, we are talking at partial cross-purposes here because PP is right and I was wrong in the original post. At a semantic level and an important one that leads to much confusion. I've tried to redress that in an update.
    I am left, however, with conviction that the points I am trying to express are valid. In short that there is a substantial gap between the grants that are selected for funding by study section decision alone (the payline) and the total number funded by the IC. That is the main point here. The number of awards I know of personally that were funded with numbers way off the relevant payline is substantial. I suppose all my colleagues could be lying about their scores over the years but...why would they?
    I have little doubt that ICs vary in the size of their pick-up window and, as expressed in my original post, find it frustrating that they are not more open about their behavior.
    As a final note, if you take overall NIH success rate numbers, an appreciation of the size of each Institute and the aggregate payline for all ICs then my point is indeed valid as originally written. (Unless there is some magical Lake Wobegone effect of all study sections being above average that I am missing) Less precise because not every IC publishes a payline, but still valid.

  • PhysioProf says:

    PP is right and I was wrong

    HAHAHAHAHAHAHAH!!!!!!!
    I love being right!
    You just made my entire fucking day! w00t!!!

  • TeaHag says:

    The RFAs (requests for applications) are one set of indicators that can be used to identify research aspects that program favors. It is well worth pouring over these lists to see if there's something there that may apply to your research interest. It means that some funding has been earmarked. The scoring etc. still applies, but it can only help to know that the NIH actively wants your application!!

  • bsci says:

    PP, I don't think you actually disagree with me.
    Let's say all institutes can fund 100 of 1000 applications. Institutes with hard paylines set their threshold to cover 90 grants and set aside funding for another 10 grants below the payline. I'm assuming, even with those final 10 grants, there's sum relationship between ranking and which are funded (i.e. something right below the hard threshold is still more likely to be funded than something with a much lower score).
    An institute with a soft payling might set the soft threshold to cover 80 of 100 grants and allow a slightly wider distribution for the rest of the grants on both sides of that cutoff.
    Unless you are saying institutes completely ignore the scores when choosing beyond the initial threshold, I didn't think I was making an overly complex statement.
    I love being right! You just made my entire fucking day!
    Simple pleasures for simple minds. 🙂

  • bill says:

    [whine]I got spamfiltered.[/whine]

  • bsci says:

    I realized I flipped by numbers in the example. In the hypothetical case, an IC with a hard threshold could fund the top 80 of 100 and select the next 20 from those ranked 80-120. An IC with a soft threshold could set that threshold at 90 and still select from those ranked 80-120. While the specific rules and metrics might differ across ICs, whether the threshold is hard or soft is irrelevant since a hard threshold could merely mean that the threshold is set lower and more money is reserved for grants outside the payline.

  • [...] grant funding. Part of that process is a long game of developing interpersonal relationships with the Program Officers that staff the NIH ICs of interest to our individual research areas. It is indubitably the case [...]

  • [...] touched on this ever so briefly before, see Program Interferes with.... and NIH Administrators [...]

  • [...] grant funding. Part of that process is a long game of developing interpersonal relationships with the Program Officers that staff the NIH ICs of interest to our individual research areas. Many scientists find the [...]

  • I got this web site from my pal who shared with me on the topic of this website and
    at the moment this time I am visiting this web page
    and reading very informative articles at this time.

  • […] of sustained NIH funding includes the long game of developing interpersonal relationships with the Program Officers that staff the NIH ICs of interest to our individual research areas. Sure, they do turn over a bit […]

  • What's up, this weekend is nice designed for me, because this time i am reading this impressive informative piece of writing here at my home.

  • Thanks for one's marvelous posting! I quite enjoyed reading it,
    you may be a great author. I will make sure to bookmark your blog
    and will come back down the road. I want to encourage
    that you continue your great posts, have a nice
    weekend!

  • […] of sustained NIH funding includes the long game of developing interpersonal relationships with the Program Officers that staff the NIH ICs of interest to our individual research areas. Sure, they do turn over a bit […]

  • […] of sustained NIH funding includes the long game of developing interpersonal relationships with the Program Officers that staff the NIH ICs of interest to our individual research areas. Sure, they do turn over a bit […]

Leave a Reply