Where the NIGMS argument doesn't add up

Jul 08 2016 Published by under Fixing the NIH, NIH Budgets and Economics

The NIGMS continues its ongoing argument for funding more labs with ever decreasing amounts of grant funding in a new Feedback Loop post.

This one focuses, yet again, on "productivity" as assessed by publication counts and (this time) citations of those publications. It is, as always, significantly flawed by ignoring the effects of Glamour publications. I've done that before and it is starting to bore me. In short, you cannot compare apples to oranges because of the immense difference in the cost of generating your average Nature paper versus a Brain Research paper. And citations don't help because getting into a Glam journal does not mean your paper will get any particular number of citations. Furthermore, there is very little chance that papers that cost 10 or 20 times more will generate ten or twenty times the citations, on average, given the skew in citation distributions and the fact that Glam journals are only hitting means in the 30-40 range. Finally, their "efficiency" measures completely ignore the tremendous inefficiencies of interrupted funding, which is a reality under the current system and also not necessarily fixed with their spread-the-wealth schemes.

The real issue of the day is the opinion of the fans of NIGMS's "conclusion*", which reads:

Overall, however, the data suggest that supporting a greater number of investigators at moderate funding levels is a better investment strategy than concentrating high amounts of funding in a smaller number of researchers.

The Sally Rockey blog entry on "mythbusting" is relevant here. As of FY2009 about 72% of NIH funded investigators had one RPG. Another 20% had two and maybe 5% had three.

That's all.

The NIGMS data analyses are big on fitting productivity lines to about the single R01 level of direct costs (~$200K per year) and showing how the productivity/cost drops off as the grant funding increases. Take a good look at the most recent analysis. Linear productivity up to $300K direct costs with the 75%ile sustained all the way to $500K. The famous original 2010 analysis by Jeremy Berg at NIGMS is pretty similar in the sense that you don't get much change in the line fit to mean publications until you get to the $600-$700K direct costs range.

There is a critical point in lining up these two bits of information which is that the NIGMS policy intent is not supported by their analysis and it can't be. One or two RPG level from Rockey's post should be interpreted in full modular R01 terms ($250K direct, usually cut to $200K, $225K direct and in NIGMS' case to 4 years by default) with a little bit of float upwards for the rare cases. Consequently, it is obvious that most NIH awardees operate in the ~$200-250K part of NIGMS' dataset. Another 20% operate in the $400-$500K direct range. In other words, well within the linear part of the productivity/cost curve.

Mean publications as represented by the 2010 Berg analysis are increasing linearly well up to the three to four grant level of $750K direct costs.

In either case, the "inefficient" grant levels are being obtained by a vanishingly small number of investigators.

Fine, screw them, right?

Sure....but this does nothing to address either the stated goal of NIGMS in hedging their bets across many labs or the goal of the unfunded, i.e., to increase their chances substantially.

A recent Mike Lauer Blog post showed that about a third of those PI's who seek RPG funding over a rolling 5 year interval achieve funding. Obviously if you take all the multi-grant PIs and cut them down to one tomorrow, you'd be able to bump funded investigators up by 15-20%, assuming the FY2009 numbers are relatively good still**. It isn't precise because if you limit the big guys to one award then these are going to drift up to $499K direct at a minimum and a lot more will have special permission to crest the $500K threshold.

There will be a temporary sigh of relief and some folks will get funded at 26%ile. Sure. And then there will be even more PIs in the game seeking funding and it will continue to be a dogfight to retain that single grant award. And the next round of newbies will face the same steep odds of entry. Maybe even steeper.

So the ONLY way for NIGMS' plan to work is to cut per-PI awards way, way down into the front part of their productivity curves. Well below the point of inflection($300-500K or even $750K depending on measure) where papers-per-grant dollar drops off the linear trend. Even the lowest estimate of $300K direct is more than one full-modular grant. It will take a limit substantially below this level*** to improve perceptions of funding ease or to significantly increase the number of funded labs.

Which makes their argument based on those trends a lie, if they truly intend it to support their "better investment strategy". Changing the number of investigators they support in any fundamental way means limiting per-PI awards to the current full modular limit (with typical reductions) at the least, and very likely substantially below this level to produce anything like a phase change.

That's fine if they want to just assert "we think everyone should only have X amount of direct costs" but it is not so fine if they argue that they have some objective, productivity-based data analysis to support their plans. Because it does not.

__
*This is actually their long standing assertion that all of these seemingly objective analyses are designed to support.

**should be ballpark, given the way Program has been preserving unfunded labs at the expense of extra awards to funded labs these days.

***I think many people arguing in favor of the NIGMS type of "small grants for all" strategy operate from the position that they personally deserve funding. Furthermore that some grant award of full modular level or slightly below is sufficient for them. Any dishonest throwaway nod to other types of research that are more expensive (as NIGMS did "We recognize that some science is inherently more expensive, for example because of the costs associated with human and animal subjects.") is not really meant or considered. This is somewhat narrow and self-involved. Try assuming that all of the two-granters in Rockey's distribution really need that amount of funding (remember the erosion of purchasing power?) and that puts it at more like 92% of awardees that enjoy basic funding at present. Therefore the squeeze should be proportional. Maybe the bench jockeys should be limited to $100K or even $50K in this scenario? Doesn't seem so attractive if you consider taking the same proportional hit, does it?

22 responses so far

  • Pinko Punko says:

    How much basic research is in the other institutes? GM has burden to keep basic science alive.

  • drugmonkey says:

    Lots. GM has no burden greater than any other IC even if they had an exclusive mission, which they don't.

  • MorganPhD says:

    There is plenty of "basic research" at other institutes. Especially if we expand the definition of basic to include the new hot buzz word of "pre-clinical". I'd guess that at least 60% of all NIH grants (maybe not $$$ amount) are "pre-clinical".

    It's just getting harder and harder to make the argument that studying something like the yeast cell cycle is "cancer research" now. So they go to NIGMS.

    I'm also not excited by the trend of the NIH sending obviously disease-relevant grants to NIGMS because the work is done in flies or worms instead of a mouse. That's bogus.

  • jmz4 says:

    Ban mouse research! Then we'll get way more publications per grant dollar. In fact we could skip the whole "doing science" part and just spend all day writing papers. Much more efficient and productive.

  • Philapodia says:

    Maybe NIH should just get rid of modular budgets all together and budget for what you actually think you'll need. If you need more money to do mouse work, ask for more. If you do cheaper science, ask for less. However, since most DCs cover salaries and not research expenses "bench jockeys should be limited to $100K or even $50K in this scenario" would be grossly unfair.

  • JL says:

    Wouldn't it be great if asking for less money would increase your chances of funding? Since money is finite and it clearly is a zero sum game (overall), how come doing things more cheaply has no advantage in the funding game?

    No wonder people throw the sink in a proposal to convince reviewers that they have the most awesomest model, costs be damned. Penny pinching is for later.

    Imagine if asking for $85k direct had a very high chance of success. It would shift the type of science done.

  • dsks says:

    "Imagine if asking for $85k direct had a very high chance of success. It would shift the type of science done."

    I confess, I feel a tad ridiculous even applying for R15s, which are super competitive now anyway, when $100K/yr directs is vastly in excess of what I need to do the research I do (hard money teaching/research U). That said, all the funding mechanisms are highly competitive now, there's really no low-hanging fruit in this gig to shoot for, no matter what $ value you're looking for.

  • That said, all the funding mechanisms are highly competitive now, there's really no low-hanging fruit in this gig to shoot for, no matter what $ value you're looking for.

    Actually, the R03 mechanism may be what you're looking for. It's two years max $50K per year. I don't know the data on R03 success rates, but in many study sections, they are reviewed essentially as "are the proposed studies worth spending $100,000 on?" In my study section, if you submit a credible R03 with proposed studies worth doing, you'll get a clearly fundable score.

  • A Salty Scientist says:

    Sadly, NIGMS doesn't normally use the R03 or R21 mechanisms.

  • Grumble says:

    "Maybe NIH should just get rid of modular budgets all together "

    Most soft-money investigators probably don't bother with submitting modular grants anymore. Why bother, when you ask for a technician, a post-doc, a student and half your salary, and the total in salaries alone is close to the modular limit? Putting together a detailed budget is marginally more complicated, so why not just ask for what you need to do the project?

    I wonder, though, whether reviewers have some degree of preference for modular grants, perhaps because asking for $350k/year is seen as greedy.

  • jmz4 says:

    "Imagine if asking for $85k direct had a very high chance of success. It would shift the type of science done."
    -It would shift where the science is done, that is for sure, by making hard money salaries way more competitive for funding. I also think it would promote the diffusion of focus in any particular lab (so you could nab 3 85k non-overlapping grants), which I would argue would be a bad thing.

    "I wonder, though, whether reviewers have some degree of preference for modular grants, perhaps because asking for $350k/year is seen as greedy."
    -I don't really understand why the budget is even submitted to a study section, initially. The consideration of the budget is supposed to be separate from the relevance of the work, right? So it creates a huge extra burden of review, because the reality is that the budget is only relevant to the proposals that get funded.

    Or is it understood on these panels that you only read the budgets once something achieves a fundable score?

  • drugmonkey says:

    The R01 mechanism permits a proposal for two modules for two years. Instant "R03".

  • Philapodia says:

    Any reviewer seeing a R01 proposal asking for just 2 modules over two years will think the PI is smoking crack.

  • Joe says:

    "I wonder, though, whether reviewers have some degree of preference for modular grants, perhaps because asking for $350k/year is seen as greedy."

    Not in my study section. If you are going to ding the proposed budget, you have to have a good reason and a specific proposal for how to reduce it. It might be something like "It's not going to take the post-doc in the collaborator's lab 5 years to do the work listed, reduce the budget for the post-doc to two years." Recently I heard one going the other way, "This new investigator can't do this work for the budgeted amount, recommend increasing the budget by 50k per year."

    "Or is it understood on these panels that you only read the budgets once something achieves a fundable score?"

    No. You don't know when you review the proposal if it is going to be in the fundable range (unless you are planning to tank it). So you should bring up any problems with the budget in the initial review.

  • Grumble says:

    "If you are going to ding the proposed budget, you have to have a good reason"

    And if you are just cheesed off that Prof. Bigwig is biting off more than his fair share, you might just not mention that and instead ding the grant for something else (which you might otherwise overlook) or just by damning it with faint praise. There are enough free parameters in the review process that doing that is perfectly possible. Which makes me think it's probably not uncommon.

    "it creates a huge extra burden of review"

    Not really. Usually I pretty much ignore the budget. I just glance over it to see if there's anything egregiously out of whack (e.g., asking for $1 million/year to run some gels). And to do an informal salary survey to make sure I'm not being underpaid.

  • AnonNeuro says:

    I think the ever-shrinking budgets requires a "return to the bench" for us soft-money folks. It made sense long ago for us to take 50% salary to manage the group of 5+ trainees covered by an R01. But now, if you can only pay yourself, a single student and half a technician, you're spending far too much on administration if you don't produce some data yourself.

    Unfortunately, I'm guessing most of that supported time goes to applying for the (elusive) second R01.

  • JL says:

    @jmz, naturally, a higher likelihood of funding for low-budget applications would have to be paired with a decreased benefit with increasing funding or grants. Otherwise people just split the applications and we end up with the same problems, just split into many more grants.

  • qaz says:

    Great. More grants to write instead of doing real work. Just what the system needs.

  • A Salty Scientist says:

    The R01 mechanism permits a proposal for two modules for two years. Instant "R03".

    The proper number of Specific Aims modules for your NIH grant...

    Dude, you recently preached to not deviate from reviewer expectations. I can't imagine that grants with 1/3 of the scope would fare well in comparison to standard R01s. Not that I'm convinced this is a good idea anyway for above-mentioned reasons.

  • drugmonkey says:

    I'm just saying... if your target IC has no R03 mech, this is a workaround. also see R21.

  • Pinko Punko says:

    A number of study sections previously have funded much basic, now comments on model organism related studies are increasingly "this is not human/not directly medically relevant"- these proposals are increasingly restricted to GM. This is anecdata, but appears to be trend.

  • drugmonkey says:

    This is the usual bullshit anecdata of everyone responding to the ongoing budget woes and increasing mouths at the trough problem.

    "NIH is no long funding [insert favored model, topic domain, clinical/animal/basic, wtfever] and the dismal results of me and my homies at study section proves it."

    One hears this all the time and from all types of interests, sometimes with what appears to be coherent ebb and flow but that seems at the level of chance and/or pendulum swinging to me.

    Of course "basic" is always as defined by various interest groups. Some people would consider BRAINI to be a massive Basic Science Restoration Eleventy. Others might claim that the mere fact that it is dedicated mostly to studying living neurons in situ means that it is not basic but rather [insert favored term for unapplied non-bench-jockey science].

Leave a Reply