NHLBI gets serious about revision churning

Nov 11 2008 Published by under Grantsmanship, NIH, NIH Budgets and Economics

Now we're talkin'! The National Heart Lung and Blood Institute (NHLBI) is looking to take the final step necessary to break the cycle of unnecessary churning of grant revisions. As I have mentioned a time or two, I think that study sections have evolved into a position in which only revised grant applications are considered seriously. Furthermore I am convinced that this refusal to award top scores until the grant has been revised at least once is a big waste of time and effort in many cases. A "waste" because I dispute that the conduct of the eventual science is changed one bit by this process. The grant proposal itself, sure, a better document. But the resulting science? The investigator is (rightfully) going to pursue the science that makes the most sense to her, not whatever she has been forced to write by grantspersonship concerns.

I've also expressed my skepticism that the recent limitation of new grant proposals to a single round of revision is going to do much to overcome this study section tendency. Of course, I had a suggestion in a post on October 15, 2008:

The policy that I think would have been much better is one in which the NIH/CSR issues percentile ranks by revision status. That way ICs could simply apply a heavy bias for -01 applications over revisions to maintain whatever their target proportions of A0, A1 and A2 applications might be.

Apparently I am not the only one to reach this rather obvious conclusion; the NHLBI has published the following proposal which was considered at the October 21, 2008 meeting of the NHLBI Advisory Council:

Based on its analysis of recent NHLBI funding, the Committee concurred with the NIH decision to eliminate A2s. (Note: The Committee commenced its work prior to the NIH decision.) NHLBI funding data from the post-doubling era (FY 2004 - FY 2007) show that a large proportion of applications with an A0 percentile score fairly far down the percentile ranking were eventually funded, often as A1s or A2s. For example, about 80 percent of applications from established investigators (85 percent from new investigators) with an A0 score at the 20th percentile were eventually funded. The data also indicate a marked shift toward A2s and away from A0s. Furthermore, the data show substantial improvement in percentile scores between A0 applications and their A1s, and likewise between A1 applications and their A2s, for new investigators as well as established investigators. Based on these data, the Committee recommended eliminating A2s.

Next, the Committee considered percentiling A0s separately from A1s. FY 2007 data show this policy would have substantially lowered the percentile scores of A0s and substantially raised the scores of A1s. A retrospective model that re-percentiled NHLBI R01s and R21s by amendment status (and excluded A2s) showed that 267 resubmissions could have been avoided in FY 2007, 199 resubmissions avoided in FY 2006, and 31 resubmissions avoided in 2005.

Apparently the proposal was well received.

The Council enthusiastically supported the proposed policy.

[h/t: microfool]

3 responses so far

  • Pinko Punko says:

    It's just arranging the deck chairs. Perfectly fine proposals are dinged for essentially meaningless reasons. When you are only funding 10%, the policies won't have much meaning. Certainly, it wastes people's time for their grants to have to go three rounds, but doesn't this just suggest are large amount of essentially similarly scorable proposals? There just is nothing between a large fraction of these grants.

  • DrugMonkey says:

    Actually Pinko, I see this as ceasing some of the deck-chair arranging so that people are freed up to load a few more lifeboats...

  • Pinko Punko says:

    I guess my problems with the reviews are that some projects will never be fundable at a 10% funding level. Ever. Given that resources are required for more ambitious projects, and resources are required for preliminary data that makes ambitious projects seem worthwhile, new investigators are certainly at a disadvantage on many fronts. Sometimes it seems that even well written grants get meaningless dings, but perhaps are eventually funded because in the two years of two rounds, there is preliminary data that can show progress. What I worry about now is them just declaring certain projects not fundable. The problem is there is more reasonable science than there is money. And careful, methodical progress (singles and doubles) does not excite reviewers like audacious plans for triples and home runs, when certainly there are enough labs that are established and have triple and HR factories ready to go.

Leave a Reply