Your Grant In Review: "More than adequately revised...". (Updated)

The NIH grant applications which will be reviewed Jun/Jul are going out to reviewers right about now. Poking through my pile of assignments I find that I have three R01 applications at the A2 stage (the second and "final" amendment of a brand new proposal). Looking over the list of application numbers for the entire panel this round, I see that we have about 15% of our applications on the A2 revision.
Oi. What a waste of everyone's time. I anticipate many reviewers will be incorporating the usual smackdown-of-Program language. "This more than adequately revised application...."


I am not a fan of the NIH grant revision process, as readers will have noticed. Naturally my distaste is tied to the current era of tight budgets and expanding numbers of applications but I think the principles generalize. My main problem is that review panels use the revision process as a way of triaging the review process. This has nothing to do with selecting the most meritorious applications for award and everything to do with making a difficult process easier.

ReviewBiasGraph1The bias for revised applications is supported by funding data, round-after-round outcome in my section as well as supporting anecdotes from my colleagues who review. Start with CRISP and search for new applications (1R01% is a handy wildcard) gated by your favorite study section or three. Or gate by your usual funding IC. What you will quickly notice is that only about 10% of applications reviewed in normal CSR sections get funded without being revised. (If you do an IC search results will be contaminated by SEPs and in-house study sections. There is no easy way to discriminate RFA-funded proposals which are "unrevised". I use quotes because there are cases in which an RFA may be reissued a year later such that some of the responding applications are revisions of previously-reviewed applications.) If you care to step back Fiscal Year by Fiscal Year in the CRISP search, you will notice the relative proportions of grants being funded at the unrevised (-01), A1 and A2 stages have trended for more revising in concert with the budget flattening. I provide an example for a single study section here but since the overall numbers are low (~20-30 grants funded each FY), you should really run the numbers for several study sections to get a feeling for broad trends. Another thing you will notice if you review a series of closely related study sections is that the relative "preference" for giving high scores to -01, A1 and A2 applications varies somewhat between sections. This is analysis is perhaps unsurprising but we should be very clear that this does not reflect some change in the merit or value of revising applications; this is putting good applications in a holding pattern.

[Update 05/07/08: I notice that writedit points to a powerpoint from the Great Zerhouni which includes (slide #57) a graph much like my example! Also note slide #54 which includes the GZ's little brag about New Investigator awards which were headed for the toilet in FY2006 and then they restored something like historical norms for FY2007. Which they accomplished with Program pickups! What NIH officiousness seems to fail to comprehend is that the problem needs to be fixed at the study section level. And that their proud stripping of assistant professors from the study sections (Scarpa presentation, slide #42) is working against this! ]

Getting back to the applications assigned to the study section on which I serve for this round, I note that many of these A2 applications received a gray zone score the last time. Percentile ranks that are in the neighborhood of 15%. While hard paylines have been running 8-10%ile recently, until we have more data we can assume that the eventual funding rate will reflect something more like 15% or more of all applications being funded. So some of these were close but not picked up the last time. If history is any judge, unless the PIs have really screwed up, the reviewers will respond to this situation by assigning scores in the 1.2 range and writing critiques meant to communicate "Will you just fund this thing already?" to the Program staff. Program will pick them up and everyone is happy, right?

No, everyone is NOT happy. It takes a lot of effort to revise a grant application, even when essentially no substantive changes are made. It takes a lot of effort for three people to review a grant application. And finally, it takes a lot of effort for the PI who has submitted her new-submission R01 that has essentially no chance of being considered closely in the study section discussion, nor of being funded. The grant-revision holding pattern wastes a lot of real NIH dollars too, a consideration that never seems to be part of the debate. For many PIs, the time they spend on grant writing is time they are not spending on optimizing the output of their lab. No, it is not supposed to work like this, technically NIH funded effort is not supposed to be spent on grant writing. But this is a scam. Of course, 100% time can be narrowly construed as 40 hrs per week, so anything over this you spend on grant writing is off the NIH clock. But c'mon. Lab resources are being maintained while awaiting funding too. Can you just fire your highly trained tech while revising your grant and just hire back a replacement 18 mo later? Hell no, you make compromises wherever you can just to keep the lights on, so to speak. Shift the tech salary (not to mention PI effort) onto the other grant you hold until the one you are revising hits. Which, of course, compromises the output of that grant. Maintaining mouse lines? Expensive research subjects like nonhuman primates? Access to institutional equipment and space (use it or lose it)? Check, check, check. There is a huge amount of taxpayer money being wasted while we natter about insisting that revised grants are "better". We can improve on this.

So this brings me back to my usual proposal of which I am increasingly fond. The ICs should set a "desired" funding target consistent with their historical performance, say 24% of applications, for each Council round. When they do not have enough budget to cover this many applications in a given round, they should roll the applications that missed the cut into the next round. Then starting the next Council round they should apportion some fraction of their grant pickups to the applications from the prior rounds that were sufficiently meritorious from a historical perspective. Perhaps half roll-over and half from the current round of submissions. That way, there would still be some room for really outstanding -01 apps to shoulder their way into funding
The great part is that essentially nothing would change. The A2 app that is funded is not going to result in scientific conduct that differs in any substantial way from the science that would have resulted from the A1/15%ile app being funded. New apps will not be any more disadvantaged by sharing the funding pie with prior rounds than they currently are facing revision-status-bias at the point of study section review.

Yet a great deal of time and effort would be saved.
..and maybe. Just maybe. This would let study section members back away from the revision bias abyss and get serious about ranking merit at the -01 stage. Over a few rounds, it might even be the case that prioritizing A2 applications for the very top scores wouldn't be such an obsession. And every so slooooowly, we might see the proportion of grants being funded at the -01 stage go back up. Personally I'd like to see as many as 70% of grants get funded unrevised.
__
UPDATE: PhysioProf supplies the long-term trends for all funded grants by revision status. Interesting to see how it developed over time. It reminds me of when the limitation for a maximum of two revisions of a given application were put into place in Oct 1996. One major argument was the relatively small number of applications that got funded as A2s and the further rarity of applications getting funded on additional revision. Assessing current trends by that logic should mean a return of A3 and A4 revisions, shouldn't it? NewR01GrantsByRevision

13 responses so far

  • TreeFish says:

    Bravo, DM.
    I love this idea, but it still wouldn't save time by decreasing the frequency of 'the waltz' (or would it?): revise an -A0 with an -A1, submit -A1, study section reviews it; revise -A1 with an -A2, submit the -A2, study section reviews it, only to have Program fund the -A1 (or -A0 in the first round).
    Either way, it would make a PIs life a heck of a lot easier. Even if Program has to cut direct costs by 20% for each grant to afford such a system (a dubious claim).

  • As someone looking at a stack with two A2s under 21%ile, I concur with your proposal that a going-through-the-motions resubmission needlessly sucks the energy of both the applicant and the reviewers.
    If I understand your proposal correctly, TreeFish is indeed correct that the *time* it takes to get one's near-miss grant funded won't be any shorter. But I still support your idea in that it reduces the needless effort of the applicant who put in an excellent application and the reviewers', "This more than adequately revised application...." dance.
    It is indeed amazing to see how the number and frequency of first submissions getting funded rose steadily after the 1996 -A2 limit was established. But now we have this erosion of first submissions and massive upcreep -A1 and -A2 apps. Sounds like we're ready for a change of some sort.
    btw, TreeFish, we're already being hit with 11-18% DC cuts at award and some ICs are cutting another 3% for Director set-asides. Not sure that even my most creative rebudgeting skills could tolerate another 20% cut.

  • DrugMonkey says:

    TreeFish and Abel,
    It is not so much that this is a sure fire all-at-once replacement. My thought is that it changes, over time, the calculus for PIs holding a just-miss score. Much like now if you were maybe 1% off the line and the PO was saying all the right things, you might decide to take a chance and not revise. That is where the first savings come in for the applicant. And TreeFish, while the revise-within-one-round is supposedly coming, it is most certainly not here yet and even when it is possible, some fraction of PIs are simply not going to be able to turn a revision around in the next round. So there would be an automatic 4mo savings right there.
    The imposition of budget cuts post-funding doesn't have anything to do with my proposal as far as I can tell....my thoughts are conditioned on whatever the available pool of $$ may be.

  • The imposition of budget cuts post-funding doesn't have anything to do with my proposal as far as I can tell....my thoughts are conditioned on whatever the available pool of $$ may be.
    Yeah, yeah, I got ya - I was just responding to TreeFish's idea that they would take a 20% DC cut for your plan to be implemented. My point was that reforms need to be made without any more hits to post-award amounts.

  • whimple says:

    If history is any judge, unless the PIs have really screwed up, the reviewers will respond to this situation by assigning scores in the 1.2 range and writing critiques meant to communicate "Will you just fund this thing already?" to the Program staff.
    These reviewers are gaming the system. I'm starting to see the wisdom in the NIH's Peer Review Self-Study's concept of not allowing any revisions.

  • TreeFish says:

    I gotcha. The impression of the PI with a 16%ile when the payline is 13%ile would be to presume that the 16%ile would get funded next round if not this round, hence saving the time/hassle of revisions and rereview. It makes sense, let's hope the peeps at NIH are Drugmonkeyheads!
    By the way, I'm meeting with the Chair of a Dept next week. I have my wishlist figured out, any advice on how to convince the Chair to grant my wishes? The wishlist is ~$600k (equip, computers, tech, usage fees for core facilities). I'm hoping that my K99/R00 will help the Chair decide I'm worth it! Any advice, beyond that already bequeathed in DM's and PP's priceless posts, would be appreciated.

  • PhysioProf says:

    Let me provide a real example. One of my R01s was scored 170/30%ile as a A0, 140/10%ile as an A1, and 110/2%ile as an A2.
    The difference between the A0 and the A1 was scientifically relevant. The difference between the A1 and the A2 was not.
    The reason that all of this is occurring is that the study sections are behaving rationally in response to what I consider an ineliminable feature of any peer review system. Peer review systems are reasonably good at somwehat-objectively distinguishing the the top 20% of grant applications from the rest.
    What they are horrible at doing--and for unavoidable reasons that have to with the subjective nature of scientific taste and judgement--is in some repeatable, consistent, objective manner distinguishing the top 10% from the next 10%. But it is this latter task that study sections are essentially being forced into, as only the top 10% are getting funded.
    So, in my opinion, what study sections are doing is trying to make this impossible and unfair discrimination that they are forced to engage in as fair as possible. And the way they are doing this is to say, "OK. If a grant made it to near the border of the top 10% on a previous submission, but didn't get funded, and then comes back, we are going to 'make sure' that it gets funded this time."
    The motivation behind this is to try to ensure that capricious "wiggle" around the funding cut-off line is not the difference between funding and absence of funding. And study sections care a lot more about this when pay-lines are 10%ile than when they are 20%ile, because it means that the grants that would be capriciously denied funding with 10%ile pay-lines are scientifically much better than those that would be denied funding with 20%ile pay-lines.
    This is not to say that this kind of study section behavior doesn't introduce other sorts of unfairness or inefficiency into the system. The point is that their behavior is rational and aimed at ameliorating what is perceived as a particularly harmful form of unfairness.
    (And I think this is my post for today! I'll gin this up into a complete post later.)

  • PhysioProf says:

    By the way, I'm meeting with the Chair of a Dept next week. I have my wishlist figured out, any advice on how to convince the Chair to grant my wishes?

    (1) Convince her that you need all those resources in order for your research program to have a good likelihood of reaching escape velocity.
    (2) Reveal to her a genuine offer from another institution that is providing at least that amount of start-up.
    (3) Make it very clear that you will not tolerate any reference to the R00 as a reason for decreasing the start-up, while still not seeming greedy.

  • BugDoc says:

    I realize that by definition assistant professors are less experienced at review than more senior faculty, and all of the hooha around the difficulties of funding these days puts the process of peer review under a microscope. However, is there any evidence that assistant professors are not doing a good job in peer review? My guess is that junior PIs put more effort in their reviews to make sure they are doing a decent job. Although senior PIs are more experienced, they are also likely to be more busy, and may not be likely to spend as much time with each grant.
    "...their proud stripping of assistant professors from the study sections..." To be fair to senior PIs, in my experience, many do try to make sure that discussion about new investigators' grants doesn't get too discouraging.

  • blop says:

    I'm sorry, I think I didn't really catch the idea.
    Let's take a concrete example: 100 new proposals/year.
    Year 1, the committee decides that 20% of the grants should be awarded but has money for 12% only. 12 grants are funded, 8 will wait next year.
    Year 2, 100 new grants, 20% deserve funding. The committee funds the 8 of last year + 4 new proposals, 16 will wait next year.
    Year 3, again 20 new proposals to fund + 16 from last year and still money for only 12 of them. So all of the new proposals + 4 of last year's proposal will wait next year.
    Year 4... I guess you got the point. The number of grants in the waiting line increases too fast to make the process sustainable.

  • DrugMonkey says:

    blop, first of all just FYI, the NIH has three rounds for consideration of applications for funding each year. irrelevant for your point, but just for accuracy in case some readers are unaware.
    The number of grants in the waiting line increases too fast to make the process sustainable.
    You are overlooking the degree to which the pool of applicants is inelastic. Not fixed exactly, just not infinitely expanding. Applicants also continue applying until they get sufficient funding, then they quit applying for a while. It only seems as though PIs must apply constantly whether they have awards in hand or not. I may be an anecdote of one but I suspect my experience generalizes. When I have limited time remaining on a single grant my rate of submission is considerably higher than when I have multiple awards in hand. Even when I anticipate a good chance of funding based on an attractive score, this decreases my submission rate.

  • […] back in 2008 I expressed my dissatisfaction with the revision-cycle holding pattern that delayed the funding of NIH […]

  • […] sounds suspiciously similar to a plan that I advanced some time ago. This post from 2008 was mostly responding to the revision-queuing behavior of study […]

Leave a Reply