Well, well, well. As my more dedicated readers are well aware one of my ongoing criticisms of the peer review of NIH grants is the seeming obsession with revision status of the application. I've just reposted this old entry from 2007.
I was, quite naturally, sensitized to this issue originally as a grant applicant. As with many of you, I developed this sneaking suspicion that complaints and bad scores on the original submission of some of my proposals were not in good faith. I mean this in the usual sense that the same or essentially unchanged parts of the proposal were stomped on very hard on the first submission and essentially ignored later. Also from the growing realization that essentially none of my more-junior colleagues and friends had received an award un-revised.
Some years down the road, I entered service on a study section and in hearing the way grants of the three different revision steps were reviewed, well, I started to suspect.
Sure enough the data* bore out my suspicion that it was hard to get a grant funded unrevised. Particularly in the study section to which I was appointed but it was a general trend. At one point I put together some numbers across a couple of related study sections, did the same thing across a few of the brain ICs and came up with a rough estimate that only about 10% of funded competing R01s got there unrevised. (The best analysis is by study section because it tends to get rid of the RFA issue which artificially enhances first-submission funding.)
More importantly, I became convinced that the revision process didn't do anything! Convinced that when you looked at the issues supposedly driving the critiques, the responses on the revision and adding a real-world consideration of the conduct of science.....that there would be no resulting change in the eventual conduct of the research. None of any significance anyway. Remember this was by seeing the eventual fate from original to A1 or A2 funding for a fair number of grants, albeit in one study section. Seeing the actual proposals, revisions and critiques. Hearing what was said in review.
We can talk endlessly about why this was happening, of course. I blame it partly on ignorance (when I started
ranting raising the issue with people involved in review I got a lot of blank stares initially), partially on the inevitability of the budget doubling and subsequent flatline and partly by self-referential cultural factors that simply go un-examined.
I came to realize that a major act of the NIH was going to be required to fix things. I recommended at some point or other simply holding over some fraction of the grants from a prior Council round for funding in the next round to deal with the backlog effect. This would have only been a partial band-aid. But it recognized that there were a whole boatload of original submission grants that were just fine and were only being scored lower because of the holding pattern. All those perfectly fine prior proposals that were back in revised form as A1s and A2s. I was pushing a "fish or cut bait" line and asking why we'd want to put PIs through the waste of time of revising and resubmitting when we all knew they'd be very likely to get funded eventually anyway.
Then NIH limited the revisions to just the single A1. I was not a fan because I suspected that in short order we would just return to the A2's showing up as thinly disguised new submissions. Well, NHLBI seems to be getting serious. A new strategy has been published:
In order to fund meritorious science earlier, the NHLBI has adopted a new funding policy that substantially decreases the need for investigators and universities to submit revised applications. The Institute will develop separate lists ranking applications according to amendment status, resulting in equivalent success rates for both unamended (A0) and amended (A1/A2) applications. A more extensive discussion of this new funding policy can be found at the following site: "Early Notice: Rebalancing Success Rates in FY 2010."
The percentile rank for funding will be 16% for an original -01A0 but only 9% for A1s and 7% for any grandfathered A2s. Now why o why have they done this?
Following the quoted link we find:
The NHLBI has observed a marked decline in the funding success rate of original, unamended versions of new and competing renewal grant applications, both of which are denoted with the grant number suffix A0. The NHLBI has also observed a corresponding increase in the number of grant applications that are funded after being amended and resubmitted once (denoted A1) or twice (denoted A2). An analysis of applications from established investigators submitted between fiscal years 2004-2007 revealed that resubmissions typically benefit by receiving marked improvements in percentile scores. The analysis indicated that a minimum of 70 percent of A0 applications receiving percentile rankings of 25 or better were eventually funded, either in their original (A0) or amended (A1 or A2) forms. Data from the NHLBI are consistent with those of other Institutes.
Exactamundo! The only question is how to break the cycle. This is one way, just put everybody on the new-proposal footing and too bad if you are one of the NHLBI proposals that is caught in the middle of this process. Frankly, the ARRA/stimulus disturbance is not a bad place to put a major upheaval if you ask me.
*all you have to do is go to CRISP and search for new grants (e.g., "1R01%") in a given fiscal year and gate by IC, study section or some such.