Kudos to NHLBI for getting serious about breaking the grant revision cycle

Nov 06 2009 Published by under Uncategorized

Well, well, well. As my more dedicated readers are well aware one of my ongoing criticisms of the peer review of NIH grants is the seeming obsession with revision status of the application. I've just reposted this old entry from 2007.
I was, quite naturally, sensitized to this issue originally as a grant applicant. As with many of you, I developed this sneaking suspicion that complaints and bad scores on the original submission of some of my proposals were not in good faith. I mean this in the usual sense that the same or essentially unchanged parts of the proposal were stomped on very hard on the first submission and essentially ignored later. Also from the growing realization that essentially none of my more-junior colleagues and friends had received an award un-revised.
Some years down the road, I entered service on a study section and in hearing the way grants of the three different revision steps were reviewed, well, I started to suspect.


Sure enough the data* bore out my suspicion that it was hard to get a grant funded unrevised. Particularly in the study section to which I was appointed but it was a general trend. At one point I put together some numbers across a couple of related study sections, did the same thing across a few of the brain ICs and came up with a rough estimate that only about 10% of funded competing R01s got there unrevised. (The best analysis is by study section because it tends to get rid of the RFA issue which artificially enhances first-submission funding.)
More importantly, I became convinced that the revision process didn't do anything! Convinced that when you looked at the issues supposedly driving the critiques, the responses on the revision and adding a real-world consideration of the conduct of science.....that there would be no resulting change in the eventual conduct of the research. None of any significance anyway. Remember this was by seeing the eventual fate from original to A1 or A2 funding for a fair number of grants, albeit in one study section. Seeing the actual proposals, revisions and critiques. Hearing what was said in review.
We can talk endlessly about why this was happening, of course. I blame it partly on ignorance (when I started ranting raising the issue with people involved in review I got a lot of blank stares initially), partially on the inevitability of the budget doubling and subsequent flatline and partly by self-referential cultural factors that simply go un-examined.
I came to realize that a major act of the NIH was going to be required to fix things. I recommended at some point or other simply holding over some fraction of the grants from a prior Council round for funding in the next round to deal with the backlog effect. This would have only been a partial band-aid. But it recognized that there were a whole boatload of original submission grants that were just fine and were only being scored lower because of the holding pattern. All those perfectly fine prior proposals that were back in revised form as A1s and A2s. I was pushing a "fish or cut bait" line and asking why we'd want to put PIs through the waste of time of revising and resubmitting when we all knew they'd be very likely to get funded eventually anyway.
Then NIH limited the revisions to just the single A1. I was not a fan because I suspected that in short order we would just return to the A2's showing up as thinly disguised new submissions. Well, NHLBI seems to be getting serious. A new strategy has been published:

In order to fund meritorious science earlier, the NHLBI has adopted a new funding policy that substantially decreases the need for investigators and universities to submit revised applications. The Institute will develop separate lists ranking applications according to amendment status, resulting in equivalent success rates for both unamended (A0) and amended (A1/A2) applications. A more extensive discussion of this new funding policy can be found at the following site: "Early Notice: Rebalancing Success Rates in FY 2010."

The percentile rank for funding will be 16% for an original -01A0 but only 9% for A1s and 7% for any grandfathered A2s. Now why o why have they done this?
Following the quoted link we find:

The NHLBI has observed a marked decline in the funding success rate of original, unamended versions of new and competing renewal grant applications, both of which are denoted with the grant number suffix A0. The NHLBI has also observed a corresponding increase in the number of grant applications that are funded after being amended and resubmitted once (denoted A1) or twice (denoted A2). An analysis of applications from established investigators submitted between fiscal years 2004-2007 revealed that resubmissions typically benefit by receiving marked improvements in percentile scores. The analysis indicated that a minimum of 70 percent of A0 applications receiving percentile rankings of 25 or better were eventually funded, either in their original (A0) or amended (A1 or A2) forms. Data from the NHLBI are consistent with those of other Institutes.

Exactamundo! The only question is how to break the cycle. This is one way, just put everybody on the new-proposal footing and too bad if you are one of the NHLBI proposals that is caught in the middle of this process. Frankly, the ARRA/stimulus disturbance is not a bad place to put a major upheaval if you ask me.
__
*all you have to do is go to CRISP and search for new grants (e.g., "1R01%") in a given fiscal year and gate by IC, study section or some such.

23 responses so far

  • JAT says:

    I have grants funded on first submission and ones that needed tweaking (all from pretty reasonable assessments). Being an applicant and a reviewer, I see both logistics and flaws of the old system. At the end, I am still for the old system. OK,maybe it is time to retire!

  • qaz says:

    DM - there's a major problem with removing the A1/A2 revision cycle, which we can see by looking at the situation over at NSF. At NSF, although one can "revise a proposal that's not funded", each revision is treated as a new proposal. What this does is make each proposal submission a true lottery, and there are very good people who are truly stuck just below the cut who never get funded. The revision cycle at NIH helps people who are in that just-below-funding range cycle into the fundable range (it just takes longer). I'm not a fan of taking two years and a lot of revision to get funded, but only funding the top 10% every cycle is actually worse.
    What the fund-at-A2 cycle did was to put you in line, which allowed you to know that you (likely) had a grant coming and weren't just playing the lottery every time. Speaking from experience, this has been absolutely critical for myself and for several of my colleagues as the tenure clock rang. I've seen a revisable NIH proposal with a score just below funding get someone a reprieve (and even tenure on the belief that they would be funded eventually [all 5 cases I know were]) but an unfunded NSF proposal gets you nothing no matter how good the comments.
    Note: I'm not defending the A2 system. I'm just pointing out that it did have some advantages. I would love a better system, but it would have to address the fundamental problem (which is NOT that newbies are having trouble). [I believe the fundamental problem is that money is not shared fairly.] Maybe it would be enough to share the funding around better - make the first R01 easy to fund, the second harder, etc. If we made it possible for people to be funded at the 25% range (or even below that) if they didn't have any other grants, then maybe it wouldn't be a problem.

  • whimple says:

    So, the reward for the study section honestly coming up with ways to improve the science, and the applicant taking that advice to heart and writing a better grant is... they move the goalposts further back on you? You'd have to be an idiot to submit a revised grant to NHLBI now. They might as well just roll dice to figure out which applications to fund. Changing the system without warning in the middle of the game is also a nice touch on the part of NHLBI. Thank god I don't have anything pending there.

  • Maybe it would be enough to share the funding around better - make the first R01 easy to fund, the second harder, etc.

    What are you, some kind of fucking commie? Why do you hate America?

  • Ally Buff says:

    Yeah CPP,
    These bloody commies are aiming at diluting out the grandeur of America and her beautiful top-notch Versailles palaces. Versailles’s imaginative curators are working so hard to captivate people’s souls and offer consolation to their maladies. And they do it for free !!!. Just out of generosity and concern for their loved country. How patriotic!. Big artists are in distress at the thought of not being able to expand their magical powers on minds and bodies.
    These commies are just revolutionary idiots that only want petty Versailles far away from cutting-edge, and elegant Art of the Art.

  • microfool says:

    Changing the system without warning in the middle of the game is also a nice touch on the part of NHLBI.

    They announced it around a year ago, so we can hope no one already submitting with NHLBI was caught unaware.

  • qaz says:

    Hey CPP - I know you're just being cute, but I'm going to use your comment to raise an important point here. (And I hate how that line has become an easy way to shut off discussions, especially when it is used by people who agree with the original point but are just trying to be *#!$ing funny.)
    There's a key difference between communism (share and share alike) and what we really need to do here (harder and harder to get larger). It is absolutely necessary that one incentivize working harder (otherwise people slack off), but diversity increases the economy. For example, it's been shown that biodiversity increases both the stability and productivity in an ecosystem (e.g. wk by David Tilman). What we need is a progressively increasing difficulty where it's possible to grow to an empire size, but it takes increasingly more work at each stage.
    I would have liked to see NHLBI say if it's your only R01 or equivalent, then we're funding at 16%, if it's your second, we're funding at 9%, and for third or more, we're funding at 7%. Now, that might have actually helped.

  • Anonymous says:

    QAZ,
    I agree with you. And I will add this: there have been and there still are people working harder and harder and harder for little or nothing. Or even worse and that is: to have to go out of science. That is the tragedy !!!. This is simply because there is not enough determination to change policies and implement sounder ones like the one you are proposing and some other more.
    NIH needs to contemplate policies implemented by other agencies in the country if they feel real responsibility for the true mission of NIH.

  • whimple says:

    microfool: They announced it around a year ago, so we can hope no one already submitting with NHLBI was caught unaware.
    Fair enough. Watch the average age of NHLBI investigators sail upwards now.

  • JAT says:

    New or old system, none works perfectly to suit everyone. A major problem, not that its is not obvious, is there is simply too little money to go around. The competition goes beyond the ideological standard of "quality of science". I am not saying that if money is abundant, then all the issues disappear (I am for improvement as long as it does not favor a specific group only). I sometimes wonder how people would have felt if the funding level stayed around, let's say, 23-28% for the last few years (that is 28-33% for the newbies). Do you think the application/review systems would have changed so drastically to where we are now? Thoughts?

  • Principle Investigator says:

    There will never be enough money as long as we perpetuate a system of exponential growth in demand - one in which each lab head can "spawn" dozens if not hundreds of trainees before he or she retires.
    I really like the idea of making it easiest to get one's first grant application funded and progressively harder after that (although I would prefer the difficulty to depend on how much other support one has, not how long one has been getting grants). In general, I think that a larger number of smaller groups would be better for the players involved, allowing more scientists the freedom to do research in independent positions once they've invested some of the best years of their intellectual lives in ever-lengthening graduate and postdoctoral training. This might also encourage more diversity, with more opportunities for differing perspectives and expertise, than when everyone in a subfield gets trained by the same one or two individuals and competes on the same handful of projects.

  • DSKS says:

    "There will never be enough money as long as we perpetuate a system of exponential growth in demand - one in which each lab head can "spawn" dozens if not hundreds of trainees before he or she retires."
    I dunno. It seems to me the NIH shouldn't panic so much about trying to ram through the postdoc backlog into unsustainable PI positions (I fear such engineering is only going to bite us in the arse further down the line anyway when we're competing with each other for funds), and perhaps accept that academia is naturally moving towards - shock horror- the pyramid system that largely underlies supposedly meritocratic institutions in the private sector*. Big labs with lots of employees, small labs with not so many employees, grants handed out principally on merit, with minor tinkering here and there to ameliorate counterproductive biases toward newbies &c. I think this would be preferable to what seems to be a wide-ranging and herculean effort to force the entire infrastructure into a conformation that is no longer energetically favourable.
    It also seems to me that this focus on increasing the number of PIs is possibly distracting the establishment from what could be the next big problem faced by academic research in the US, which is an appropriate supply of postdocs. Right now, that supply is heavily subsidized by foreign grads, but that supply is not guaranteed in the future, particularly if India and China continue to pump resources into domestic research and become more aggressive in trying to keep their brains in their own back yard.
    As it goes now, a possible outcome 20 yrs down the line is that we'll have too many PIs competing for too little funds, and too few hands to actually do the work in the first place. All because of this intense effort to prevent a staff scientist class emerging in the sector of public research.
    On that note, this is an interesting paper on the subject.
    * well, generally meritocratic with the inevitable slice of cronyism mashed in, but at least that's a tangible flaw on which to focus corrective policy measures.

  • A Reader says:

    "As it goes now, a possible outcome 20 yrs down the line is that we'll have too many PIs competing for too little funds, and too few hands to actually do the work in the first place."
    This assumes that the current model -- A PI who is essentially a personnel and public relations manager supported by semi-autonomous skilled labor -- is the only possible model.
    This model (at least in the biological sciences), however, is a relatively recent phenomenon, driven by increases in funding and the rise of research biologist as a profession. It's a competitive model in today's struggle for recognition and grant money, but arguably a terribly inefficient use of scientific resources and scientists.
    As postdocs and grad students get more scarce, PIs will be forced (or have time) to go back to the bench themselves and do the work themselves. I think that's good for science. The representatives of the work will actually know what they're talking about and be directly responsible for the work. The trainees that work in these situations will no longer be factory slaves, but actually apprentices again.

  • qaz says:

    Why do people think that there's a shortage of grad students and postdocs? Is this a field-by-field thing? At our BigStateResearchU, we haven't seen any decrease in postdoc candidates, and (with the bad economy), we've seen a large increase in graduate student candidates.
    That being said, the "exponential growth" of one scientist training several dozen young-uns (over a lifetime) is not a problem as long as NIH gives up on the "the only training worth doing is replicating research scientists" concept. Let these people learn how science works, do their PhD (and postdoc if they want), doing good science along the way, and then go on to do other useful things.
    Note: I am NOT advocating a "broken pipeline". If we stop thinking "the best go to research, the rest are failures", then there wouldn't be a pipeline to break. I AM advocating graduate PhD-level science training for the rest of the population - imagine if our politicians actually understood science (or even critical thinking) for example. A lot of professions would be improved by having scientific training. (But they don't need it, you say. I say, why can't they have it? Why can't spending five years doing some good science not be a part of someone's path in life, even if they don't go on to do NIH-R01-Research?)

  • This model (at least in the biological sciences), however, is a relatively recent phenomenon, driven by increases in funding and the rise of research biologist as a profession.

    It is also driven by the dramatically increased interdisciplinarity of the biological sciences over the last several decades. It is simply impossible today for a single scientist to be competent at physically performing the wide array of technical approaches that contribute to the most interesting and groundbreaking studies. Being good at assembling and managing teams of scientists with disparate expertise is necessary to succeed as a PI doing this kind of science, and is not some kind of deficiency.

  • A Reader says:

    "It is simply impossible today for a single scientist to be competent at physically performing the wide array of technical approaches that contribute to the most interesting and groundbreaking studies. Being good at assembling and managing teams of scientists with disparate expertise is necessary to succeed as a PI doing this kind of science, and is not some kind of deficiency."
    I agree, but think that collaboration does not necessarily require a manager.
    "Why do people think that there's a shortage of grad students and postdocs? Is this a field-by-field thing? At our BigStateResearchU, we haven't seen any decrease in postdoc candidates, and (with the bad economy), we've seen a large increase in graduate student candidates."
    There is a shortage of good graduate students and postdocs. I similarly have not observed a shortage of applicants, but the quality seems to be going downward. This is consistent with a report I recently saw discussed in Nature or Science or something, showing that the number of top-tier (e.g. best) scientists that continued in science decreased during the late 1990s, while the number of lower-tier (e.g. marginally qualified) scientists that continued on in science continued to increase. It is easy to imagine that the best-and-brightest could (and did) find other better opportunities in the late 1990s, given the tech economy. That particular phenomenon may have ended. But the overall problem for U.S. Science still exists, because the best-and-brightest are increasingly lured out of the U.S. (or avoid it altogether). Germany plans to increase science funding 5% per year for the next few years. And this is nothing compared to the boom in China. China has increased science funding 20% per year starting in about 2000, and about 25% a year since 2005. Many of us know talented foreign born faculty who are being lured back home. And some of us know talented American-born faculty who are being lured overseas. I am actually in Europe now, funded by European money, being tempted. Fuck NIH. I am getting my work done better elsewhere.
    So what should the U.S. do? It can't increase the science budget without cutting something else. About the only places in the U.S. budget with enough money to make a difference to the science budget are social services and military expenditures.
    Cue political discussion...

  • Many of us know talented foreign born faculty who are being lured back home. And some of us know talented American-born faculty who are being lured overseas.

    This is absolutely true. I was recently in an industrialized foreign country, and the whole time I was there, pretty much everything I saw--food, housing, transportation, science, education, leisure, etc--was demonstrably better than in the United States.
    There is no doubt that we are a declining military oligarchy that has spent so much fucking money on war and enriching the oligarchs, that our entire society is going to pot. It is sad and hilarious how those oligarchs have convinced the rabble whose pockets they are draining dry that USA #1!!!, and that it would be terrible if we became more like Europe or Japan.
    What these fucking morons don't get is that Europe and Japan are much BETTER places to live than the US for the overwhelmingly vast majority of citizens, and if we don't wake the fuck up, instead we are going to end up like some motherfucking South American shithole like Colombia, where everyone other than the oligarchy lives in a fucking cardboard shack.

  • DrugMonkey says:

    top-tier, lower-tier? Admit it Reader, you just made that shit up, didn't you? How the hell is some national survey supposed to discern who are the good and bad trainees within each program? or do you mean that this is labeled by *program* reputation? In which case your assertion is totally false. On face.

  • A Reader says:

    You suck, DM. How dare you demand evidence? This is the internet!
    Here's a link to the report, which I haven't actually read:
    http://www.heldrich.rutgers.edu/uploadedFiles/Publications/STEM_Paper_Final.pdf
    Fig. 6 (page 26) is key.
    Science and Nature both had blurbs about it. Science had the longest blurb: Science 30 October 2009:Vol. 326. no. 5953, p. 654
    This link may work: http://www.sciencemag.org/cgi/content/summary/sci;326/5953/654-a?maxtoshow=&HITS=10&hits=10&RESULTFORMAT=&fulltext=Steady+as+She+Goes&searchid=1&FIRSTINDEX=0&resourcetype=HWCIT

  • DSKS says:

    Are there really firm correlations to be made between highschool highflyers (or even college level for that matter) and the eventual movers and shakers in any sector of the economy after more influential factors such as class, ethnicity and gender have been accounted for?
    It seems to me that identifying quality in postdocs would ideally mean addressing their productivity and impact, but of course it's difficult to correct for extraneous factors such as inadequate training and the reality that success, particularly on the cutting edge, can rest on a substantial degree of luck inre mother nature choosing to be accommodating or not to a certain technical advance, hypothesis or experimental design.
    Certainly there are good postdocs and crap postdocs, but I'm not sure that anything other than personal experience and letters of reference is going to make it easy to identify one from the other, let alone to say with any degree of clarity that the pool is getting dumber or brighter.

  • A Reader says:

    "Certainly there are good postdocs and crap postdocs, but I'm not sure that anything other than personal experience and letters of reference is going to make it easy to identify one from the other"
    Publications = Best predictor. People either produce or they do not.
    Definitely no one looks at your grades. The lab you trained in means a bit, but only insofar as people respect that lab. The institution doesn't mean diddly as long as potential employers know your lab. Letters don't mean much, surprisingly, because they're all fake blather. But a phone call -- which usually means something bad needs to be said that can't be written down -- is of course the end for you. But PIs only make The Phone Call to friends. Otherwise they are generally happy to finally be rid of crap people.
    If you are a grad student or postdoc, the implications of this should be clear: PUBLISH YOUR SHIT. Don't worry about your grades as long as you don't get kicked out of the program. Don't worry about sucking up too much for letters, as long as you can get decent letters, especially from important people. Sucking one Nobel penis is better than making a thousand others like you. And in any case, no one will care what people say about you if you publish. Future employers will know you're an ass, but they'll still hire you as long as it looks like you'll pad their CV.

  • A Reader says:

    Future employers will know you're an ass, but they'll still hire you as long as it looks like you'll pad their CV
    ...or bring in grants...

  • Svetlana says:

    It is not the case that prior notice was given by the NHLBI on the new revision policy:
    The original Post Date of the"rebalancing success" notice was 19 March, 2009.
    A1s and A2s considered at October 2009 counsel were submitted 5 March 2009.
    It is likely that had those submitting last chance A2s had known what was in store for them on Mar 5 they would have taken a different course of action... perhaps submitting their A2 as a new A0 app.

Leave a Reply