Is a prior interval without funding irrelevant to grant review?

Jan 03 2011 Published by under Grant Review, NIH Careerism

A comment from williams on a prior post intrigues me. S/he asserts:

My opinion is that reviewers should assess the scientific merit of a grant in itself (impact/significance/ PI’s scientific potential). Period. If an investigator has had an interval of no support that is irrelevant to the science she/he is proposing to do at this time in point.

In theory, I would agree. In theory.

The trouble is that this sort of assumes "all else equal". And I just don't see anything other than vanishingly rare circumstances meeting this standard.

Now to sidetrack for a little bit, the discussion was mired down by the term "failed grant" which to me means an interval of funding that has resulted in some less-than-productive outcome. arzey had said

You can almost always learn more about how to do an experiment from a failed experiment than from a successful one. Or to paraphrase: all happy grants/PI’s are alike, every unhappy grant/PI has lots to teach us.

The commenter williams, however, seems to be obsessed with failed grant applications, i.e., those which do not win funding. A different matter, in my view and somewhat less interesting. Of course we should learn from our applications that fell short. With the comment I started with, we perhaps reach a happy middle ground.

The Investigator component of grant review looms large. Despite the fact that formally speaking the NIH award is to a University and that PIs and other staff could be swapped around willy-nilly in theory, the participating investigators are a huge factor. A large part of the assessment of Investigators is the publication record of the participating staff. Not just mere publications, either, but oftentimes an assessment of how publications fit in with the prior and ongoing research support. If you have relatively little research support, well then reviewers are going to extend more latitude for a publication record that is less than expected. If you have a huge amount of funding, these expectations can rise.

Now let us acknowledge that williams is correct that grant review is supposed to focus on the qualities of the current plan. A good grant score on a new application (i.e., not a continuation of an existing project) is not supposed to be an award for doing well in a prior interval of funding. It is supposed to be an informed prediction that the current proposal will be successful if funded. Likewise, an excellent grant application should not be penalized for apparent deficits in the past work of the participating investigators under a different award.

Herein lies the trouble. Most people assume that past performance of an investigator is a good predictor of future performance, leaving the specifics of a given scientific plan aside. So there is an overwhelming tendency to view a wildly successful prior interval of funding (aka, "track record") as a good predictor that the current application will result in similar success.....just because of the PI. Similarly, there is a tendency to view a lack of success in a prior interval as a prediction that the current application will not go well.

Favorably disposed reviewers will be looking for some reason to excuse what looks like dismal past performance, I hasten to reassure you. In fact, I have recommended before that if there is some apparent deficit in your own track record, you do what you can to provide an explanation in the application. Subtly. The advocate reviewer needs something to work with. I have personally seen all sorts of explanations, not excuses, explanations in grants. They can go down pretty well. Everything from trailing spouse issues to health conditions to child bearing to local weather disaster to local institutional screwage. If the reviewers like your proposal, they are looking to come up with reasons why your past interval of suboptimal performance does not hold as a predictor of your future performance leading the project.

All this does, however, is confirm the validity of the more general assumption that past performance predicts future performance. Right?

So how would we view an interval without funding? Here we are talking a mid to late career investigator who has had prior support, say, and then has gone for some serious interval (say 2+ years) without a grant. Should the Investigator criterion take a hit? Is this any different from going too long without any scientific publication? Does it, at some level, say something about an investigator if s/he is unable to keep the lab funded*?

I think it has to be a consideration. It is as much a part of the assessment of Investigators as any other traits and credits, in my view.
__
* to be clear I am talking relative to expectation. There are going to be some job categories for which continual funding is not an expectation. And some for which it is something more than an expectation. Let us not get distracted yelling about subfield differences and job types.

21 responses so far

  • proflikesubstance says:

    Track record related to the most recent previous grant was specifically discussed for borderline candidates i the NSF panel I was on in the fall and those with perceived deficiencies in publication or prior grant awards were dinged as a result. I don't know whether this is a general thing or specific to the panel I was on.

  • physioprof says:

    If you only have a single R01 and you fail to keep it continuously funded, I consider that to be a sign of poor judgment.

  • Caro Monkey says:

    Yes. Good judgment is a highly desirable quality for any individual or committee. Always.

    In this particular scenario, after having assessed the scientific merit of a grant for itself (naked value), an enhancement/correction factor could be applied by looking at the past impact on the field of a specific PI/institution. And that should apply to any and all grants/projects.

    The concept and reality of “Failed grant” ought to be analyzed and evaluated in all versions. Because there might be “marriages” (investigator/institution) extremely successful in acquiring and maintaining “scientific dollar support” over a long period of time. And yet they might have failed in having had a true impact in advancing science and better scientific and health practices. Past Impact should not be an absolute determinant but an important modifying factor for predicting returns on a present grant(s).

    Of course, we are moving ourselves in a context that is not and will never be totally objective. Coming up with a rational, balanced set of criteria to correct/enhance the “to be funded projects” would help reviewers and grant committees avoiding potential biases and maintain the spirit and practice of peer review: “that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences --

  • "If you only have a single R01 and you fail to keep it continuously funded, I consider that to be a sign of poor judgment."

    Are R01s the only workhorse funding sources in the world of basic biomedical science? I am assuming you would not make the same statement of someone who acquires, say, only 1/3 of their funding from the NIH.

  • Odyssey says:

    R01's are to primary workhorses for biomedical research in the US. Yes, there are many other sources of funding, but the NIH is the 800 pound gorilla and R01's are its main source of bananas.

    In some ways what CPP said is right in all fields. To rephrase, if you only have one major grant and fail to keep it... And I suspect the poor judgement he's referring to is not so much the failure to renew a grant - under the current climate that can happen to the best of scientists - but more to having just the one major grant. As some people in my department have discovered in the last year or so, trying to maintain a research program with just one grant is a very, very risky enterprise with dire consequences should you not renew.

  • DrugMonkey says:

    Caro Monkey,
    I once heard a Big Pharma retiree claim only half of people in that industry ever worked on something that ended up as a medication. Your claim that there are long term funded scientists who never have "true impact" seems a bit naive to me. Great ideas that don't happen to work out are legion. Is it a failed research program just because it was wrong in the end? Or less useful than hoped? Or got scooped by something better? Impact on research trends that only lead to therapies 30 years later are hard to assess in real time.

  • Caro Monkey says:

    DM,

    The concept of "true impact" based on half of biomedical projects ending up in truly useful MEDICATIONS is not only a fallacy but, in my view, a wrong expectation. Public health and biomedical research are much more than finding medications.

    Public health and biomedical research are more about ways of thinking, exploring and educating to endlessly find better alternatives to minimize disease and eliminate ignorance and poverty. It is also about tangible specific aims that expand the structure, function and perception of life, at all levels.

    A much harder ‘Research System” to look for if there is not an intentional pursue of creating as many opportunities as possible. Opportunities that run on creativity and diversity tracks and with a demonstrated legacy of passing the torch to younger generations.

  • Because I know you don't mind people chipping in occasionally with a "this is how my country does it":

    The CIHR (Canadian equivalent of the NIH) and some of our other funding agencies have CV formats that include a dedicated "Interruptions and Delays" section. Here's the official wording:

    "Identify any administrative responsibilities, family or health reasons, or any other factors that might have delayed or interrupted any of the following: academia, career, scientific research, other research, dissemination of results, training, etc. Common examples of an interruption/delay might be a bereavement period following the death of a loved one, maternity/parental leave, or relocation of your research environment. Limit the list to one page. Descriptions might include the start and end dates, the impact areas, and the reason(s) or a brief explanation of the absence."

    None of the PIs I work with have felt the need to use this section, but it's great to have the option!

  • Exiled American says:

    Cath@vwxynot?,

    Thank you for chipping in with “how your country does it”. Canada is known and recognized for sound policies affecting all aspects of life.

  • drugmonkey says:

    CM- I was making this analogy to make it clearer in a situation where the deliverables (medications approved by some regulatory body) are undeniably obvious. You said

    Because there might be “marriages” (investigator/institution) extremely successful in acquiring and maintaining “scientific dollar support” over a long period of time. And yet they might have failed in having had a true impact in advancing science and better scientific and health practices. Past Impact should not be an absolute determinant but an important modifying factor for predicting returns on a present grant(s).

    This suggests that you can make some claim to assessing what has "advanced science" or had a "true impact" within whatever long period of time you are referencing here. I say it is very hard to make these calls even after a few years or decades have elapsed. So how on earth can this sort of analysis contribute to concurrent assessment of a grant application?

    I know you don’t mind people chipping in occasionally with a “this is how my country does it”:

    That is such a Canadian way to put it. I LOVE people chipping in with examples from other funding agencies! And I am instantly a big fan of a default section for "Interruptions and Delays". This is frikken AWESOME to include as an expectation. I am beside myself.

    Canada is known and recognized for sound policies affecting all aspects of life.

    HAHAHAHAHAHAHA. Except curling, I feel certain you meant to say.

  • Physician Scientist says:

    Past impact, publications, etc. should definitely factor into a funding decision. If, without a good reason, somebody was so not motivated as to let their funding lapse, why wouldn't they simply take the money, sit around for another five years and then when their dean gets on them submit another grant application.

    Quite frankly, as a youngish prof., I have a grant application in almost every cycle. I work very hard to secure the data to do this, and I expect my colleagues to do the same. If I have a lapse in funding, its my own damn fault and I should be penalized. I'm sitting on my third study section this cycle, and you can be sure I'm expecting some progress when looking at renewals or submissions with a lapse in funding.

  • physioprof says:

    Quite frankly, as a youngish prof., I have a grant application in almost every cycle.

    This is extremely wise. The rule of thumb I have followed throughout my career and that I recommend to junior faculty I mentor is that one should plan to submit on average about two competing (i.e., new or competing renewal) R01 applications per year. (This, of course, includes resubmissions.)

  • drugmonkey says:

    What I recommend to not-so-junior faculty is "Don't get complacent! Submit some moar apps!"

  • anon says:

    "you can be sure I’m expecting some progress when looking at renewals or submissions with a lapse in funding."

    physician scientist - how do expect progress with no fucking funding? I wrote and submitted ~40 grant applications (not all to the NIH, of course), ran out of start-up money (which was modest), published a couple of papers anyway, had mountains of preliminary data, and the study section still said that my progress was inadequate. I don't understand what's being used as a measuring stick or what qualifies as 'some progress'.

  • Degenerate Physician says:

    Physician scientist,

    I bet you never overwhelmingly proposed in any of your grants to find effective therapeutics for the devastating illness of your interest. That’s your problem and your based-evidence lack of progress. There are investigators who, for the past 15 years, have “advanced science” and have had “true impact” by investigating the delivery of effective drugs for the devastating illness of the century. And their advancement has been highly estimated by their Review Committees. No interruption or major delays in their funding. But yes, they don’t stop publishing their “advances” (whatever advance might mean). Nor do they stop sending proposals by the month. There is assessment “even after a few years or decades have elapsed”.

  • Degenerate Physician says:

    PD. Yet, they hardly asked themselves about potential mechanisms to explain the disease. Neither 15 years ago, nor now. What was important was to get the money and as much as possible

  • drugmonkey says:

    anon,

    I do hope you were trying more than one study section. It is possible that you ran into one that was really bad (that's relative) about this prelim data stuff. Or they just really didn't get your proposals.

  • anon says:

    dm, thank you for the comment. I did try more than one study section, and eventually ended up getting funded by the NSF instead. Both study sections I tried have since been disbanded. They both sucked, and yes, likely did not get my proposals. Regardless, I felt the "inadequate progress" comments were bs. Anyway, I'm going to hold my nose and keep trying at the NIH.

  • drugmonkey says:

    Anyway, I’m going to hold my nose and keep trying at the NIH.

    That is a very wise decision. If you were able to be funded by the NSF, and are remotely health-related, you should be able to eventually get NIH to see things your way. It may take some effort to get there, but it can be done. obviously.

  • Physician Scientist says:

    anon - sounds like progress to me. I would be looking for "honestly trying" v. complacency, and the discussion centers on lapse in funding rather than trying to initial funding.

    Degenerate Physician - if you only knew about the translation science we're trying to do...its very difficult to do translational work when you have hospitals without true electronic medical records, hospitals that work their physicians to death and give them little time for intellectual academic curiosity (hence difficult to collaborate with), regulations that make genotype:clinical phenotype:disease course correlations very difficult, etc. Its very difficult, but we try. and yes, this does take money.

  • [...] I have a new favorite plan to help with this little problem, thanks to Cath of VWXYNot who commented: The CIHR (Canadian equivalent of the NIH) and some of our other funding agencies have CV formats [...]

Leave a Reply