Grantsmack: Overambitious

Aug 25 2015 Published by under Grant Review, NIH, NIH Careerism, NIH funding

If we are entering a period of enthusiasm for "person, not project" style review of NIH grants, then it is time to retire the criticism of "the research plan is overambitious".

Updated:
There was a comment on the Twitters to the effect that this Stock Critique of "overambitious" is a lazy dismissal of an application. This can use some breakdown because to simply dismiss stock criticisms as "lazy" review will fail to address the real problem at hand.

First, it is always better to think of Stock Critique statements as shorthand rather than lazy.

Using the term "lazy" seems to imply that the applicant thinks that his or her grant application deserves a full and meticulous point-by-point review no matter if the reviewer is inclined to award it a clearly-triagable or a clearly-borderline or clearly-fundable score. Not so.

The primary job of the NIH Grant panel reviewer is most emphatically not to help the PI to funding nor to improve the science. The reviewer's job is to assist the Program staff of the I or C which has been assigned for potential funding decide whether or not to fund this particular application. Consequently if the reviewer is able to succinctly communicate the strengths and weaknesses of the application to the other reviewers, and eventually Program staff, this is efficiency, not laziness.

The applicant is not owed a meticulous review.

With this understood, we move on to my second point. The use of a Stock Criticism is an efficient communicative tool when the majority of the review panel agrees that the substance underlying this review consideration is valid. That is, that the notion of a grant application being overambitious is relevant and, most typically, a deficiency in the application. This is, to my understanding, a point of substantial agreement on NIH review panels.

Note: This is entirely orthogonal to whether or not "overambitious" is being applied fairly to a given application. So you need to be clear about what you see as the real problem at hand that needs to be addressed.

Is it the notion of over-ambition being any sort of demerit? Or is your complaint about the idea that your specific plan is in fact over-ambitious?

Or are you concerned that it is unfair if the exact same plan is considered "over-ambitious" for you and "amazingly comprehensive vertically ascending and exciting" when someone else's name is in the PI slot?

Relatedly, are you concerned that this Stock Critique is being applied unjustifiably to certain suspect classes of PI?

Personally, I think "over-ambitious" is a valid critique, given my pronounced affection for the NIH system as project-based, not person-based. In this I am less concerned about whether everything the applicant has been poured into this application will actually get done. I trust PIs (and more importantly, I trust the contingencies at work upon a PI) of any stage/age to do interesting science and publish some results. If you like all of it, and would give a favorable score to a subset that does not trigger the Stock Critique, who cares that only a subset will be accomplished*?

The concerning issue is that a reviewer cannot easily tell what is going to get done. And, circling back to the project-based idea, if you cannot determine what will be done as a subset of the overambitious plan, you can't really determine what the project is about. And in my experience, for any given application, there are going to usually be parts that really enthuse you as a reviewer and parts that leave you cold.

So what does that mean in terms of my review being influenced by these considerations? Well, I suppose the more a plan creates an impression of priority and choice points, the less concern I will have. If I am excited by the vast majority of the experiments, the less concern I will have-if only 50% of this is actually going to happen, odds are good if I am fired up about 90% of what has been described.

*Now, what about those grants where the whole thing needs to be accomplished or the entire point is lost? Yes, I recognize those exist. Human patient studies where you need to get enough subjects in all the groups to have any shot at any result would be one example. If you just can't collect and run that many subjects within the scope of time/$$ requested, well.....sorry. But these are only a small subset of the applications that trigger the "overambitious" criticism.

42 responses so far

  • Philapodia says:

    It'll never happen. This will be a key critique to fund person, not project.

    Review for small town grocer-type: "This project is over-ambitious"

    Review of same grant for BSD-type: "This vertically ascending, paradigm shifting proposal will generate new incredibly valuable new data that will move the field into warp drive. The glam-pub track-record of the applicant demonstrates that they are leaders in their field and will be able to make significant advancements of our understanding of the nature of the universe, cure all cancer, and grant immortality to the true scientists worthy of funding so they may continue to find Truth."

  • Ola says:

    With the way modular budgets are going, pretty much getting out of bed in the morning is beginning to seem overambitious!

  • SidVic says:

    Ola,
    Keep liquor on the bedstand, and take a shot first light . I've found it helps.

  • Noncoding Arenay says:

    In the 'Person, not Project' era: You unknown new PI person, how dare you submit a non-R21/Ro3 proposal?! Downright overambitious.

  • Pinko Punko says:

    If you aren't in over ambitious territory you are in "incremental" territory. Does this seem "no win" yes it does.

  • Pascale says:

    "If you aren't in over ambitious territory you are in "incremental" territory. Does this seem "no win" yes it does."

    All research is incremental. There's always a mechanism behind the mechanism behind the mechanism, like a set of infinite Russian nesting dolls...

  • Dave says:

    This vertically ascending, paradigm shifting proposal will generate new incredibly valuable new data that will move the field into warp drive. The glam-pub track-record of the applicant demonstrates that they are leaders in their field and will be able to make significant advancements of our understanding of the nature of the universe, cure all cancer, and grant immortality to the true scientists worthy of funding so they may continue to find Truth

    Have you been hacking DMs summary statements again?

  • Philapodia says:

    @Dave

    This was from my last application. It was triaged.

  • JustAGrad says:

    So where does this leave people who receive an overambitious critique from one reviewer and underambitious from another? Should we just redouble our attempts at getting the reviewers excited for the project?

  • Pinko Punko says:

    DM can you please indicate how an efficient review can be distinguished from a biased or disengaged review? Why should program be convinced by straight Stock Critique? How is it clear that program is getting best advice if no details are included. If "work" is not shown, critique may be lazy. Hard to know.

  • Philapodia says:

    "So where does this leave people who receive an overambitious critique from one reviewer and underambitious from another?"

    Submitting an A1 and hoping for new reviewers who think it's just right.

  • lurker says:

    The applicant is not owed a meticulous review. True, BUTT.......

    ....ASS paylines get tighter, aren't all reviewer's just resigned to pick a single grant or maybe two out of their pile of 10+ to fight for in the discussion phase? Knowing this, efficient time management would result in all my energy on the 1 or 2 grants I'd care about, and then go lazy with the stock critiques on the remaining 8+.

    The "overambitious" dismissal is an easy device to use as grant review becomes more and more circumspect. One dood's shorthand is another dood's laziness. After seeing exactly what was written on a funded grant from a BSD, I am absolutely convinced what is written/presented in a grant is no-where-near as important as how big your gonadal appendages are or whose BUTT you've brown-nosed. "Overambitious" for noobie-me is "a Paradigm Shift" for the BSD.

  • qaz says:

    I have seen overambitious as short-hand for three problems.

    1. There is not enough funding in this project to accomplish this goal. (An R21 with 8 aims that would take a dozen people ten years to complete - yes, I have seen such.)

    2. This proposal reaches beyond the technical skills of the PI at hand. This can be a criticism of young PIs who don't realize how hard some techniques are or senior PIs who think they can just hire it into their labs. (For example, an R01 that says "we'll spend a year getting this technique working / finding this reagent / breeding this mouse and then we'll have four years to do our work", when study section knows from experience that the first step is going to take them 2-3 years.)

    3. A proposed project that assumes a conclusion not in evidence. In particular, a project where they say "first we'll assume that X is true" then we'll explore the consequences of X, when study section is pretty sure that X will turn out to be a lot more complicated than they think.

    Usually, I think of "overambitious" as the first of the three, but I fail to see how any of these change under a "people not project" system. We're still not providing infinite money and we still have a timelines and we still have logic.

  • Noncoding Arenay says:

    "The concerning issue is that a reviewer cannot easily tell what is going to get done."

    And how does this tie into the "name" of the reviewer? Lesser known PIs probably get this comment more than the better known ones because said reviewer can tell what is going to get done in the latter case because they know the PI's capability/efficiency/track-record/what have you. While not a full-fledged 'person, not project' scenario, it does introduce some amount of it into the review outcome. Basically what Philapodia said in their first comment.

  • drugmonkey says:

    PPunk-and meticulous review can be stupid and wrong. how is program to know? The whole system depends on trusting the reviewers, not trying to game out afterwards which ones can be ignored and which drop pearls on every review.

    qaz- you are not grasping that people-not-projects review doesn't care if what is proposed is able to be accomplished with X resources over Y interval of time.

    Lesser known PIs probably get this comment more than the better known ones because said reviewer can tell what is going to get done in the latter case because they know the PI's capability/efficiency/track-record/what have you.

    yep. and this is to be assiduously opposed by those of us in the Coalition of the Decent.

  • Philapodia says:

    "yep. and this is to be assiduously opposed by those of us in the Coalition of the Decent."

    Does this coalition have a secret handshake or wear snazzy ball caps that say "Make the NIH Great Again!"?

  • Grumble says:

    "this is to be assiduously opposed by those of us in the Coalition of the Decent."

    No, it isn't. "Investigator" is a criterion that NIH wants reviewers to score. If you, as a reviewer, feel that the investigator is top notch (and believe that her science will continue to be top notch, based on a long history of being top notch), why shouldn't your opinion be reflected in your overall score?

  • Noncoding Arenay says:

    @Grumble - I don't think this is meant to separate the superstars from the commoners. I think it is meant to differentiate the bozos from the capable (wrt to said proposal).

  • Noncoding Arenay says:

    What I mean is a score of 1-3 for the investigator is not going to positively sway program with the same magnitude that a score of 4+ would in the opposite direction.

  • E rook says:

    "So where does this leave people who receive an overambitious critique from one reviewer and underambitious from another?"

    Advice given to me was to have a section in each aim (or sub aim, as necessary) explaining feasibility for the amount proposed and justification for not doing more work. To address the under-ambition, you can always say it goes back to budget & time, but scientifically you've prioritized for a reason and will get to the other stuff on future projects. Looking inward, this happened to me once, and it was a failure to communicate the scope well. Though I never got an R01, they were all "well-written and clearly explained."

  • Draino says:

    Sometimes it feels like public education is failing because the kids don't learn anything besides how to take tests. Now are we learning anything besides how to write grants?

  • Grumble says:

    @NCA: Right, but a reviewer can decide to weight the overall impact score such that Investigator is disproportionately represented. Program *does* care about overall impact.

    @Draino: Bingo. It's a tragic waste of time and talent. The time we spend in pursuit of the perfect grant is time not spent in pursuit of scientific progress.

  • Pinko Punko says:

    DM- some reviewers are better than others- if everyone limits to StockCritique where are their checks and balances? And how does system guard against biases- whether anti-newb or anti-diversity or anti-club? Critiques should be compelling. You don't have to write 5 pages to be compelling. You can be short/efficient and compelling, but likely you aren't entirely generic. Meticulous and wrong still gives the applicant something to leverage against in resubmission- it establishes dialog with other reviewers. Generic comments give nothing and this hurts the process.

  • Grumble says:

    "Generic comments give nothing and this hurts the process."

    IME, usually comments like "overambitious" are accompanied by some indication as to why the reviewer(s) felt that way, typically along the lines of one of qaz's 3 reasons. There is absolutely no benefit to the grant writer if the reviewers to go into detail because the only possible response to an "overambitious" critique is to propose fewer/less involved experiments.

  • Philapodia says:

    I think what a lot of us object to is the perception of moving goalposts with review. In regard to the investigator criterion it seems that if you're well known then you get to kick from the 20 whereas if you're a newer investigator you have to kick from the 50. There is a chance you can make the fieldgoal from the 50, but the odds are much better if you do it from the 20. A kick from the 50 is also much more ambitious than a kick from the 20 and you're less likely to think that the investigator can make it.

    Since we all spend a month or two of concerted effort to generate data and write these stupid grants, it seems like an inordinate amount of effort is wasted on what comes down to a crap shoot. As E-rook stated above: "Though I never got an R01, they were all "well-written and clearly explained."". This means that writing a clear and well-written grant isn't nearly enough, you have get the reviewer excited which is something you can't really control. Reviewing unsolicited grants against each other is comparing apples to oranges. If your reviewers happen to like apples they will get excited, whereas if they have a penchant for oranges they won't be as excited. The science and impact of the two grants may be similar, but the preference of the reviewer will win out to the detriment of the applicant and the NIH.

  • Noncoding Arenay says:

    "This means that writing a clear and well-written grant isn't nearly enough, you have get the reviewer excited which is something you can't really control."

    While I agree to an extent with the above, I think in today's funding climate this very statement underscores the need for younguns to go out there, get yourself on the podium at multiple conferences and invited talks and perform your way into the minds of as many fellow scientists as you can. While it won't give you control over their excitement, it will certainly increase their enthusiasm for your application and get you closer to that 20 yard line.

  • Philapodia says:

    "the need for younguns to go out there, get yourself on the podium at multiple conferences and invited talks and perform your way into the minds of as many fellow scientists as you can."

    Yes, younguns need to get out of their comfort zone and go get to know the people likely to be on your study section, although I think personal interactions are better. I serendipitously met a long-standing member of the study section I submit to at a recent meeting and starting chatting about science/politics/other BS. It turns out we have similar interests and get along quite well, and hopefully that interaction will help move the goalpost forward a bit.

  • jmz4 says:

    @philipodia
    Isn't that the "club" mentality everyone is railing against?

    Are there no benefits of writing grants? I think if we transitioned to a fully people not projects system, you'd waste a lot of money on poorly thought-out, overambitious, or redundant experiments. This has beeny experience in HHMI labs and labs with alternate support (eg foundation or institute).
    Lazy and stock critiques are the result of overburdened reviewers without enough money to spread around.
    But shouldnt it be the SRO's job to modulate the influence of these and give more creedence to the more well thought-out and explicated reviews? Maybe they should get some sort of modifier on the score before passing it on to the council to reflect this role?

  • Philapodia says:

    "Isn't that the "club" mentality everyone is railing against?"

    Yes, but in the real world I want to keep the lights on in my lab and feed my children, so I hustle just like I'm sure most of the rest of us do. We're in sales, not science, and relationships matter.

    "Are there no benefits of writing grants?"

    I think there is as they force you to think through projects, even though they likely won't turn out the way you think they will. What's wasteful is writing dozens of different projects that will never see the light of day.

  • Eli Rabett says:

    Eli has been in NSF panels where the PO said: "I need a better review to defend the non funding decision when the angry (one can sense this through the phone) PI and her University president call." And then they lock the doors until it is delivered.

  • L Kiswa says:

    What Eli said. The last NSF panel I was on the PO said this for essentially every proposal that was declined and made it clear we would not adjourn till the reviews were sufficiently substantive.

  • qaz says:

    DM says... "qaz- you are not grasping that people-not-projects review doesn't care if what is proposed is able to be accomplished with X resources over Y interval of time."

    So, wait, are you now supporting the sinecure arguments? That we should be funding labs at a reasonable level as long as they are continuing work? I thought you were deadset against this!

    If its a true people not projecct proposal, then there is no proposal. It's just "I've done this work in the past, give me money for me to do more work." Therefore overambitious never enters into the discussion. However, if its "I have this pedigree, here's where I'm thinking of going in the future" then there will be accusations of overambition.

  • drugmonkey says:

    I am describing, not supporting, qaz.

  • drugmonkey says:

    qaz- let me circle back around to your point 3. Why does it matter? If we want PI Smith to take a whack at Question M and all....why not extend her the rope to do so? Why play this game of fantasy money whereby she is supposed to magically develop the project to a point of greater feasibility before getting the real money?

    This feeds the Preliminary Data chase. Which is a bad thing for all of us.

  • qaz says:

    OK. I too will describe and not defend. 🙂

    If PI Smith says xe is going to work on Question M, then people are going to want to know what direction they are working at, whether that direction is technologically possible at all, and whether that direction is feasible before PI Smith dies of old age (or the funding runs out or PI Smith loses their job or the world ends or whatever).

    Your argument will be (correctly) that this is really a project not people assessment hiding within the people not project paradigm. This is why people not projects only works looking to the past not the future, which makes it hard to get in. (Which is why typical people-not-project systems are project proposals to get in [1st newbie R01] and then progress-reports not proposals to continue [renewal].) To be fair, this is very much how the old system worked, in the "Golden Age of NIH", back when we were spending 12% GDP on science and there was enough money to do this....

    More importantly, in relation to the overambitious stock-critique at NIH, solving "overambitious" does not require project development to the point of feasibility, but rather project proposal to the point of feasibility. (This is the source of the mistake that people think you need to have the project complete before proposing it. In my observations at the study sections I've gone to, you need to prove that the project is doable [for example, that you have the techniques working in your laboratory], which is a very different thing.)

  • drugmonkey says:

    I tend to agree applicants overworry about progress versus feasibility wrt Prelim Data but then I got a summary statement that tends to support the paranoid meme

  • Juan Lopez says:

    Monkey, I am one who thinks that reviewers expect preliminary data which supports the hypothesis, not just that we can do the project.

    Yesterday I was chatting with a BSD and telling him about our recently submitted application. He said: cool, any preliminary data that the hypothesis holds? It's a cool hypothesis, and I have no doubt that you can do the project. But without evidence that it will actually happen, I doubt you will get funded.

    My response: but there is so much circumstantial evidence (it is true in several similar systems), and the hypothesis so clean... That, even if it is not true, it would be a valuable thing to know.

    He smiled.

    He knows that this is a study we can do, no doubt about it, and agrees that the potential results are exciting. But he made it clear that without the evidence, we will likely not get funded.

  • Grumble says:

    Let's put it this way. 10-20% of grants get funded. As a reviewer, you want to fund the "best science" that is most likely to have the greatest impact. If one grant has evidence that the hypothesis is likely correct, and another one only has evidence that the experimenters can get the techniques to work, which one would you pick as the surer bet?

    Doing the science first, then submitting the grant is becoming more common because it works, and that is due to the same reason that everything else is fucked up in grantlandia: not enough money.

  • drugmonkey says:

    I pick the one with the most interesting and relevant hypothesis under investigation.

  • drugmonkey says:

    Question for Grumble: should we not actively fight the bad things that have emerged as cultural imperatives for NIH grant review? Have we no responsibility to fix things ourselves before NIHdom does something harmful in a clumsy attempt to solve perceived problems?

  • Grumble says:

    Determining which of several different hypotheses is the "most interesting and relevant" is extremely difficult (at least for me) because most of the hypotheses I read about in grants actually seem really cool and worth testing. It's the minutia that differentiate one grant from another. Because there is not enough money for all good grants to be funded, the minutia are the basis for the decision. That is why such things as preliminary data take on such enormous importance, when really they should hardly matter at all.

    So what is it, exactly, that you are asking us to fix? The system is not broken because of our behavior, which is nothing more than a rational response to the grant economy as it is. It's broken because there is not enough money in that economy for the number of participants in it.

  • […] consider captive breeding program for enigmatic little walking fish Grantsmack: Overambitious In the US, computer science is (unfortunately) a privilege The growing global battle against […]

Leave a Reply