The R01 still doesn't pay for itself and reviewers are getting worse

Jul 11 2016 Published by under NIH, NIH Careerism, NIH funding

I pointed out some time ago that the full modular R01 grant from the NIH doesn't actually pay for itself.

In the sense that there is a certain expectation of productivity, progress, etc on the part of study sections and Program that requires more contribution than can be afforded (especially when you put it in terms of 40 hr work weeks) within the budget. Trainees on individual fellowships or training grants, undergrads working for free or work study discount, cross pollination with other grants in the lab (which often leads to whinging like your comment), pilot awards for small bits, faculty hard money time...all of these sources of extra effort are frequently poured into a one-R01 project. I think they are, in essence, necessary.

I had some additional thoughts on this recently.

It's getting worse.

Look, it has always been the case that reviewers want to see more in a grant proposal. More controls, usually. Extra groups to really nail down the full breadth of...whatever it is that you are studying. This really cool other line of converging evidence... anything is possible.

All I can reflect is my own experience in getting my proposals reviewed and in reviewing proposals that are somewhat in the same subfields.

What I see is a continuing spiral of both PI offerings and of reviewer demands.

It's inevitable, really. If you see a proposal chock full of nuts that maybe doesn't quite get over the line of funding because of whatever reason, how can you give a fundable score to a very awesome and tight proposal that is more limited?

Conversely, in the effort to put your best foot forward you, as applicant, are increasingly motivated to throw every possible tool at your disposal into the proposal, hoping to wow the reviewers into submission.

I have reviewed multiple proposals recently that cannot be done. Literally. They cannot be accomplished for the price of the budget proposed. Nobody blinks an eye about this. They might talk about "feasibility" in the sense of scientific outcomes or preliminary data or, occasionally, some perceived deficit of the investigators/environment. But I have not heard a reviewer say "nice but there is no way this can be accomplished for $250K direct". Years ago people used to crab about "overambitious" proposals but I can't say I've heard that in forever. In this day and age of tight NIH paylines, the promises of doing it all in one R01 full-modular 5 year interval are escalating.

These grants set a tone, btw. I'm here to tell you that I've seen subfield related proposals that do seem feasible, money-wise, get nailed because they are too limited in scope. In some cases there is enough study-section continuity involved for me to be certain that this is due to reviewer contamination from the aforementioned chock-full-o-nuts impossible proposals. Yes, some of this is due to SABV but not all of it. It ranges from "why you no include more co-investigators?" (a subtle spread-the-wealth knock on big labs? maybe) to "You really need to add X, Y and Z to be convincing" (mkay but... $250K dude) to "waaah, I just want to see more" (even though they don't really have a reason to list).

Maybe this is just me being stuck in the rut I was trained in. In my formative years, grant review seemed to expect you would propose a set of studies that you could actually accomplish within the time frame and budget proposed. I seem to remember study section members curbing each other with "Dude, the PI can't fit all that stuff into one proposal, back off.". I used to see revisions get improved scores when the PI stripped a bloated proposal down to a minimalist streamlined version.

Maybe we are just experiencing a meaningless sea change in grant review to where we propose the sky and nobody cares on competing renewal if we managed to accomplish all of that stuff.

37 responses so far

  • MorganPhD says:

    My experiences with grants (in my PD lab) is that you only propose experiments/ideas that are 90% finished. Preferably, propose experimental directions whose data are already in papers that are in the review process. That way, your hypothesis looks rock solid because you already know the answer. Then you get 3-5 papers out in the first year of your grant cycle because they were basically already finished. Then spend years 2-4 generating cool new data, and repeat...

  • JL says:

    I remember as a postdoc reading the proposal that paid for me. It would have taken five people ten years to do the work (in fact, more than a decade later, that lab keeps working on it, and getting funded, and they are not close to getting it done.) I thought that it was crazy that reviewers bought it. Now I know that this is the norm in their field, and that there's no chance they could get funded if they propose a realistic amount of work.

  • dr24hours says:

    I can't help but think that with funding lines as tight, and scores as variable as they are, there can be no meaningful analysis of grantsmanship. Everyone has to be good. Once you're in the group that's good, it's luck.

  • Dr Becca says:

    This is exactly my experience. One recent grant went back to the same study section 4 times and each time they asked for more and more controls, even though at that point the number of experimental groups would have made proper statistical analysis near impossible, let alone the financial strain on having to buy all those animals.

    As 24h says, once you get in the "good" zone, it's all luck of the reviewer assignment draw.

  • qaz says:

    I see and hear "overambitious" all the time in the study sections I review at. It's never about money though. (As was noted in the NIGMS discussion, one can always recommend increasing the budget.) But it is often about time. If an experiment is not feasible in the proposed time, I'll often see proposals get dinged. Especially if there's an aim to cut.

  • Ola says:

    YMMV. I've heard "over ambitious" and "can't be done modular" several times in the last year, and as recently as last month. You need to diversify your study section portfolio DM.

  • drugmonkey says:

    You need to diversify your study section portfolio DM.

    I'm pretty sure you have been around this blog long enough to realize the comments are my diversification strategy.

  • Pinko Punko says:

    I tend to see default "budget is too high" comments when actually realistic budgets are proposed. And with some institutes that automatically slash the budget, this seems...myopic.

  • drugmonkey says:

    I tend to see default "budget is too high" comments when actually realistic budgets are proposed.

    And when the SRO insists that they identify a place where it needs to be cut the response is....?

  • PepProf says:

    Have to disagree. Perhaps this really is very study section specific. In the study section I am funded from, there are often calls to strip down proposals. My R01 resubmission was actually funded after being asked to remove various subaims. Another SS that I sit on frequently brings up the overambitious charge. In fact, for 2 application, the panel actually recommended a budget increase. In my experience, there is a real premium put on proposals that are focused, succint, and not larded up with unacheivable experiments.

  • drugmonkey says:

    This used to be the case for my favorite study sections.

  • SidVic says:

    The system is turning us into hornblowers and bullshit artists.

  • drugmonkey says:

    I think you are correct.

  • potnia theron says:

    Actually, my experience at the last study section of which I was part (http://wp.me/p5iOh1-Fm) there was an effort to address exactly this point. People asked "is this a realistic amount of work"? and when people said this needs "X,Y,Z" experiments, someone (often) said: "Yes, but what are we asking for?"

    My memory is that the specific criticisms tended to be about the particular measures being used, and the utility of this set of experiments. Or that they would measure something (in a live animal) that wouldn't be possible to do.

    These weren't R01s, again, but perhaps the most vulnerable (young investigators).

  • lurker says:

    With this situation, how many of us will throw in the towel on our academic labs? How much more deck chair reshuffling before all of us small-time grocers shutter our labs and go work for big pharma or biotech or patent law?

    How are you, DM, able to survive and keep the lights on? Will there be a day that I wish won't happen when you post on your blog that this grant chase is too much for you? That you will say "NO MAS!", and you'll stop giving us your sagely grant/career advice on your blog? Where will I go to procrastinate from writing this grant resub?

  • Grumble says:

    "The system is turning us into hornblowers and bullshit artists."

    That is PRECISELY what is happening, and what has been happening for years.

    Ironic, isn't it? Scientists are the one group of people for whom the truth MATTERS, more so than for any other group of people. And here we are, forced to make shit up on a regular basis.

    That is why I feel very strongly that The System (of project-based NIH funding) needs to be radically changed.

  • Peprof says:

    I have found that the more biomedical/translational leaning study sections are the most guilty of this crap. Most of my applications go to a very basic science SS that remains on what I would call the periphery of NIH's mission. The downside is that securing a grant seems to be even more challenging (vanishing pie slice). The upside is a resistance to bloated research plans and appeals for more and more useless subaims. Plus, we don't have the obligatory tithing to bs MD collaborators in an effort to 'legitimize' the relevance of our science. In my estimation, along with inflation of the administrative ranks as Universities, the MD-supporting clinical science-industrial complex is the single biggest problem with the NIH funding landscape.

  • David says:

    As someone who has provided grants outside of the NIH/NSF world, feasibility in terms of time and money are a huge factor in our reviews. When my group is giving some lab money, we need an answer to a specific question. We regularly ding proposals that we know can't be completed (in part because we are unlikely to get additional funds for that project).

    I wonder if the lack of engagement from the NIH process (i.e. no one making the decisions is committed to the success of the project). Where I work, if the grant fails, I fail. I don't think that is a consideration in the NIH/NSF world. [by fail, I mean doesn't produce something useful; it's ok if the technical hypothesis didn't yield what we hoped]

  • shrew says:

    My real worry is, suppose we got everything we always wished for, return to pre-doubling NIH funding trends, steady inflation-locked increases in R01 spending power...

    This tone in reviews will have already been set. Once the genie is out of the bottle in terms of ridiculous demands, who can put it back? The people who come of age scientifically right now will just learn that "this is how it's done" and we will treat it like an inevitability that you promise way more than you can deliver, just like insane reviews demanding more work than is germane to the study at hand are normal for publications.

    All of this crazy financial competition in science is driving all of these bad behaviors, but I am concerned that if we could return the financial competition to normal tomorrow, it would be too late to make things sane again.

  • Brain says:

    Perhaps promising the moon stems in part from the decline in renewals? My last time on a study section almost all applications were new submissions. If you don't have to write a progress report (for your renewal), where's the stick to prevent you from overpromising? Plus I always revise my aims when my budget gets cut.

  • Dave says:

    This blog is fucking depressing. Off to top myself.....

  • dsks says:

    This blog is fucking depressing. Off to top myself.....

    Can I have his Gilsons when he's dead?

  • Curiosity says:

    At least in my neck of the woods, the revolution in techniques -- optogenetics x specificity, whole brain electrophysiolgical monitoring, full EM-level circuit reconstruction, massive ethograms, etc -- has contributed to a kind of frenzy in proposals. I was floored in a recent study section at how a proposal that promised the sun AND stars in 5 years, was not only not dinged for lack of feasibility (which seemed obvious to lowly me), but was exalted above all because YES! these are the experiments that must be done! My sober kill joy attitude was treated as jealousy. (Don't you just WISH you could pull this off, scum.) There is a new grantsmanship art to be mastered that includes grandiose vision and *appearance* of feasibility. Where reality lies is anyone's guess; where renewals of this phase end up will be fascinating to watch. And yes, friends, modular.

  • cookingwithsolvents says:

    In my field of physical sciences you'd best propose something doable for the $/time budgeted. At least in DOE/DOD/NASA etc. funding land. It reflects pretty badly on you if you miss your funding spend rate (over OR under). Now, in NSF land the funding rates are so low I have no clue and I don't have current NSF funding so I can't comment.

  • AcademicLurker says:

    I'm currently a co-investigator on several R01s that are basically what program projects used to be, back in the day.

  • Grumble says:

    "My sober kill joy attitude was treated as jealousy. (Don't you just WISH you could pull this off, scum.)"

    Really? They called you scum? Or said "you're jealous"?

    You should have held your ground and loudly insisted on voting outside the range because the experiments are simply not feasible in 5 years. This often has a "monkey see, monkey do" effect - once one person does it, other join. Kind of like driving on the shoulder to get around a traffic jam. It can be half an hour before someone tries it, then suddenly 50 people are doing it.

  • Emaderton3 says:

    @ PeProf

    My grants go to the biomedical/translational type study sections. I have always written my grants based on what I think I can reasonably get done, and I routinely get slammed for it. My mentors keep telling me not to be "realistic" in what I put in there but to beef it up more. Very sad.

  • drugmonkey says:

    Word Grumble. All of this and more.

  • frustrated scientist says:

    I have heard that my approaches were oversold (as the only negative in a review) - guess what, if I don't do that where do I end? Exactly (BTW I don't think I exaggerated). And I have had aims in my grants that were clearly described as preparation and foundations for future clinical studies - and I got criticized that nobody would know if this would work in patients and the major flaw was that we didn't do this in humans - seriously, people?

  • drugmonkey says:

    I'm currently a co-investigator on several R01s that are basically what program projects used to be, back in the day.

    I just saw one of these! You are totally right. slamming together multiple labs with little bits to triangulate, just like the ole P01. Only for less money, I think.

  • Curiosity says:

    "You should have held your ground and loudly insisted on voting outside the range because the experiments are simply not feasible in 5 years. "

    I agree on so many levels. Yet I think this particular instance was telling: while I would be AMAZED if half the grant were accomplished, the fact remains that I do not know for sure that it can't be done. There was consensus among the haves that sure, it was ambitious, but hey, what the hell. It was the have-not saying it couldn't be done. In this case, I folded, in part to acknowledge their consensus; in part because it was true that I didn't know FOR SURE and it was cool science. The disparity in resources can amplify the confidence game that is part of all of this.

    I wonder if in some of these cases whether unfunded BRAIN initiative grants are just being recycled into R01 standard RFAs explaining the weirding -- program project stuff.

  • Yizmo Gizmo says:

    "The system is turning us into hornblowers and bullshit artists."

    Hey, whadda yoo got against hornblowers and bullshit artists?

  • qaz says:

    DM - We were recently told by a program officer that P01's were being phased out by the institute in question and were told that we should submit a multi-PI $499,999 R01 instead. I think this is part of the "multi-PI" push that NIH has been creating. In some institutes at least, this shift from P01 to large-R01 is explicit policy.

  • drugmonkey says:

    I suppose it doesn't matter what it is called....

  • jmz4 says:

    @qaz
    Yeah, the line I hear from PIs at my iLAF is to go after big (1.5m) multi-PI RO1s targeted at the special RFAs.

  • dsks says:

    "And I have had aims in my grants that were clearly described as preparation and foundations for future clinical studies - and I got criticized that nobody would know if this would work in patients and the major flaw was that we didn't do this in humans - seriously, people?"
    I got the same response in the reviews from a special emphasis AREA R15 review panel at NCI. I was a bit confused about it, given that I assumed the eligibility criteria for that mechanism would put it out of bounds for med schools and whatnot, until I did a little Reporter PI stalking and saw the kindsa folk I was competing with. Then I laughed. And wept.

  • jmz4 says:

    http://www.cell.com/cell-stem-cell/fulltext/S1934-5909(16)30162-X

    Do we have thoughts on this paper published by Sally Rockey? It seems awfully defensive and I'm not sure how they conclude that NIH funding policies play no role "mouths at the trough" problem.

Leave a Reply