Your Grant in Review: Competing Continuation, aka Renewal, Apps

Jan 28 2016 Published by under Grant Review, NIH, NIH Careerism

In the NIH extramural grant funding world the maximum duration for a project is 5 years. It is possible at the end of a 5 year interval of support to apply to continue that project for another interval. The application for the next interval is competitively reviewed alongside of new project proposals in the relevant study sections, in general.

Comradde PhysioProffe addressed the continuation application at his Ftb joint. NIAID has a FAQ page.

The NIH Success Rate data shows that RPG success rates were 16.8% in 2013 and 18.1% in 2014. Comparable rates for competing continuation RPG applications were 35% in 2013 and 39% in 2014. So you can see why this is important.

I visited these themes before in a prior post. I think I covered most of the issues but in a slightly different way.

Today I want to try to get you folks to talk about prescriptives. How should a competing continuation / renewal NIH grant application be reviewed?

Now in my experience, the continuation application hinges on past-productivity in a way that a new application does not. Reviewers are explicitly considering the work that has been conducted under the support of the prior award. The application is supposed to include a list of publications that have resulted from the prior award. The application is supposed to detail a Progress Report that overviews what has been accomplished. So today I will be focusing on review mostly as it pertains to productivity. For reference, Berg's old post on the number of papers per grant dollar is here and shows an average output of 6 papers (IQR about 4-11) per $250K full modular award*.

Quoted bits are from my prior post.

Did you knock our socks off? This could be amazing ELEVENTY type findings, GlamourPub record (whether “expected” for your lab or not), unbelievably revolutionary advances, etc. If you have a record of this, nobody is going to think twice about what your Aims may have been. Probably won’t even give a hoot whether your work is a close match to the funding IC, for that matter.

We should probably separate these for discussion because after all, how often is a panel going to recognize a Nobel Prize type of publication has been supported by the award in the past 5 years? So maybe we should consider Glamour publications and amazing advances as two different scenarios. Are these going to push any renewal application over the hurdle for you even if the remaining items below are lacking? Does GlamMag substitute for direct attention to the experiments that were proposed or the Aims that guided the plan? In the extreme case, should we care if the work bears very little on the mission of the IC that has funded it?

Were you productive? Even if you didn’t WOW the world, if you’ve pumped out a respectable number of papers that have some discernible impact on a scientific field, you are in good shape. The more, the merrier. If you look “fabulously productive” and have contributed all kinds of interesting new science on the strength of your award(s), this is going to go down like gangbusters with the review panels. At this level of accomplishment you’d probably be safest at least be doing stuff that is vaguely in line with the IC that has funded your work.

Assuming that Glam may not be in the control of most PIs but that pedestrian, workaday scientific output is, should this be a major credit for the continuation application? We don't necessarily have to turn this into a LPU sausage-slicing discussion. Let's assume a quality of paper commensurate with the kind of work that most PIs with competitive applications in that particular study section publish. Meets the subfield standard. How important should raw productivity be?

Were you productive in addressing your overall goals? This is an important distinction from the Specific Aims. It is not necessary, in my view, that you hew closely to Aims first dreamed up 7 years prior to the conclusion of the actual study. But if you have moderate, or disappointing, productivity it is probably next most-helpful that you have published work related to the overall theme of the project. What was the big idea? What was mentioned in the first three sentences of your Specific Aims page? If you have published work related to this broad picture, that’s good.

This one is tricky. The reviewers do not have the prior grant application in front of them. They have the prior Summary Statement and the Abstract as published on RePORTER. It is a decent bet the prior Aims can be determined but broader themes may or may not come across. So for the most part if the applicant expects the reviewers to see that productivity has aligned with overarching programmatic goals, she has to tell them what those were. Presumably in the Progress Report part of the continuation application. How would you approach this as a reviewer? If the project wasn't overwhelmingly productive, didn't obviously address all of the Aims but at least generated some solid work along the general themes. Are you going to be satisfied? Or are you going to downgrade the failure to address each Aim? What if the project had to can an entire Aim or two? Would it matter? Is getting "stuck" in a single Aim a death knell when it comes time to review the next interval of support? As a related question if the same exact Aim has returned with the argument of "We didn't get to this in the past five years but it is still a good idea"? Neutral? Negative? AYFK?

Did you address your original Specific Aims? ...this can be a big obsession of certain reviewers. Not saying it isn’t a good idea to have papers that you can connect clearly to your prior Aims. ... A grant is not a contract. It is quite natural in the course of actual science that you will change your approaches and priorities for experiments. Maybe you’ve been beaten to the punch. Maybe your ongoing studies tell you that your original predictions were bad and you need to go in a whole new direction. Maybe the field as a whole has moved on. ... You might want to squeeze a drop out of a dry well to meet the “addressed Aims” criterion but maybe that money, effort and time would be better spent on a new direction which will lead to three pubs instead of one?

My original formulation of this isn't quite right for today's discussion. The last part is actually more relevant to the preceding point. For today, expand this to a continuation application that shows that the prior work essentially covers exactly what the application proposed. With data either published or included as ready-to-submit Preliminary Data in the renewal. Maybe this was accomplished with only a few papers in pedestrian journals (Lord knows just about every one of my manuscript reviews these days gets at least one critique that to calls for anywhere from 2 to 5 Specific Aims worth of data) so we're not talking about Glam or fabulous productivity. But should addressing all of the Aims and most if not all of the proposed experiments be enough? Is this a credit to a competing continuation application?

It will be unsurprising to you that by this point of my career, I've had competing continuation applications to which just about all of these scenarios apply, save Glam. We've had projects where we absolutely nailed everything we proposed to do. We've had projects get distracted/sidelined off onto a subsection of the proposal that nevertheless generated about the same number and quality of publications that would have otherwise resulted. We've had low productivity intervals of support that addressed all the Aims and ones that merely covered a subset of key themes. We've had projects with reasonably high productivity that have....wandered....from the specifics of the awarded proposal due to things that are happening in the subfield (including getting scooped). We've never been completely blanked on a project with zero related publications to my recollection, but we've had some very low productivity ones (albeit with excellent excuses).

I doubt we've ever had a perfect storm of sky-high productivity, all Aims addressed and the overarching themes satisfied. Certainly I have the review comments to suggest this**.

I have also been present during review panel discussions of continuation applications where reviewers have argued bitterly over the various productivity attributes of a prior interval of support. The "hugely productive" arguments are frequently over an application from a PI who has more than one award and tends to acknowledge more than one of them on each paper. This can also involve debates about so called "real scientific progress" versus papers published. This can be the Aims, the overall theme or just about the sneer of "they don't really do any interesting science".

I have for sure heard from people who are obsessed during review with whether each proposed experiment has been conducted (this was back in the days when summary statements could be fairly exhaustive and revealed what was in the prior application to a broader extent). More generally from reviewers who want to match publications up to the scope of the general scientific terrain described by the prior application.

I've also seen arguments about suggested controls or key additional experiments which were mentioned in the summary statement of the prior review, never addressed in the resulting publications and may still be a criticism of the renewal application.

Final question: Since the reviewers of the competing continuation see the prior summary statement, they see the score and percentile. Does this affect you as a reviewer? Should it? Especially if in your view this particular application should never have been funded at that score and is a likely a Programmatic pickup? Do you start steaming under the collar about special ESI paylines or bluehair/graybeard insider PO backslapping?

DISCLAMER: A per usual, I may have competing continuation applications under current or near-future review by NIH study sections. I am an interested party in how they are reviewed.
*This probably speaks to my point about how multi-award PIs attribute more than one grant on each paper. My experience has not been that people in my field view 5 papers published per interval of support (and remember the renewal application is submitted with the final year of funded support yet to go, if the project is to continue uninterrupted) as expected value. It is certainly not viewed as the kind of fabulous productivity that of course would justify continuing the project. It is more in line with the bare minimum***. Berg's data are per-grant-dollar of course and are not exactly the same as per-grant. But it is a close estimate. This blog post estimates "between 0.6 and 5 published papers per $100k in funding." which is one to 12 per year of a full-modular NIH R01. Big range and that high number seems nigh on impossible to me without other funding (like free trainee labor or data parasitism).

**and also a pronounced lack of success renewing projects to go with it.

***I do not personally agree. At the point of submitting a competing continuation in year 4 a brand new research program (whether b/c noob PI or very new lab direction) may have really only been rocking for 2 years. And large integrated projects like a big human subjects effort may not even have enrolled all the subjects yet. Breeding, longitudinal development studies, etc - there are many models that can all take a long time to get to the point of publishing data. These considerations play....let us say variably, with reviewers. IME.

15 responses so far

  • Pinko Punko says:

    I have a lot of comments on this, but running to class now- more later.

  • Dave says:

    One immediate comment/question I have relates to the original aims issue. For example, if your aim 1 got scooped, you're not going to continue doing it just for the sake of saying you did it, right? In the grand sciencey scheme of things, it makes no sense to do that. But in reality, you are likely to something highly related to the original aims, just maybe in a different model/tissue/pathway whatever. So if you stay in the same ball park, is that sticking to the general? Seems to be a lot of room for interpretation here, which I guess is what you're saying really.....

  • drugmonkey says:

    but what if you figured there wasn't enough left to profitably pursue me-too or follow up work and the reviewers disagree? or, you know, are big fans of replication.

  • MoBio says:

    You asked: "Does GlamMag substitute for direct attention to the experiments that were proposed or the Aims that guided the plan? In the extreme case, should we care if the work bears very little on the mission of the IC that has funded it?"

    Actually I have this situation and discussed with my PO. Was told that since the multiple S and N papers were not 'directly' related to my prior specific aims (even though on the same receptors we study) that the review group would not take kindly to this. Suggested that instead of writing a 'renewal' I should write a 'new grant'.

    Reading between the lines it sounds like Program would not consider this progress relevant to the prior aims and would rather see a new grant.

  • Dave says:

    .....or, you know, are big fans of replication.


  • drugmonkey says:

    MoBio- I do hope you tested that out b/c....hilarious if anyone dissed you.

  • Sam says:

    I love the *** part of this post. I'm about to put in my first renewal at the end of year 4 and feel like I really only just hit my stride these past two years. It's been scaring me.

  • Ola says:

    Can't let the whole "5 year interval" go by uncommented on. Unless you're NI/ESI and get the full 5 (depending on institute policy), most of us mortals are stuck on 4 years. The problem is that really means ~2.5 years (assuming renewal will fund on the A1).

    Say you got funded July 1 of 2013, grant runs out June 2017 (4 years). Submit CR Feb 2016 (NOW!!!), reviewed June 2016, pink sheets back then submit A1 in Oct 2016 (assuming problems can be turned around quickly), A1 reviewed Feb 2017, council in May, next fund cycle begins July 2017. Absent this timetable, it's no cost extension superhappyfuntime!

    So, you get precisely 31 months to do 5 papers worth of science, and get the manuscripts in revision or accepted so they can be listed on the CR application as completed. That's about 6 months per paper. My last paper took 2 years for the science, and another 2 years being crapped on across 5 journals before finally being published.

  • Luminiferous æther says:

    "That's about 6 months per paper."

    Simple arithmetic is nice and all, but it doesn't apply to publications like you describe now, does it? You have multiple projects in play at any given time and at the end of those 31 months, you (hopefully) have at least enough for a couple if not three decent papers.

    "So, you get precisely 31 months to do 5 papers worth of science"

    Are reviewers really that idiotic? I doubt it. Tell me you are exaggerating. I am early career and yet have to serve on SS, but I'm pretty sure that as a reviewer I would notice that the grant is just a couple of years in and would not expect five solid papers in that period of time.

  • Pinko Punko says:

    4 year cycles hammer smaller labs. There are always grants from larger labs with labor on fellowships, so to go with actual larger budgets, also effectively larger on top of that. Have rarely seen previous aims be at issue especially if there was productivity. Only time it has come up has been when new aims are similar to previous.

    I can't remember if the data were previously broken down into 1st, 2nd etc renewal. In my field the culture was for long running grants developing basic themes. Clearly other study sections have cultures where grants are more like projects. This has been discussed here previously regarding the future directions critique. When culture used to long running grants, culture seems to demand evidence for longer term payoff. This raises the bar and favors certain types of application.

  • The Other Dave says:

    I have had plenty of reviewer comments about productivity, but never any indication that anyone knew (or cared) what I promised to do with previous funding.

    This sort of annoys me, because I put a lot of thought into those plans, and I have an excellent record of actually following through with good results.

    Yet at the same time I know people whose proposals are basically their latest in-prep CNS paper with some never-to-be-fulfilled promises attached. They have no intention of doing anything in the proposal. They think proposals are for getting money, not describing things you actually plan to do.

    Honestly, I have never figured out whether to be upset by this or not. On one hand, it bugs the hell out of me that these people have no intention of actually doing the work. It seems sort of fraudulent. On the other hand, I think it's silly to force people to avoid spending money on another better project, if they happen to have one.

    I think the higher rates of renewal (vs new applications) are due to two things: 1) ANY previous funding helps you look good. 2) Everyone attributes every publication to every grant, whether it was supported by a particular grant or not. If you have 3 R01s, then you can have one-third of the productivity per dollar as someone with one R01, and you're going to look just as good or better (because you're also more likely to have people supported by training grants, and have supplements, etc) in the fucked-up rich-get-richer system that NIH has built with the help of already-rich cronies.

  • Drugmonkey says:

    I think I lean towards the "follow through" model myself. I also think you should get credit for this in review but I don't experience this as anything anyone loves about my continuations. This is particularly acute when there have been multiple grants funded on ~similar topics and we are the only ones to actually produce the goods.

    Anyway, back in the day no doubt renewals had high success rates because any one of several performance measures was enough. That doesn't seem to be so anymore.

  • Grumpy says:

    This is really an extraordinary blog, as a new PI I am really grateful for it.

    How do you maintain such a high content level and still manage to do anything else (like demonstrate sufficient productivity for renewal)?!

  • drugmonkey says:

    Blogging takes far less time than everyone seems to assume.

  • Eli Rabett says:

    Especially if you get your post doc to do it.

Leave a Reply