This Post Has 35 Comments

1. So bottom line, don’t bother resubmitting a triaged A0. Got it.

2. I would very much like to see this broken out even further to address what a NI/ESI should do in this setting. I talked recently to my program officer who indicator that in her portfolio, half of funded ESI’s were triaged the first time. Offers some hope, but there are still likely a lot of triaged ESI’s that are never funded or not funded at resubmission.

3. My bottom line – resubmit my 26% A1 as an A0. Still trying to figure out how exactly to do that by Feb 5th…

   I just had an unscored R21, will have to parse the posted data to see if R21’s fared differently. Probably not!

   One really needs a thick skin to play this game in order not to take these rejections personally.

4. Why, do you think, do A1’s that went unscored at the A0 stage do so poorly? Is the fact that it was triaged a black mark that cannot be easily overlooked? Is it that the same reviewers get the application again, and not liking it the first time, don’t like it again? Are most of the people that write grants that get triaged likely to make certain grantsmanship mistakes that lead them to write bad A1’s?
   I have recently seen grants from really good scientists (who have been well-funded in the past) go unscored. This observation makes me think that if the pink sheets give you good options for revision (clarifying a missed point, adding prelim data that you have or can easily get, etc.), then you should revise and resubmit. The numbers in this post make me wonder if one would be better off resubmitting the application as an A0 to avoid a black mark and maybe get different reviewers and then have 2 more shots at funding.

5. This is sort of what I try to explain to some of my younger PI’s but now I have numbers to back it up. Thanks! In my experience, it is rare a triaged application will be funded on resubmission. We have had a couple over the years, but I think the safer route is to scrap the application and start from scratch. Even when the criterion scores don’t look too bad – if you get a feeling that the reviewers shrugged when finishing the application, the chances are the application is not going anywhere. The experienced PI’s have no trouble with this, but I do have to wrestle with younger PI’s regarding this point!

6. …..my program officer who indicator that in her portfolio, half of funded ESI’s were triaged the first time

   Hmmmm, that sounds…..not quite right. Does she mean that of all ESIs that eventually got a grant funded, half of them had their first ever application triaged? Doesn’t necessarily mean the same app.

7. Why, do you think, do A1’s that went unscored at the A0 stage do so poorly?

   I suspect a large part is that there are other revised apps coming in that were scored in the prior round. While we still have this tendency for reviewers to at least start from the prior score, this gives the previously triaged A1 a hole to climb out of.

   Is the fact that it was triaged a black mark that cannot be easily overlooked?
   I am not certain the mere fact of triage (vs, say, a 50%ile score) is the problem, although perhaps there are still some reviewers that have a huge mental block on this.

   Is it that the same reviewers get the application again, and not liking it the first time, don’t like it again?
   The review process doesn’t do very well at communicating “this is so boring/meaningless that it doesn’t matter what you do, it ain’t getting a good score ever”. Some think they can glean this from comments (hi PP!) but I’m less certain. Also, it may be the case that more of the section members than just the current three will find it boring. So new reviewers is no guarantee.

   Are most of the people that write grants that get triaged likely to make certain grantsmanship mistakes that lead them to write bad A1’s?

   I think you have this confused a bit. sure there are going to be bad grant writers that struggle with every aspect, including revision in face of critique. but having written a lot of triaged apps and having been given scores that are probably as good as it got in a given study section round…I am very much not convinced that grantsmanship is the main driver. This is buttressed by my years on study section and other happenstance views of the proposals of highly-funded luminaries…they write decent grants, sure, but not so awesome as to be a clear difference in grantsmithing compared to us plebes that fight for every one. The relative triage protection of the biggest PIs over the past decade and a half has more to do with their track record than their grant crafting.

8. “The relative triage protection of the biggest PIs over the past decade and a half has more to do with their track record than their grant crafting”.

   I totally share this view.

9. There ought to be a more official distinction with regard to reasons for triage, such that if an app is walloped heavily on Significance, the proposal is just plain blocked from resubmission period.

   Alternatively, based on these data, just make it official and block all triaged applications from subsequent resubmission. Maybe then use the extra reviewer time and money to bring back the A2, perhaps restricting it to A1 proposals that come in under ~30%ile or something.

   Hell, I think any proposal that consistently scores better than 20%ile should be allowed to be resubmitted ad infinitum until it gets funded. Having to completely restructure a proposal because it couldn’t quite make the last yard over what is accepted to be a rather arbitrary pay-line is insane.

10. “The relative triage protection of the biggest PIs over the past decade and a half has more to do with their track record than their grant crafting.”

    Very true…and the real problem with this is the circular logic involved in maintaining said track record.

11. Joe says,

    “The numbers in this post make me wonder if one would be better off resubmitting the application as an A0 to avoid a black mark and maybe get different reviewers and then have 2 more shots at funding.”

    Fine and dandy, but if they (NIH’s nitpickers) come to the conclusion that your “A0” is really an “A1”, this strategy won’t work – NIH will either reject the application or assign it the A1 designation (not sure which). To avoid that outcome, you’d have to make your new A0 significantly different from the triaged A0.

12. I think what you said at the end is the most meaningful: if you don’t resubmit your chances are zero. That’s my calculation anyway.

    FWIW I had an RO1 funded on an A2 that got triaged as an A0 and an A1.

13. It is my understanding that a change in mechanism counts as a new grant even if the studies are identical. This may not be helpful for some applications or needs. As an example, I recently changed a triaged 2 year R21 to a 3 year R01. The A0 of the Ro1 got scored. Waiting on Feb/Mar study section for A1 outcome. I know of several folks that have pealed out an Aim from an triaged R01 that had enthusiastic support and converted that Aim to an R21–with some success.

14. PR-
    You are underlining my point that many “A0” proposals that do well were previously reviewed in some other guise. NIH has to know this but keeps right on pretending than they have shortened the time to funding a proposed idea.

    It KILLS me. They need to fix study section proclivity for the air traffic pattern delay.

15. “For any given round, new applications are submitted one month and then amended applications are due the next month. … what else are you going to do? Take the low odds or take the guarantee of zero odds?”

    DM you have expressed this view before, but I think there is a fallacy here, at least as I see it applied to myself, because my time is both limited and fungible. Experience tells me that substantially revising and resubmitting an R01 is going to cost me ~50 hours, and that will be 50 hours that I am guaranteed not to be working on any of the five manuscripts on my desk waiting only to be written, or doing the research and planning to set new studies in motion, or spending time with my kids that are still young enough to want to spend time with me. Each of these alternative uses of my 50 hours comes with its own cost-benefit analysis, and there are strong arguments in favor of each. I have not previously thought the expected value of resubmitting a triaged application to be worth my time, and the data you are showing here only confirm my thinking. It is nice though to see the real data on the funding success of these applications, though!

16. First off, thanks DM for the analysis and stats. I agree that grants that were not funded (triaged or not) are making re-appearances as a new A0.

    Congrats Namnezia on getting funded after triage – I never was able to climb out of the hole (even with an A2).

    Meshugena313 in re: thick skin. Indeed. One of the hardest lessons to learn. But I think the more important and difficult lesson is the one that Crystaldoc mentions: the tradeoff between time on grants vs. time on mss, especially for ESI. If you don’t have the pubs to back up your productivity, experience and feasibility, there is no point in writing the grant.

    I agree that the holding pattern (we’ll get to it next time) – either implicitly or explicitly has a lot to do with the problems.

17. Good point CD! Manuscripts have to keep going out, for sure.

18. I’ve always felt it is better to go with the Devil you know (the reviewers that reviewed the A0) and try to placate them then the Devil you don’t know (a new set of reviewers and comments). If an application is triaged I throw out the aim that garnered the least enthusiasm and add a new aim supported with new preliminary data (since the data presented in the A0 has hopefully been published). In other words, I revise the application. Window dressing does no good for a triaged application. I like that idea that somebody mentioned previously about a “do not resubmit box” — that would certainly take all doubt out of the decision.

19. I have seen many triaged grants go from triage to funded. On one study section, I saw a grant go from triaged to top grant in the next cycle of the study section. You’ve already worked hard on a triaged grant, turn it around and fix it, if you can.

    It’s the “if you can” that matters. Just as with a paper “rejection”, you need to read the reviews. Some rejections are actually “revise and resubmit”, while other rejections are “don’t waste your time”.

    As to DM’s issue about air traffic control, DM seems to have the opinion that there are two options: “waiting in line” and “getting good grants funded early”. But I claim that there are two options: “waiting in line” and “playing the lottery”. Because there is so much noise in the fundability of grants (there is a difference between top 10% [funded] and triaged, but little to no difference between top 10% [funded] and 20% [not funded]), the actual funding ends up being a lottery. There was an advantage to the air traffic control system – you knew you were in line (or had a good chance to be). What we really need is a system where you know you have a good shot at continuing getting funded because you’ve been making good progress previously.

    I’ve suggested this before, and will suggest it again: 1. there is a high bar to get into the system, based on the proposed work-to-do, 2. once you are in the system, renewing a grant is based on productivity, with very clear expectations. Each PI would be allowed one grant in this easily-renewable category. Specifically, there are two kinds of R01s – a “primary” and “secondary+”. To get the initial primary R01, the PI has to go through a full study-section-like nasty review process. To renew the R01, the review process is entirely “was good work done in the last cycle”, and people should know that they’re either “doing fine” or “in trouble”. If you want more than one R01, you have to fight in the whole nasty review process for additional ones. If you lose your primary R01 (because you screwed up a cycle), you have to go through the high bar to get back.

    This would do three things: (1) it would remove the waiting-in-line issue that DM hates (as do I), (2) it would reduce both the grant-writing burden (if you just want your one R01, you don’t need to write much) and the reviewing burden (because we wouldn’t be wasting time reviewing all those extra grants), and (3) it would give you the ability to plan.

    I should point out, this is very much what my older colleagues tell me was the system in their time. Renewing a grant was easy if you had made progress. Lots of people spent entire careers on a single R01.

20. PS. @GAATTC – There is officially a “do not resubmit” category in study section review at NIH. Something like “not suitable for reconsideration” or the like. But I’ve never seen it used.

21. PPS. In my suggestion, secondary R01s would not be renewable. If you want to continue the work on the secondary R01, you would need to fight in the nasty study section each cycle.

22. My first RO1 went from triaged on the A0 to funded on the A1, so it is possible.

    On the other hand, the biggest difference between to A0 and A1 was that my first independent papers as the PI of a new lab had appeared. So maybe it was just a case of “show us that you can actually set up a functioning lab”.

23. AcademicLurker,

    Not sure with other ICs, at least NINDS are funding ESI up to 25%tile. I assume you were a new investigator/ESI. Do you mind telling us the %tile or the impact score of your A0 and Α1?

24. The A1 was 15%. I still benefited from HLB’s new investigator policy. The payline that year was 12% but they boosted all new investigator scores by 10 points, I believe.

25. It also depends on the way it is triaged. If it is triaged because of a consistent poor score across the board, it is probably not going to do well in revision. If it is triaged because you have a split between reviewers who gave it all 1s and a reviewer who completely trashed it, then the flaw could be in the proposed work (high numbers are right; trash it) or in the writing (you didn’t communicate the significance to that reviewer; it can be fixed).

26. But qaz- this is “established investigator” you speaking. What would noob-you have thought when facing a tiny slice of the pie and a raft of plodding grey beards sucking up 85% of the grant funding? Based merely on demographic advantage and not merit. Think through the implications of your plan.

27. DM – Are you kidding? How is that different from what we have now? Lots of graybeards sucking up 85% of the grant funding?

    It’s certainly what I faced on entering the system. At the time, the only help new investigators got was that program would occasionally (very occasionally) help a new investigator with a special pedigree by reaching down just below the payline. New investigators were reviewed in the same mix with 20-year veterans. Now, there’s K99/R00’s and DP2’s and ESI and NI help and lots of other special stuff.

    I would argue that my suggestion would actually help the noobs because most of those greybeards would not be fighting for the noob money, they would have their steady R01. If we are really concerned about this, we could have separate competitions for getting-a-primary R01 (that first high-bar-entry) and for secondary-R01s. My proposal would also (I argue) reduce the number of extraneous second R01s, because people could survive on one R01, which should free up some money. (Not much, I know.)

    There’s no question that we would need to have the right amounts of money divided among the different groups – this much for established investigators, this much for new slots entering, etc. What NIH needs to do is to treat funding sets like a diversified portfolio – this percentage assigned to young, risky, and growing faculty, this percentage assigned to continuing, ongoing, stable faculty, this percentage assigned to special extra R01s, etc. Everyone knows that the easiest way to lose your shirt in the stock market is to chase the “best stocks” by trading constantly. There’s no reason to have the noobs fighting with the returning established faculty.

28. Are you fine with the notion that your stable grant might be $250K whereas the next person might get $750K as her baseline grant?

29. No, I’m not. I agree that that’s a difficult point. And it might be a reason not to implement my stable-R01 idea, because we might find ourselves in the same boat as indirect costs where certain universities, fields, or individuals negotiate themselves sweet deals that make their budgets inordinately expensive. (*)

    But I’ve reviewed LOTS of grants for lots of places in a variety of fields (including clinical, as well as NSF, NIH, and NASA) and I’m pretty convinced at this point that there is absolutely no reason for an R01 to be $750K without bloat.

    There are grants that are not basic science grants that might need that much money, but then we are no longer in the equivalent set of goals. I think it’s a major mistake to include a basic science grant in the same category as a Phase X multi-site clinical study. Which again speaks to the portfolio point.

    * Actually, a much simpler and smaller proposal would be to allow study section to take budget into account when scoring a grant, so that we were scoring based on “is the impact of science worth this expense” rather than scoring based on “impact of science” and then deciding whether the budget is too high. I’ve seen lots of grants where the cost is what the grant would cost (particularly at that university in that city) but that we could do two grants (in another field at another university in another city) for the same price. So we can’t say “the budget is too high”, but we would say “that impact isn’t worth the price”.

30. What a crock, qaz. In other words, your “basic science” at wherever you are is relatively inexpensive so everyone else should have to go through extraordinary justification. But not you, you should get your default basal amount with minimal competition.

    Sorry, try again.

31. Should study sections also be provided a list of the funded grants and PIs already doing a certain technique/model/what all and make sure to do ongoing balancing of the NIH for it?

32. It also depends on the way it is triaged. If it is triaged because of a consistent poor score across the board, it is probably not going to do well in revision. If it is triaged because you have a split between reviewers who gave it all 1s and a reviewer who completely trashed it, then the flaw could be in the proposed work (high numbers are right; trash it) or in the writing (you didn’t communicate the significance to that reviewer; it can be fixed).

    Another key factor is which criterion scores were bad. If the Significance and Innovation were generally excellent, but the Approach was not, then that is a good bet to resubmit. If the Significance and/or Innovation were poor, then that is much harder to overcome.

33. But, if it were about the science, shouldn’t it matter if the work could be done less expensively elsewhere?

    Companies are doing this calculus all the time and have effectively gotten states to compete heavily for their work. Boeing is being given tax breaks in Washington, for example, that the federal research funds don’t (the particular bone people pick is on sales tax for equipment, mind you WA doesn’t have a income tax, so they’re not recouping federal funds on salaries like the higher tax states)

    If, as federal taxpayers, we think placing work at a certain institution creates benefits for the institution and the region, why isn’t it reasonable to expect the institution and area to invest in the endeavor as well?

34. Nonprofits like Universities and Research Institutes don’t enjoy tax breaks zb? Of course they do.

    but yes, you have the first plausible argument I’ve heard on this topic. Kudos.

    Question. Is this for all federal contracts for goods and services or just the exceptionally high profile and high budget items? Do all participants in the chain that produces the latest fighter jet participate? Or only the principal contractor?

    If it *is* only the biggest companies that get the breaks….does the government expect the smaller ones to cost account with efficiency similar to the larger companies? Or is this just a deal by deal scenario?

    I think what we’ll find if we follow your scenario down is that at best it matches the federal indirect cost situation. A situation of high variability in tax payer co-support for the companies providing the government with what it wants. Also a high variability in profit margin. Universities and nonprofit research institutions already come out ahead on the profit issue….

35. My current R01 was triaged and funded on A1. I couldn’t imagine giving up the relatively easy opportunity to respond to critiques and resubmit. What is there to lose other than your sanity?