Your Grant in Review: Throwing yourself on the mercy of the study section court

Aug 24 2016 Published by under Careerism, Grant Review, NIH, NIH Careerism, Uncategorized

A question and complaint from commenter musclestumbler on a prior thread introduces the issue.

So much oxygen is sucked up by the R01s, the med schools, etc. that it tends to screw over reviews for the other mechanisms. I look at these rosters, then look at the comments on my proposals, and it's obvious that the idea of doing work without a stable of postdocs and a pool of exploitable Ph.D. students is completely alien and foreign to them.

and extends:

I personally go after R15 and R03 mechanisms because that's all that can be reasonably obtained at my university. ... Postdocs are few and far between. So we run labs with undergrads and Masters students. Given the workload expectations that we have in the classroom as well as the laboratory, the R15 and R03 mechanisms support research at my school. Competing for an R01 is simply not in the cards for the productivity level that we can reasonably pursue...

This isn't simply fatalism, this is actual advice given by multiple program officers and at workshops. These mechanisms are in place to facilitate and foster our research. Unfortunately, these are considered and reviewed by the same panels that review R01s. We are not asking that they create an SEP for these mechanisms - a "little kids table" if you will - but that the panels have people with these similar institutions on them. I consider it a point of pride that my R15 is considered by the same reviewers that see the R01s, and successfully funded as well.

The point is that, the overwhelming perception and unfortunate reality is that many, many, many of the panelists have zero concept of the type of workload model under which I am employed. And the SROs have a demonstrably poor track record of encouraging institutional diversity. Sure, my panel is diverse- they have people from a medical school, an Ivy League school, and an endowed research institution on the West Coast. They have Country, and Western!

I noted the CSR webpage on study section selection says:

Unique characteristics of study sections must be factored into selection of members. The breadth of science, the multidisciplinary or interdisciplinary nature of the applications, and the types of applications or grant mechanisms being reviewed play a large role in the selection of appropriate members.

It seems very much the case to me that if R15s are habitually being reviewed in sections without participation of any reviewers from R15-eligible institutions, this is a violation of the spirit of this clause.

I suggested that this person should bring this up with their favorite SROs and see what they have to say. I note that now that there is a form for requesting "appropriate expertise" when you submit your NIH grant, it may also be useful to use this to say something about R15-eligible reviewers.

But ultimately we come to the "mercy of the court" aspect of this issue. It is my belief that while yes, the study section is under very serious constraints these days, it is still a human behavior that occasionally lets real humans make rational decisions. Sometimes, reviewers may go for something that is outside of the norm. Outside of the stereotype of what "has" to be in the proposal of this type. Sometimes, reviewers may be convinced by the peculiarities of given situation to, gasp, give you a break. So I suggested the following for this person who had just indicated that his/her R15s do perfectly well in a study section that they think would laugh off their R01 application.

I think this person should try a trimmed down R01 in this situation. Remember the R01 is the most flexible in terms of scope- there is no reason you cannot match it to the budget size of any of the other awards. The upside is that it is for up to five years, better than AREA/R15 (3 y) or R03 (2 y). It is competitively renewable, which may offer advantages. It is an R01, which, as we are discussing in that other thread, may be the key to getting treated like a big kid when it comes to study section empanelment.

The comments from musclestubmler make it sound as if the panels can actually understand the institutional situation, just so long as they are focused on it by the mechanism (R15). The R15 is $100K direct for three years, no? So why not propose an R01 for $100K direct for five years? or if you, Dear Reader, are operating at an R03 level, ask for $50K direct or $75K direct. And I would suggest that you don't just leave this hidden in the budget, sprinkle wording throughout everywhere that refers to this being a go-slow but very inexpensive (compared to full mod) project.

Be very clear about your time commitment (summers only? fine, just make it clear) and the use of undergrads (predict the timeline and research pace) in much the same way you do for an R15 but make the argument for a longer term, renewable R01. Explain why you need it for the project, why it is justified and why a funded version will be productive, albeit at a reduced pace. See if any reviewers buy it. I would.

Sometimes you have to experiment a little with the NIH system. You'd be surprised how many times it works in ways that are not exactly the stereotypical and formal way things are supposed to work.

27 responses so far

  • Boehninglab says:

    Interesting approach. I don't think I have seen someone propose an R01 at R15/R03 yearly cap. Great idea and would probably stimulate interesting discussion at panel.

  • drugmonkey says:

    Right? And this person's claimed success with R15 shows that the battle is about getting the study section to really focus on the individual situation of his or her institution. Not really about the quality of the science, or the R15 would have never fared very well.

  • Pinko Punko says:

    I suspect the ability of the panel to judge that style of R01 will relate to what fraction of R01s they see that are really three R01s in one. I will say that the R15 review is sort of variable but I think that for panel I am familiar with, they do much better job with R15s than say R21s. Didn't somebody say on Twitter that sometimes panels will fight for the R15 because it's a breath of fresh air after the bloodbath on R01s?

  • musclestumbler says:

    Wow, thanks for the promotion to the front page DM!

    I like the idea in theory, because it might finally level one of the perceived biases. My only worry (and the reason why I'm talking myself out of it right now) is that this might open up a "clearly he/she/it doesn't understand how this mechanism works, and should instead compete for the R15"... a circular firing squad, if you will. Believe it or not, I chatted with my PO over a slight variation on this, and they basically said "hey, if you have R15 eligibility, then compete in that pool". The POs LOVE funding these grants- if we can escape the bloodbath of the study section, that is...

    Back to my point, though, I just feel that the panels need to be diversified (dirty word, I realize), but in terms of scale/workload.

  • chemstructbio says:

    In my experience on a study section that reviews R01s and R15s, we are specifically told by the SRO that the "scientific quality" of R15 applications does not need to be on par with R01 applications. The science has to be good, not necessarily great, and we are instructed that it is the student training aspect that should guide our scoring the most—would we want to recruit students graduating from the lab as postdocs to our labs.

    I suspect that a blanket proposal that reworking a 3yr R15 ($100K/yr) that typically targets this study section to a 5y R01 ($100K/yr) would not fly well if the science is not great, regardless of the budget requested.

  • chemstructbio says:

    (I used a lot of that's in that last sentence)

  • My study section loves R03s, R21s, R15s, K01s, and any other non-R01 mechs. If you submit a good one, you will get an outstanding-to-exceptional score, because we are not diluting our R01 percentile rank list by doing so.

  • musclestumbler says:

    @chemstructbio- "The science has to be good, not necessarily great, and we are instructed that it is the student training aspect that should guide our scoring the most—would we want to recruit students graduating from the lab as postdocs to our labs."

    I don't know whether to be professionally of personally insulted by this, or both. Also, you were given poor direction. Despite what you were told, this mechanism is NOT meant to be a training grant- look up the announcement of you have to. I guess I'm supposed to be happy to act as a training conduit to your lab where the "real" science occurs. This is why non-R01 panelists are critical.

    But hey- thanks for demonstrating the "no R01 = inferior" mentality!

  • drugmonkey says:

    Your PO is not wrong, on the "use R15 if you are eligible". However you might as well *also* write R01s if you want to and have a suitable idea/plan. Even a full mod or bazinga!! one. Why not?

  • SidVic says:

    Thanks guys this was an informative post/comments. musclestumbler money-is-money- don't let prickly pride get the best of you. Any rich, elderly lonely widows out there that want to endow a chair- call Sid.

    The r15 comes distributed 300K in a single fiscal year. Thus, it is possible to stack them up. This is RO1+ level money. 20 yrs ago a guy told me that if you submitted a r15 that was coherent it was sure fire. This has changed, but it is easier to score one- no doubt. It is correct that the POs will tell you that it is strictly science based- not training. Nonetheless i ignore this and include a extensive section on our undergraduate honors program and undergraduate research mentoring apparatus. I may have even burnished it up a little and made it sound better than it is in practice. that stuff might be ignored, but including it probaly won't hurt.

  • musclestumbler says:

    @Sid- let me know if you find the widow, and if she has a friend....

    But yes, the R15 has been a great mechanism for me. The only issue I have to correct you on is the stacking-you can't have more than one at a time (hence the R03).

    As for prickly pride, to paraphrase the movie Red Dawn, it keeps one warm at night, and the grant writing fingers nimble.

  • SidVic says:

    I got this from a PO, Yes, not more than one at time, but it counts as a single fiscal year. It can be dispensed over 3 years by the university. You can recieve award every year.

    Ahhh, a check of reporter indicates that you are correct- unfortunate .

    Find your own endowment bubba- you thinks ROWs grow on trees.

  • drugmonkey says:

    Goals are still to 1) support meritorious research; 2) build research capacity at underfunded institutions and 3) expose undergraduates to meritorious research.

    How different study sections and reviewers emphasisize and interpret these in judging applications varies. I was first exposed to R15 review culture which interpreted the review charge as to prioritize institutions that served underrepresented populations. I thought it was an explicit part of the FOA because of the way reviewers talked about it.

    I also found that debates over relative emphasis on 1) versus providing an excellent experience for undergrads (these can lead to very different conclusions about an application) were probably the most vigorous.

  • Lisa says:

    This is a really interesting idea, but I'm not sure it would fly in my study section. We see quite a few R15s along with our R01s and a couple of R21s each round. The scoring for R01s is COMPETITIVE!!! Frighteningly so. Since R15s aren't percentiled with the R01s, the scoring tends to be more lenient. Our SRO makes it very clear that R15s are expected to be excellent science, just like the R01s, but that we are also able to consider the training potential as well (that's why you have to fill in all that info about the institution, what graduates do after, etc.). I think we do have a few study section members from R15 eligible schools, but I feel like the rest of us are fairly well oriented as to differences in teaching load, access to students, etc.

    We have one R15 recipient who has submitted several grants to our study section who does AMAZING work that I think could be competitive for an R01. I'd like to see this person try DM's suggestion (R01 at R15 levels). But although this person scores 1s on their R15s, I think it'd be really hard to get comparable scores on an R01 because our study section is so ridiculously competitive. I'm sure they all are these days. It would be an interesting experiment.

  • DNAdrinker says:

    I was assigned to review a non-traditional R01 a few years ago. It was from a guy at a university in South America. I thought it was a decent idea and had potential, but it got triaged. One of the major problems was that it was written in the wrong format, it didn't have the standard sections specific aims, research design, etc. It was written more like a review article. People didn't even understand it. Part of my review was just saying "this isn't bad, but you need to massage it into standard format. get a book or something".

    He resubmitted. I wasn't assigned the resubmission, but another member gave an impassioned plea for it. He was only requesting like $30,000/year and it was an important problem. (He just needed some supplies/equipment. I don't think his university had any idea of indirect costs and all salaries were paid for.) Anyway, the study section gave it a great score.

    So there is hope. You just need to attract the right people who are open minded.

  • Ass(ociate) Prof says:

    This is an interesting and informative discussion. I am at an R15 eligible institution and have submitted both R01 and R15 proposals over the years. At the outset, I was admonished to submit only R01s, because people rarely go from R15 to R01 funding, and the end goal of the institution was to build R01-level research.

    Over a number of years I submitted both R01 and R15 applications, sometimes the same projects with different levels of investment and scope. As of last spring, the record was 4 R01s submitted, 2 scored and 3 R15s submitted, 3 triaged. What I could get from the R15 reviews was that they went to SEPs with no interest in either the mechanism or the science. One of those scored R01s as a second year Assistant Prof was an A1, and it was 2010, so I couldn't even send a variation of the project for several years due to the no A2.

    So, yeah. What do do for the next cycle? R01 with smaller scope or R15?

  • Microscientist says:

    I am also at an R15 institution. A few notes that may be of interest
    1) Some larger institutions have tried to "game" the R15 mechanism by having their vet school or dental school be declared R15 eligible. This really ticks off the rest of us, who really do have a different gig. But- these institutions are still there and often do get funded.
    2) There are a few R15 only special emphasis panels. NIAID has one for Microbiology. My experience with this panel is that you get none of the more generous scoring that you might see in a regular R01 panel. It is populated with folks from R15 institutions- which is how I figured out point #1. So damned if you do... damned if you don't.
    3) The sample R15 grant that NIAID has posted is very revealing. Or rather, the posted pink sheet is. Based on my experience submitting to an all R15 panel, those reviewer scores would have resulted maybe being discussed, but highly unlikely to be funded. But it got funded, with a good impact score- and it was in a standard RO1 panel.
    4) I'm glad to hear that some of you have found POs supportive of R15s. My last PO couldn't care less- and has thankfully retired. Hopefully my new PO will be more supportive.

  • drugmonkey says:

    I have often quoted a reviewer from a study section I was on years ago. "It feels unAmerican not to get behind R15 proposals." Everyone nodded. There *are* people who will be enthusiastic about them. IMLE.

  • The Other Dave says:

    Or, you know, NIH could save a buttload of money and (according to loads of social science research) get more accurate and consistent assessments by GETTING RID OF STUDY SECTION. Just get appropriate 'mail in' reviews like many private and European funding agencies do, rank the proposals, and allocate funding.

  • drugmonkey says:

    How did that work out for the Canadians recently?

  • The Other Dave says:

    @DM: Oh, yea. Most people love panel discussions. The same set of social science research I alluded to earlier shows that discussion leads to increased confidence in assessments at the same time accuracy and consistency (from group to group) go down.

    But to your point: By all accounts the CIHR changes were poorly implemented. I think the data are not in yet.

    Fact is, per capita and dollar (equivalent), European science is WAY more productive than in the U.S. Why? Maybe we should look at their system before we throw more money at ours.

    NSF review has been more similar to the new CIHR review for a long time -- more independent external reviews. Unfortunately in my experience the external reviews tend to get blown off during panel meetings, but those reviews still outnumber the panel member reviews, and any recommendations (or not) for funding have to take them into account.

    So how has that worked out? NIH was set up to cure cancer, and has had a budget several times that of NSF every year for half a century. Where are we? Meanwhile, dude. Look at the internet and computer technology you're using. And the engineering that went into your lab equipment. Half the great science-related stuff you talk about on this blog are NSF -- not NIH -- success stories.

    NIH is a horribly bloated and inefficient tax money well. It's the military-industrial complex for geeks.

  • DJMH says:

    Fact is, per capita and dollar (equivalent), European science is WAY more productive than in the U.S.

    [citation needed]

    All the evidence I know of demonstrates that Europeans are keen to come to the US for a postdoc, and in many cases would prefer to stay here to do science, whereas much less so in the other direction. The reason Euro science is so "efficient" is because the bigshots are crowned early and face no competition from uppity youngsters.

  • SidVic says:

    TOD- i think the phrase is "boat load of money" - keep it clean

  • drugmonkey says:

    DJMH- that is the same thing that I hear from French and German postdocs in my field. The sinecure-for-life sounds great...but it comes at the expense of a severe and highly connections-based selection filter.

  • jmz4 says:

    "So how has that worked out? NIH was set up to cure cancer, and has had a budget several times that of NSF every year for half a century."
    -Actually, the NIH was created by the Ransdell act of 1930 for the purpose of "ascertaining the cause, prevention, and cure of disease affecting human beings, and for other purposes."
    Seven years later the NCI was set up. Also, the NIH has only had "several times" of the NSF for only about 20 years or so:
    "Where are we?"
    -Much, much closer than we have been for many cancers, with things like immunotherapy and ChAR therapies. Not to mention things like childhood cancers, where 5 year mortality went from 100% to 20-30% (e.g.

    Building things is always easier than understanding them, unfortunately, for medicine, you often need to do both.

  • Juan Lopz says:

    Nice jmz4. thanks.

Leave a Reply