Chin up, grantseekers!

Sep 22 2014 Published by under Uncategorized

I'm in the middle of a slightly unusual grant reviewing task. It's weird in several ways but all you need to know is that all of the proposals were previously scored VERY highly. As in highly enough to get funded, highly.

 

You might imagine that these proposals would all be kick ass wonders of grant preparation, supportive preliminary data, innovation, slick and seamless plans of attack and all that jazz.

 

If you have been paying attention to my continued assertions about what happens in regular study section you will not be surprised in the least to learn that this is not so.

 

These suckers have warts all over them. Some of them, in my view as just one reviewer, are pretty terrible on many of the usual dimensions.

 

Just goes to show you.

36 responses so far

  • Dude, we saw this exact same thing in the NIAID example grants.

  • Former Technician says:

    I think grants are held to a higher standard than they used to be. My boss is constantly surprised that his grants are no longer funded just because he is on them. I have read the proposals that were renewed time after time. Some do not even make sense.

  • Drugmonkey says:

    These are all very recently reviewed.

  • "grants are held to a higher standard than they used to be"

    This is a meaningless statement per se, as grant funding is based solely on the rank ordering of grants, not the "standards" used to rank grants. What these kinds of comments reflect is the realization of PIs that never had any problem keeping their grants in the top 25% to get funding that now they need to be in the top 10%. And there has been a corresponding shift in study section behavior that used to just give great scores to all of the top 25% of grants and only parse out differences lower down that now are distinguishing the perceived top 5%, 10%, and 20%. All the PIs who used to regularly bring up the rear of the top 25% now think their grants are held to a "higher standard than they used to be".

  • Dave says:

    I'm co-investigator on an R01 that I was convinced would be triaged, but it ended up with a 2nd percentile score. Every other grant from this PI has been triaged, so it seems like a pure fluke to me. The impact score was like 25 too, so I'm guessing it was an especially shitty batch of grants for that study section.

    But, at the end of the day, it's the winning that counts.

  • qaz says:

    As with any real measurement, a grant review score is an additive combination of a real, but unknowable number plus noise. The noise term is very large and tends to swamp the ability to differentiate grants.

  • Drugmonkey says:

    I disagree. It implies there is a single quality dimension. this is not so.

  • Ola says:

    I've seen this also at AHA, where they often have a 2nd round of council-esque type review by external reviewers, after the proposals have already made it through a regular study section. It is often the case that I'm left thinking "how the hell did this one get past the first round?"

  • rxnm says:

    "If you have been paying attention to my continued assertions about what happens in regular study section you will not be surprised in the least to learn that this is not so. These suckers have warts all over them."

    That is because stock critiques are for excluding grants we don't like, not for punishing grants we do like.

  • damit says:

    Come on.....haters can always find a way to hate.

    Yes, I have seen highly scored projects where the pi essentially pissed down their leg...and ones chosen for select pay that did the same.

    What's your point?

  • Drugmonkey says:

    As always, my point is that it is not about writing some theoretically perfect application.

  • Mytchondra says:

    You want an AHA grant? You gotta know Tony, in the back of the joint. Otherwise don't call us, we'll call you, nkay?

  • dsks says:

    As I See It Yes.

  • DJ says:

    One question (from a PI who is about to send in a resubmission of a 12th %ile A0):

    If most of the summary statement comments are reasonably addressed, how do new reviewers (i.e. people who are reviewing the A1 without having reviewed/seen the A0) calibrate themselves? I've been told by many colleagues that a 12% A0 is almost a slam-dunk for funding on resubmission, but I'm much less convinced. They argue that it is very hard for reviewers to actually lower your category scores from the A0 if most weaknesses are taken care of. But I'm extremely paranoid that the A1 will just be whole new batch of reviewers and another crap-shoot. It feels to me like getting a worse percentile score is a real possibility.

  • MoBio says:

    @DJ:
    What I've seen happen frequently with these grants which barely miss being funded is that I'll review them again--and again give them a great (or greater score) and be informed a week or so before the meeting that they were funded.

    So I'd recommend addressing any concerns that can be addressed--with new data if needed and hope for the best.

  • Philapodia says:

    So the moral of the story is that it doesn't matter how well you write a proposal or how much effort you put into it, grants are really funded at the bar during conferences when you're chatting with your drinking buddies who are are on study section? Is that about right?

  • Joe says:

    @Philapodia "grants are really funded at the bar during conferences"
    I find this not to be the case, and reviewers are strictly forbidden from discussing proposals they reviewed outside of the study section meeting.

    @DJ "It feels to me like getting a worse percentile score is a real possibility."
    Reviewers are told that the score a proposal got the first time through is not relevant and should not be used in considering how to score a proposal. I have seen proposals that just missed the mark the first time get triaged the next time. However, I think reviewers are somewhat swayed by a good score on the A0 and a nice and responsive introduction to the new proposal. Remember, at least some of the guys that gave the good score before are probably sitting in the room with you, so you are going to have to carefully and diligently investigate and explain any problems if you are going to give the proposal a poor score.

  • Philapodia says:

    Talking about a cool new project you're working on with your buddy over a nice single-malt after a break-out session and discussing a proposal actively in review are two different things. My question is, if your buddy knows about the project in your proposal when it drops into their review hopper will they declare COI and leave the room (if they aren't actively collaborating with you)? I suspect not in the majority of cases. Those are the people who you want in the study section because they know how Super!!eleventyty!! your work is and that it deserves to be funded (and maybe a little quid pro quo in the future). Besides, has anyone ever been caught and drummed out of Science talking about a review outside of the cloister? If anyone has any anecdotal tales it would be interesting to hear about them. The idea that "reviewers are strictly forbidden from discussing proposals they reviewed outside of the study section meeting" IMO is a rule without any real salt behind it and is meant to scare reviewers into compliance. It's an honesty policy, and while I do agree with it I suspect it's been violated a time or two.

  • drugmonkey says:

    how do new reviewers (i.e. people who are reviewing the A1 without having reviewed/seen the A0) calibrate themselves?

    This is one of THE most ridiculously frustrating aspects of doing grant review, IMNSHO. I have yet to come up with any reason this is a good idea. It is variable, of course, because some of the reviewers on a panel will have been there for the prior review! So there's another random factor adding variability to the process. I'd say it really comes down to whatever pressure there is within reviewer (and sometimes at the discussion table) not to bring up new stuff. Also, this is one of those things driving review towards an assessment of the document rather than the proposal- this I don't like.

    But I'm extremely paranoid that the A1 will just be whole new batch of reviewers and another crap-shoot. It feels to me like getting a worse percentile score is a real possibility.

    A worse score is and should be a real possibility. Review is by comparison with other proposals in that round and it is even explicit that reviewers are not supposed to anchor review of the A1 to the score of the A0. From that baseline, you should be happy that there IS in fact a strong tendency for reviewers to start at your prior score. So it is not a "crap-shoot", in fact you have a steady wind blowing in your direction on this. Particularly when it was a just-missed score like a 12%, I would say.

    So the moral of the story is that it doesn't matter how well you write a proposal or how much effort you put into it, grants are really funded at the bar during conferences when you're chatting with your drinking buddies who are are on study section? Is that about right?

    How anyone can possible conclude this from my comments I cannot imagine. You are absolutely wrong.

  • drugmonkey says:

    My question is, if your buddy knows about the project in your proposal when it drops into their review hopper will they declare COI and leave the room (if they aren't actively collaborating with you)? I suspect not in the majority of cases.

    "suspect" is just pulling things out of your hat. Of course none of us can know the extent this happens. What I can say is that I have always found people to take study section seriously and to conflict themselves out on a rather generous basis. OTOH, if everyone conflicted themselves out of proposals from people who they had a drink with at conferences, who they'd discussed data for more than 2 min with, etc, this would pose a serious problem for expert review. I'm in a field where active people are active and all go to the same smaller conferences and hang out at the same sessions at larger conferences. By the time you got done claiming a given reviewer either "hated" or "was too friendly with" the PI of a given app then CSR would never get anything reviewed.

    Those are the people who you want in the study section because they know how Super!!eleventyty!! your work is and that it deserves to be funded (and maybe a little quid pro quo in the future).

    Not everything can get a fundable score and reviewers who are pumping clearly crap apps do not have much influence on the rest of the panel. Reviewers cannot behave on "quid pro quo" because they would have to give everyone a great score, all the time. Doesn't happen.

    The idea that "reviewers are strictly forbidden from discussing proposals they reviewed outside of the study section meeting" IMO is a rule without any real salt behind it and is meant to scare reviewers into compliance. It's an honesty policy, and while I do agree with it I suspect it's been violated a time or two.

    Sure. "a time or two". This is worth bleating about? out of 60,000 apps per year?

    I disagree that the role of review rules is to "scare" reviewers as if they are just eager as a beaver to commit review shenanigans. As I said, the vast majority of the people I've seen reviewing take their responsibilities seriously. It also matters what sort of talking out of school you mean. Some sort of conspiracy between reviewers on the same panel to cause a certain fate for a given app is really super bad stuff. You can't have "discussion" of an app outside of the presence of the Federal Official. I have never caught so much as whiff of that. In context this appears to be what you are referring to here?

    Telling someone what happened during the discussion of their grant, after the fact, is pretty bad but talking only generally about the review culture and expectations of a panel is actually okay.

  • drugmonkey says:

    With that said, I can also say that I have recently heard a sordid tale of a PI asking another PI for special treatment for an application whilst simultaneously hinting about quid-pro-quo for the other PIs applications. Let's just say the evidence was overwhelmingly clear.

    So I'm not being naive here. It can and does happen that there are attempted shenanigans and, one assumes, successful shenanigans. Yeah, there is error in the system. But I am skeptical that these cases represent more than a small minority of grant outcomes.

  • Established PI says:

    DM - Why weren't these proposals funded, and why are you re-reviewing them? Just wondering what we're talking about here.

  • How could it possibly be a conflict of interest to have seen someone present work at a conference, discussed it with them over beers, and then to review their grant and/or manuscript relating to the same work? If this were the case, the only non-conflicted reviewers for most grants and manuscripts would be people with the least relevant expertise. Nor is it a conflict-of-interest to review your good friends' grants and manuscripts. Conflict-of-interest requires a lot more than this: direct financial interest in the outcome of review, current or very recent collaborator, employed by the same institution, or family member.

  • drugmonkey says:

    E PI- I don't think I mentioned whether they were funded or not. As I said, we're talking about a very unusual review task. One which, if you are paying attention to my remarks about review, is something I could not possibly have refused once asked to participate.

  • Philapodia says:

    @DM: I never thought you were being naïve. I took the moral as I saw it from the following lines of your original post: “It's weird in several ways but all you need to know is that all of the proposals were previously scored VERY highly.” Followed a few lines later by “These suckers have warts all over them. Some of them, in my view as just one reviewer, are pretty terrible on many of the usual dimensions. Just goes to show you.”. I understand were trying to make another point about grants not needing to be perfect, but how would something that’s considered terrible be scored very highly in the first place without some guardian angel looking over it? This wasn’t clear (unless it relates to sunk cost from your previous post?) and suggested that something was going on outside of honest review.

    I agree that talking about a review outside of cloister is wrong, but you also want people who know about your work to review your proposal. There isn’t a COI unless you are actively collaborating with your buddy or have a financial interest in the work, so it’s all good to have your drinking buddy review your proposal. That was my original point, is that if people know and like your work it’s easier to achieve a favorable score, so networking at the bar can help get grants funded that may have struggled more otherwise. People within the field will know about your work if you’re doing your job right, and while there probably isn’t any real quid pro quo going on (Mr. Chow’s snarky voice was playing in my mind as I wrote that, I guess I was misconstrued as being serious) there is a tendency for people to support who/what they like or know and not give as much slack to something unfamiliar (at least subconsciously). Therefore grants are really funded at the bar during conferences.

  • drugmonkey says:

    Ah. Great point. The first place to start is that I am but one reviewer with various biases in what I think is important within and between the five criteria.

  • drugmonkey says:

    Not sure where you jump from what a given reviewer likes, scientifically speaking, to direct interpersonal lobbying though.

  • E-roock says:

    I really don't think grants are funded at the bar like Phila seems to be suggesting. All the hard hitters I know are getting a good nights rest or back in the room working on abstracts, papers, and grants, while everyone else is at the bar. Or left the conf early to get back to work. I actually know, personally, several people on my SS of interest and none of the ones I know will be seen hob knobbing at the bar at Big Conference. I do not discuss specifics with them re grants, but other general questions like where they see the field going and what are some of the major areas Scienz should focus on. Also maybe give your 2 cents too, justified with intellectual rigor....

  • Davis Sharp says:

    CPP: ... Nor is it a conflict-of-interest to review your good friends' grants and manuscripts.

    From the NIH COI page:
    Appearance of a COI: Any situation that could cause a reasonable person with all the relevant facts to question the impartiality of the reviewer or that leads a reviewer to question his or her objectivity means that the reviewer: May not participate in the evaluation of that grant application.

    You should declare a conflict with your good friends' grants.

  • Philapodia says:

    Perhaps they've gone to a different bar (or sushi restaurant / wheat grass shop / coffee house) to discuss the Scienz? I've been at meetings where the big doods are huddled together at various and sundry places, but perhaps my field just likes to drink...

  • drugmonkey says:

    How do you know you are a "reasonable person with all the relevant facts", DS?

  • Comradde PhysioProffe says:

    Next round of study section, I would consider the PIs of six of the ten grants I am assigned "good friends", meaning people I see at least once a year at meetings, hug hello, and drink and party with. I'm still gonna spread their motherfucken scores. Some of the grants we are reviewing this round were submitted by people who just recently cycled off the section as permanent reviewers. They spent two days, three times per year, for four or more years hanging out and working hard intensely with many of the current members. These relationships do not create conflicts of interest, and anyone who has ever been on a study section would know exactly why.

  • sop scientist says:

    CPP; I agree 100%

  • Spiny Norman says:

    Joe says: "reviewers are strictly forbidden from discussing proposals they reviewed outside of the study section meeting."

    AHAHAHAHAHAHHA. Ha. ha.

    I'm always knocked on my heels when someone on study section starts telling me about my or a colleague's review… but it's happened to me, and on a reasonably regular basis. Usually there's EtOH involved.

    And before you ask: No, I do not, ever, ever, solicit that sort of information. I would not put a colleague on SS in that position.

  • Joe says:

    @Spiny
    I was responding to a comment that made it sound like SS members were talking over the grants with each other or with the applicants before the scoring and discussion of those grants. I haven't seen that happen. And yes, SS members are strictly forbidden from talking about the applications and discussion outside the meeting.
    Have I heard people talk about how a grant was discussed after the fact? Yes, though not often. Also, I think it is uncool when a colleague tells me he knows I reviewed his proposal and he's about to send in the revision that is now so much better. I can't talk about that, so don't try that kind of stuff.

  • Spiny Norman says:

    "Also, I think it is uncool when a colleague tells me he knows I reviewed his proposal and he's about to send in the revision that is now so much better. I can't talk about that, so don't try that kind of stuff."

    Wow. Tone. Deaf.

Leave a Reply