Nip this in the bud right friggin now, CSR!

Nov 09 2011 Published by under Grant Review, NIH

There was a rumor on the Twitts today about a study section experience in which the triage line was severe (35%, yikes)...but that isn't the bad part. In a 10% or worse payline environment, discussion of the top 35% seems appropriate.

What was absolutely horrifying was the blocking of the usual (IME) rule that any reviewer could lift any application out of triage and insist on discussion.

I believe this to be an absolutely fundamental safety valve to avoid the frequently bemoaned notion that "one biased reviewer torpedoed my grant".

I myself have found occasion to pull grants up for discussion. I doubt I ever got those particular versions funded but I know for sure of ones that got funded on a subsequent revision. I can't prove that my pulling it up for discussion led to the favorable outcome for the revision. But I think it safe to assume that if this path is replicated elsewhere in CSR then, statistically, this has an effect.

Don't get me wrong, one reviewer saves are going to be rare...but what if it were your grant?

8 responses so far

  • David / Abel says:

    Agreed, enthusiastically. Getting a score from discussion, even a lukewarm one, definitely improves funding probability on subsequent submissions relative to an outright triage. Given aggressive triaging behavior, it's absolutely essential to preserve the time-honored practice of allowing any reviewer to bring up any app for discussion.

    I say this also because of one of your old mantras about applicants not trying to guess who on study section were your reviewers. More often than not when I serve, I don't get those apps where I'm an expert - and vice versa. So, something I may triage may have substantial merits to disciplinary drill-down experts that should be brought to light.

    Hence, I would object enthusiastically if blocking this practice were being instituted across CSR.

  • drugmonkey says:

    Also important for when sub discipline nerds miss the big picture, or Oldsters beat up a Noob, or theoretical wars within a discipline erupt. Or.....

  • anon says:

    Does the CSR provide any rationale for blocking this rule? For instance, does eliminating this rule save time? If this practice is in place, at what expense is it? I've never been on a study section, so I don't know what goes on. If there are too many applications to deal with, then maybe I can see why. If no one is allowed to bring up an otherwise triaged application, does this leave more time for discussing the other applications and does it help to separate the top 10% from the rest?

  • New Asst Prof says:

    I just finished serving on a Special Emphasis Panel in which this happened too. Our triage line was 28% (gulp!). And while a couple of apps got brought up out of triage, the chair more than insinuated that panel members should only do this for grants that they had reviewed.

  • longtime lurker says:

    Our study section only allows resurrection after all the ones have been discussed that made the initial cut. At that point we are told to only bring one up if we seriously think we can argue for it to be scored higher than 1/2 of those we have already discussed (i.e., for an initial 50% triage rate, that would mean this one would actually have to be in the top 25%)...since it is very unlikely that more than half of those discussed will be fundable anyway.

  • drugmonkey says:

    At that point we are told to only bring one up if we seriously think we can argue for it to be scored higher than 1/2 of those we have already discussed

    GRRR. Where does this philosophy end? I mean, why are we discussing anything that is more than, say, the top 15% when the paylines are 7-8%, maybe 10% at best, amirite?

    The reason to pull one up is because it deserves to get discussedin the opinion of at least one reviewer. The fact that triage is unanimous across the entire panel should, in my view, be meaningful and is a very good thing. Yes, yes, people might fail to speak up where they were thinking "maybe" and the backstopping is not perfect. But there is a chance for a mistake to be unmade. Therefore, triage must be viewed as a "voted" outcome of the panel in some critical sense.

    That is an absolutely essential contribution to faithful function of the system in my view.

    Also, in my experience, mistakes do get unmade through this process. We all know that triage/not triage is a meaningful distinction. As is the score/percentile rank within the supra-payline, sub-triage band of scores. Meaningful to program pickups, the chances an applicant will revise the application and very likely meaningful for the fate of the application on re-review*.

    *given a tendency of reviewers to benchmark to the prior score, even though they are not supposed to do so. also due to the nature of the incremental argument- I would bet that the mere familiarity of the entire panel with the application due to the discussion of a prior version has an impact on the review of a revised version.

  • drugmonkey says:

    If no one is allowed to bring up an otherwise triaged application, does this leave more time for discussing the other applications

    Not meaningfully, imo. I've seen at best maybe 2 applications pulled up from triage in any given round of review. Even if it were as many as 4, well, this is within the usual variance of tied apps at the triage line, bringing up ESI/NI apps and/or nonR01 mechs so that the triage burden doesn't fall disproportionately etc.

    IME, meetings are held over two days. Early in my service we were going two full days and ending at 5pm. Later on, we were regularly done by noon, 1pm or so. 5-6 more apps could have been discussed if necessary. At the least.

    Perhaps these policies are coming with meetings that are scheduled for a single day and the NIH is saving on the hotel costs? That could be a significant reason. But I'd still rather see the triage line drawn lower and continue to permit the possibility that one or two apps would be pulled up from triage.

    and does it help to separate the top 10% from the rest?

    I think the goal is to have a system which prevents one, two or even three assigned reviewers from having absolute, irrevocable power over the fate of that application. I am much more comfortable with the system which, formally speaking, gains the input of 20-30 folks. Even if in most cases it will be a rubber stamp of the assigned reviewers, it seems very critical to me that the possibility of a correction is present.

  • I am not sure about this, and I am too busy too look it up right now, but I seem to recall something in the federal laws and/or administrative regulations that control CSR behavior that the decision to triage is required to be unanimous, thus leading to the "any one reviewer can force an application to be discussed". But maybe I'm just talking out my ass.

Leave a Reply