When Peer Review Breaks Down

May 07 2008 Published by under Uncategorized

DrugMonkey posted yesterday about the "A2 bump" and other study section behaviors that are designed to create a "holding pattern" for grants that will ultimately almost-certainly be funded, but only as a subsequent resubmission. Although he alluded to the fact that this kind of behavior is greatly encouraged when funding is very tight, he didn't really explicitly lay out why. So here we go!

Let's start with a real-world example. One of my R01s was scored 170/30%ile as a A0, 140/10%ile as an A1, and 110/2%ile as an A2.
The difference between the A0 and the A1 was scientifically relevant. The difference between the A1 and the A2 was not.
The reason that all of this is occurring is that the study sections are behaving rationally in response to what I consider an ineliminable feature of any peer review system. Peer review systems are reasonably good at somwehat-objectively distinguishing the the top 20% of grant applications from the rest.
What they are horrible at doing--and for unavoidable reasons that have to with the subjective nature of scientific taste and judgement--is in some repeatable, consistent, objective manner distinguishing the top 10% from the next 10%. But it is this latter task that study sections are essentially being forced into, as only the top 10% are getting funded.
So, in my opinion, what study sections are doing is trying to make this impossible and unfair discrimination that they are forced to engage in as fair as possible. And the way they are doing this is to say, "OK. If a grant made it to near the border of the top 10% on a previous submission, but didn't get funded, and then comes back, we are going to 'make sure' that it gets funded this time."
The motivation behind this is to try to ensure that capricious "wiggle" around the funding cut-off line is not the difference between funding and absence of funding. And study sections care a lot more about this when pay-lines are 10%ile than when they are 20%ile, because the grants that would be capriciously denied funding with 10%ile pay-lines are scientifically much better than those that would be denied funding with 20%ile pay-lines.
This is not to say that this kind of study section behavior doesn't introduce other sorts of unfairness or inefficiency into the system. The point is that their behavior is rational and aimed at ameliorating what is perceived as a particularly harmful form of unfairness.

4 responses so far

  • screwed says:

    The point is that their behavior is rational and aimed at ameliorating what is perceived as a particularly harmful form of unfairness.
    That's the nice way of saying it.

  • juniorprof says:

    Just got back from a dinner where we had a conversation about this and reached essentially the same conclusion, but in many more words. Nicely put!

  • BugDoc says:

    Your post makes a lot of sense: reviewers can't really fix the essential problem that NIH funds are woefully tight right now. I'd be interested to hear thoughts about the role of our universities in being part of the solution to the current problem. Not sure what the policy is where you are PP, but as with many R1U, we are encouraged to pay a minimum of 50% salary from our grants and also pay high tuition costs for all of our students (even the students past candidacy who do not take classes). These costs benefit the university(in addition to the indirects), but substantially decrease the amount of research dollars available from our grants. It seems like rather than just trying to slap a bandaid on the problem by creating some money available for bridging funds, universities might consider amending their policies on charging faculty salaries and student tuition on research grants. I realize this is unlikely to ever happen, but I think that universities helped to create this funding crisis, and should participate in the solution.

  • To follow up on BugDoc, it always seemed absurd in graduate school after I finished taking classes that my PI was paying tuition for the graduate students. In the humanities, the tuition was waived. In theory the costs of me being in the lab were being covered by the overhead from the grants (> 65%). That is another thing, some of the top tier research universities have very high overheads while their endowments expand in leaps and bounds. This just doesn't seem right. In a certain respect the US taxpayers are subsidizing the endowments of these private universities. Other research universities I know have overheads closer to 40-50% which seems more reasonable. On a pre-overhead budget of $800,000 per year that is a difference of between $200,000 and $120,000 per year on overhead.

Leave a Reply