An observation for those reading along with a comment thread that is developing on a prior post. Gummibear asserted:
I also have to add that the quality of the NIH peer review system needs an external audit. Things that are going on there are quite unimaginable in journal peer review.
It emerged that our commenter was ticked about a grant review. Surprise, surprise.
Like regularly writing utter nonsense in summary statements, with complete impunity. An example from my recent experience: a reviewer was unfamiliar with the field and wrote a whole critique full of rubbish. He/she 'luckily' went too far and devoted a paragraph to certain methodology, expressly describing my use of it as 'strange'. It was then easy for me to provide a list of literature references to identical approaches and prove that the 'strangeness' resulted solely from the reviewer's state of mind and education. So I did in an appeal.
There is a little more detail but really it is going to be hard to evaluate the specific claim of mistreatment outside of going through the grant app and critiques ourselves. Nevertheless, I like to look for the general points. I arrive at this:
It is most useful to dissociate your disagreement with an established NIH process from your own particular treatment within the process.
Because there may very well be something that is an arguable problem with the process. Fine. In the case of what Gummibear is describing it seems like it might be an instance of what I consider to be reviewers inappropriately trying to predict empirical outcome rather than evaluating the approach (not sure I grasp the subtlety here but it isn't really the point). In other cases it might be the inevitable tension between finding a reviewer who has direct expertise and one who is not a potential competitor. (Because believe you me, the complaint against the expert yet supposedly in competition reviewer is another frequent one). There are very frequently going to be people (~80-90% of them per round, these days) who think the review of their project was of poor quality.
Everybody is up against the same system. So it is not specifically unfair to you if you ended up on the short end of the reviewer quality stick. It is going to happen and frequently does. But sometimes it works in your favor, does it not?
Question: Would you be crying APPEAL!!111!! if the inexpert reviews happened to miss the glaring flaw* in your approach and you ended up with the money anyway? Of course you wouldn't. You would respond in a manner such as what R(r)evere(s) recently observed:
3. YES!!! (pumps fist in air) THEY BOUGHT IT!
As Comrade PhysioProf mentioned, there is a process.
Holmes, I feel your pain, but what you are talking about is a "scientific difference" that NIH intends to be addressed via the ordinary peer review process.
If your problem with the instance of peer review you are appealing could be addressed by explaining scientific shit to the reviewer and inducing a realization that you were, indeed, correct in the first place, then it is not appropriate for appeal. Period. The NIH Web site explains all of this very clearly, and *they* get to define their terms, not you.
Exactly. And when you call your Program Officer all outraged, s/he talks you down and makes the exact same points. A good fellow grant writing colleague should say the same thing. "Yes, sometimes review is less than perfect but the process is to revise and resubmit your grant." "Yes, the timeline for this blows and there could be other approaches (like the ones some nutty blogger suggests) but for right now this is the system." "The fact that you got a bad review is not evidence that you have been selected for special hosing and deserve special treatment."
The trouble comes in when you are unable to step outside of your narcissistic outrage to see that there is nothing special about your case. Nothing.
When you can't see this, well...that gets you put in the crazy file. And that is not a good place to be when it comes to Program. Ever heard of the fable of The Boy Who Cried Wolf? If you are in the crazy file, your chances of a getting a Program pickup or even a solicitation for rebuttal go way down.
There is also the chance that you will be unable to control your outrage when you are writing your Introduction on the resubmission you finally manage to write. This is not a good place to be either. I've seen some doozies where the application sounded much like our commenter, just convinced the prior application had received a screw-job of a review. This doesn't go over well, even when you've managed to switch study sections entirely. The reviewers may actually agree with you that the review comments were in error or at least that your rebuttal is convincing. But if you come off as a spoiled brat who thinks that s/he should exist outside the existing flawed process...you go in the crazy file. And you don't want to be there with reviewers either.
*Yeah. They all have problems. There is no such thing as a perfect grant proposal. Deal.