Your Grant in Review: Independence, Efficiency and Expertise

Jun 10 2009 Published by under Grant Review, NIH

One of the subtle* aspects of NIH grant review that interests me is the competition between independence of review and other more practical matters. As you will recall, the meat of the review is generally three individuals who are assigned to the application as primary, secondary and tertiary reviewers. Up until very recently, the identity of the other reviewers on grants to which an individual was assigned were made available at the start.
This is an obvious violation of the independence of review, is it not?

In an ideal world you would desire the individuals reviewing your application initially to be independent of each other. And the system always respected that in one key area- no reviewers are permitted to contact or discuss applications with each other anywhere other than at the meeting of the study section.
So why did they use to provide the names of the other reviewers? I have no idea. However I can speculate based on my experiences that it helps with the efficiency of the process and the ability to ensure appropriate domain coverage for the application. By way of example, in our standard Bunny Hopper example, you might have an individual that was highly expert in muscle contraction physiology...but didn't know jack squat about bunnies, being a kangaroo expert and all. Conversely, you might have an expert in behavioral bunny locomotion models who didn't know much about neuromuscular junction pharmacology. Etc. Since most reviewers feel that they should do a decent and comprehensive job, they frequently spend some extra time doing background reading in an area in which they are not entirely expert. Knowing that another reviewer has the requisite expertise helps to alleviate the pressure to do this. You basically trust her to cover the stuff which is obviously in her domain. Efficient.
Trouble is, that this has a tendency to decrease the review of particular aspects of the application from three people's takes to the opinion of a single person. Not so good. Or not ideal, anyway.
There is another slightly weird aspect to this situation in that a reviewer could not (I think) determine who was reviewing the other applications assigned to the panel up until the app started being discussed at the meeting. Of course, the so-called "read phase" of review started a week before when all the critiques were uploaded and made available (with the preliminary scores) to the entire panel. Doing due-diligence and reading over the sometimes highly variable or disparate critiques, one had variable amounts of knowledge** about the reviewers. This might or might not influence outcome, of course. Remember that any member of the panel can write additional critiques and/or insist that a given application be discussed. It is not inconceivable that knowledge of who wrote what criticism and assigned a particular score might influence willingness of a reviewer not assigned to the application to take action.
Well, for the past couple of rounds this has become almost moot. One does not know until the meeting who was assigned to review the applications that one has received. Of course, one can guess**. But I did note one interesting feature in the online review system with the new scoring procedure. There is currently a way to view all the sub-scores which are assigned to a given application and these are identified by reviewer name. So now we have even more information about reviewer identity than ever before.
This has to be a bug, not a feature, right?
*read, obscure; possibly of interest to nobody other than me
**if you are on the panel for some time you might have some pretty good bets about who wrote what about a particular application assigned to other reviewers.

5 responses so far

  • qz says:

    I appologize that this is question rather than comment. Could give us some idea about the new scoring system? I just got a score of 24 with no percentile. Just anxious to know what it means....

  • DrugMonkey says:

    qz, my understanding is that within the 1-9 scale the panel voted an average of 2.4. I do not know how this will translat to percentiles- next to no info yet because it is so new.

  • Danny O'Rerio says:

    qz -- That's about the 10th percentile* I'd say cautious congratulations are in order.
    *Obvious disclaimers to this estimate apply.

  • Dr. Feelgood says:

    Yeah any score in the 2s is extremely excellent. It means there are minor fixable issues but that the application is excellent. However, there is some front loading on apps going on because people are afraid to use the 9 point scale and are mostly keeping to the 1-5 range....Aaah psychology. This will be to your advantage now until people learn to spread 'em.
    Doc F

  • pinus says:

    Lets hope they fear the spread just a little bit longer.

Leave a Reply