Reminder: The purpose of NIH grant review is not to fix the application

Oct 07 2016 Published by under Grant Review, NIH Careerism, NIH funding

A question on my prior post wanted to know if my assertions were official and written or not.

DM, how do you know that this is the case? I mean, I don't doubt that this is the case, but is it explicitly articulated somewhere?

This was in response to the following statements from me.

They are not charged with trying to help the PI improve his or her grantspersonship*. They are not charged with helping the PI get this particular grant funded on revision. They are not charged with being kind or nice to the PI. They are not charged with saving someone's career.

They are not charged with deciding what grants to fund!

The fact that we are not supposed to so much as mention the "f-word", i.e., "funding", has been communicated verbally by every single SRO I have ever reviewed under. They tend to do this at the opening of the meeting and sometimes in the pre-meeting introductory phone call. Many SROs of my acquaintance also spit this out like a reflex during the course of the meeting if they ever hear a reviewer mention it.

The rest of my statements are best evaluated as I wrote them. I.e., by looking at the the NIH review guidance material to see what the reviewers are instructed to do. There is a complete absence of any statements suggesting the job is to help out the applicant. There is a complete absence of any statement suggesting the job is to decide what to fund. The task is described assertively to:

Make recommendations concerning the scientific and technical merit of applications under review, in the form of final written comments and numerical scores.

As far as more positive assertions on the "fixing applications" front go, the most direct thing I can find at present is in the instruction on the "Additional Comments to Applicant" section of the critique template (take a look at that template if you've never reviwed). This document says:

As an NIH reviewer, your written critique should focus on evaluating the scientific and technical merit of an application and not on helping the applicant rewrite the application. But what if you desire to provide some information or tips to the applicant? The Additional Comments to Applicant box is designed just for that purpose.

My emphasis added. In case this isn't clear enough, the following can be taken in the context of the other guidance document comments about reviewing the scientific and technical merit.

Your comments in this box should not be about the scientific or technical merit of an application; do not factor into the final impact score; are not binding; and do not represent a consensus by the review panel. But this type of information may be useful to an applicant.

Clear. Right? The rest of the review is not about being helpful. Comments designed to be helpful to the applicant are not to contribute to the scientific and technical merit review.

Now the comment also asked this:

What fraction of reviewers do you think understand it like you say?

I haven't the foggiest idea. Obviously I think that there is no way anyone who is paying the slightest bit of attention could fail to grasp these simple assertions. And I think that probably, if challenged, the vast majority of reviewers would at least ruefully admit that they understand that helping the applicant is not the job.

But we are mostly professors and academics who have a pronounced native or professionally acquired desire to help people out. As I've said repeatedly on this blog, the vast majority of grant applications have at least something to like about them. And if academic scientists get a little tinge of "gee that sounds interesting", their next instinct is usually "how would I make this better". It's default behavior, in my opinion.

So of course SROs are fighting an uphill battle to keep reviewers focused on what the task is supposed to be.

11 responses so far

  • JeanM says:

    My experience of 20 yrs on various study sections has been similar. Since the 1990's we have been told that we do not make funding decisions. Funding is strictly the purview of program. However, back in the 90's, reviewers' critiques were meant to be helpful to the applicant and criticism was constructive. To some extent, this forced the reviewer to make sure their comments were accurate and fair. With the revamping of the scoring system and the inclusion of bullet pointed (not always helpful) strengths and weaknesses, the reviews have been directed simply at perceived merit. IMHO the bullet pointed critiques (the number of strengths and weaknesses appropriately balanced to reflect the scores) have resulted in less thorough and less thoughtful reviews. Comments such as "the project relies on standard approaches and is therefore not innovative" will often result in an unscored application. Perhaps there is no better approach than the "standard" one. With no discussion and no check on the accuracy/reasonableness of the reviewers' comments, novel and interesting applications can be unfairly doomed.

  • drugmonkey says:

    With the revamping of the scoring system and the inclusion of bullet pointed (not always helpful) strengths and weaknesses, the reviews have been directed simply at perceived merit.

    Having read some of the justification for this change, I would argue that this was explicitly designed to curtail the "help the applicant" behavior and to refocus the reviewers on scientific merit of the proposal in front of them. One might observe that the banning of the A2 was likewise meant to interfere in the cycle of critique and "if I just fix it like they told me to I deserve to get funded" revisions. As well as the corresponding refusal of study sections to take applications seriously on the A0 submission.

  • Grumble says:

    I have, on a number of occasions, used the "comments to the applicant" box to provide "helpful" advice. I think it's a nice feature because it provides an outlet for our alleged helpfulness - but I say "alleged" because I rarely see any other reviewers using it.

  • chemicalbilology says:

    I can attest that in every review panel I've ever been on in the past six years, there has been an explicit mention that we are not supposed to try to help the applicants with improving their proposal, but only with understanding what the weaknesses were.

    I can also attest that in just about every review I've been on, there have been SOME reviewers in the room who explicitly said they wanted to try to help the applicants do better on their next submission, and others who reminded them that it's not what it's about. So, at least in my review circles, this is a thing that gets talked about.

  • Ola says:

    I once used the additional comments box to essentially tell the applicant never to darken the doors of NIH with this kind of shit ever again. All 3 scores were 8/9, so they probably didn't need to hear that, but it helped me to vent after having wasted a few hours of precious review time.

    I've also used the box to make the odd snarky remark such as "you probably ought to have more than 12 references in your literature cited section" or "the use of 24pt comic sans makes it difficult to take the figures seriously".

  • qaz says:

    Realize that this shift was part of the destruction of the old in-network system of grants being like papers (submit, get reviews, answer the reviews, get published/funded) and more into a straight competition/lottery. There was a time when you were highly likely to get the same reviewers (and even if you didn't), so that the rereview would be mostly about whether you addressed the reviews. There was a very strong attitude against moving goal posts. With the shift to "just address the merits" (effectively asking if it should be funded or not - yes, yes, I know we're not allowed to use the f word, but given scores, percentages, and funding lines, we all know that's the point), each submission becomes more independent and harder to predict. Helping make the grant better was part of that two-step cycle, which provided stability. The new system is much more unstable, which means each person submits more grants, there is a lower chance of getting funded, funding is more variable (so some have too much and some too little), and the whole system is much more stressful.

    You can make this arguement that study section should be "just the facts/scores, ma'am", but this has damaged a process that worked for our predecessors better than it does for us.

  • Cashmoney says:

    "Helping to make the grant better" was always a group delusion.

  • Joe says:

    I have sometimes used the "comments to the applicant" box to mention some BS issue brought up in the discussion that the applicant would need to address but that I would not list in my "weaknesses" points (because it was wrong).

  • iGrrrl says:

    DM: "Having read some of the justification for this change, I would argue that this was explicitly designed to curtail the "help the applicant" behavior and to refocus the reviewers on scientific merit of the proposal in front of them."

    FWIW, I think you have it right. This was explicitly said by an NIH policy officer at a regional meeting I attended in 2009. The context was a discussion of the new review forms and the upcoming changes to the grant format.

  • baltogirl says:

    I use this section to tell applicants that long paragraphs of underlined italics are extremely difficult to read, and the like. (why not spend a minute helping?)

  • Sara says:

    most of these "research" grants and proposals are nothing more than various permutations of american hustling and opportunism. Reducing anything to a buck. Alexis de Toqueville has reported on this. And Walter A. McDougall.

    America is nothing more than a failed business venture to dupe the hamsters into thinking their "research" actually mean something. Add in some junior profs who are eager beavers, senior narcopaths who are still clutching onto a mediocre tenure cake joke-- and you have a feces show. Well-connected swindlers, ever smiling to acquire more, and more. A question of values. Why american failed by Morris Berman. Most likely this factual info will be deleted, censored, or modified.

Leave a Reply