More thoughts on the new bullet point NIH summary statement

Jun 08 2009 Published by under Careerism, Grant Review, Grantsmanship, NIH

The change to a very brief, bullet point format (sample of old narrative version) for the written critique of NIH grant applications will have many effects, bad and good. One I've been pondering recently has to do with the review of applications for which a previous version has been reviewed. This could be for an A1 revision of a grant application which did not receive a fundable score the first time it was submitted or for a competing continuation of a grant which has already been funded.
This change of format is going to make these reviews really, really difficult.


As I have previously noted, the review of such applications takes place on the basis of incomplete information because the reviewer does not have access to the prior grant application (funded or not).

The reviewer of the amended application is provided with the most recent summary statement and the revised application. Period. There is no provision of the actual prior application(s) or prior summary statements (for an A2 revision) and certainly not the prior score ranges, the identity of the prior reviewers or anything else.
The first and biggest problem is the failure to provide the prior application to the reviewer to facilitate assessment of "responsiveness" to the prior critique. This means that a new reviewer for the application is basically guessing on the basis of the summary statement what was contained in the prior application.

In the past, there was a good chance that the prior critique would provide enough overview of the application that you had a good idea what was in it. Now this will not be the case, there will just be evaluation limited to the 2-3 most important points.
It is going to be impossible to tell whether a competing continuation application represents new directions from the original, whether the project did anything related to what it said it would, whether it exceeded the prior goals, whether the PI heeded any of the critical comments after getting funded, etc. It will be impossible to tell if a revised application actually represents any sort of revision or not.
This may be good or it might be bad, there are certainly things about this aspect of review that I think have significant positive and negative effects on the process. The one certainty is that it sure as heck is going to be difficult for reviewers to meet the responsibility to evaluate certain issues with competing continuation and revised applications.

3 responses so far

  • Danny O'Rerio says:

    I think it's a change for the better. One thing that's always bothered me is the tendency to give good scores not so much on the goodness of the application in front of you, but the fact that the applicants managed to jump through all the requested hoops. It's kind of like a subliminal contract gets set up whereby reviewers say what they want, and then feel guilty about not giving good scores when the applicants live up to their end of the deal. The same thing happens when papers are reviewed, and leads to the same problem -- reviewers find stuff to suggest just because they think they should, and then eventually give a thumbs up that indicates effort over value of the end product.
    Basically, what I'm trying to say is that the history of a grant should not matter. Just judge the proposal in front of you. If the authors got good advice in previous rounds and took it, it should be a good proposal. If they didn't, then maybe it won't be. But whether they did or did not take any advice should have nothing to do with whether the proposal should get funded.

  • It is going to be impossible to tell...whether the project did anything related to what it said it would[.]

    WOOHOO!

  • Ben says:

    I liked the old system better; both as a reviewer and as an applicant, there was a sense of the potential to improve an application each round. This feels more arbitrary.

Leave a Reply