I like the new Summary Statement Format

Nov 10 2009 Published by under Grant Review, NIH

So the new bullet-point NIH grant review format has been in place for two rounds and I am finally hearing a bit of feedback from friends and colleagues. I also have had a chance to be subjected to a nonzero number of reviews as an applicant, instead of only as a reviewer.
Some of the chatter I am hearing reflects confusion.. with a lot of comment that the person (applicant) can't tell how to interpret things. Even more frightening is one report of a Program Officer making the same complaint-after all the new format was supposed to be to help the POs make their decisions! That's not a good thing.
From my small sample, I think it is perfectly fine. I mean, I used to go through the old summary statements with two highlighters- one for positive comments and one for negative comments -in the past. This just distills the process. The comments I've received are no more confusing than in the past and a lot of extraneous nattering has been left off.
I like it.

11 responses so far

  • JAT says:

    I have not met a single applicant who liked the new critique format (except for the ones who got funded as they could not care less what went into the critique...true with the old format). General consensus is that applicant can clearly see what likes and dislikes are because they are listed under strengths and weaknesses. But it is often difficult to get a real sense of why the dislikes when they are sort of in the grey zone... lost in translation type of thing. It is bit like writing a paper with bits and pieces of data put together in a random fashion with no explanation. This makes writing introduction a bit tricky as one can easily interpret wrongly the reasoning behind the dislikes. Personally, I prefer to have more information than I need than not enough if given a choice. I however agree that we can all do without those completely irrelevant blah blahs (e.g., a full regurgitation of the what aims 1,2,3 are...).

  • Neuro-conservative says:

    The new format is ideally suited for reviewers to just cut and paste their favorite StockCritiquesTM into the slots. The old regurgitation blah blah blah at least encouraged the reviewers to tailor their comments specifically to the application at hand.

  • The new format is ideally suited for reviewers to just cut and paste their favorite StockCritiquesTM into the slots. The old regurgitation blah blah blah at least encouraged the reviewers to tailor their comments specifically to the application at hand.

    At least the new format is honest, in the sense that it makes starkly clear that reviewers basically form their judgments George W. Bush style--"I looked the application in the eye and my gut told me it needed to be firebombed"--and then pull reasons out of their asses to justify them.

  • Fizz says:

    We got a comment "xyz is not a suitable cell line for this work". Nothing else! What can I make of that?
    The old format at least had some details which could be worked with. I feel I have to read between the lines in the new format...and then hope for the best.

  • DrugMonkey says:

    yeah but Fizz that's just bad reviewing no matter the format. the present bullet points (and the NIH sample which I posted before, see the link above) allow for a good reviewer to tell you why in a comment like that. conversely, people surely had such unadorned and unexplained comments in statements past...

  • Anonymous says:

    @Fizz,
    In 2002, I had a primary reviewer (well funded even though he was young) telling me that my cell lines were not suitable and proposed that animal models at hand by big guys in my Department were the appropriate ones to use. He gave me reasons for the "inadequacy of the cell lines" but his reasons were deviating and wrong. I believe that he 's risen to full professor with a 1.5 million dollar (several grants).

  • qaz says:

    As a reviewer, I found it much harder to write a good, communicative review using bullet points. I found that it was really hard to communicate subtlety. [That's subtleTY not subtleLY.] (Take what Fizz#4 got this time "xyz is not a suitable cell line for this work". It's quite possible that there are a lot of issues with xyz and this line of work. In the previous format, the reviewer would have been encouraged to write a paragraph about why xyz was not suitable, which would have given Fizz#4 a chance to respond (or maybe even to agree with the reviewer.)]
    One can write any program in any computer language, but it's a LOT easier to write good code in Python or Pascal than in Forth or APL. Format matters. To say "that's just bad reviewing" doesn't solve the issue since we don't know if the reviewer would have been more likely to write a communicative review in the old format. (What we really need is data here. All we've got is anecdote.)
    In my last reviews, I went back to writing paragraphs. I just split them up by the new topics (which is fine).
    I don't have a lot to say as an applicant, the only reviews I got in the new format were written in paragraphs. (Apparently, I'm not the only one who didn't like writing bullet points.)
    It seems to me that the new review format encourages playground-level discussion "xyz is not suitable" "is too" "is not" "is too" "is not" - though I guess with the new A1 limit, we only get one back and forth.

  • The thing I like the most about the new format is that you get feedback on all of the criteria - however formulaic and non-specific it might be - rather than a rambling mess of nonsensical critiques. The bullet points were short, to the point and concentrated more on general areas of concern rather than the fact that Buffer A was not appropriate. My major quibble is that the feedback and comments weren't all sunshine and roses but that's not really anything to do with the new format!

  • JAT says:

    qaz: I found that it was really hard to communicate subtlety. [That's subtleTY not subtleLY.]
    Exactly! I too continue to write in paragraphs and then cut and paste appropriately.
    The new format is a way to get us ready for the complete toss out of "revised" application in the future. Remember the murmurs: Treat every grant like a new grant
    Got one question I hope you guys can help (sorry if this question deviates from the current discussion topic)
    As NIH is making a rule that if one fails after A1, the application must be submitted as new to a point that it is not recognizable as the old. My question is how would a person do this if the A1 just misses the payline? That is, the A1 is pretty much in an outstanding range under current budget outlook. How does one make an outstanding grant to look "new" , defined as new objectives and specific aims? I asked many colleagues and even some NIH staff...all I got was a smirk.

  • Another biomedical researcher says:

    @JAT It's the "just-miss" A1s that are completely screwed under the new format that prevents further revision and requires a significantly changed application to qualify as new. CSR is being very vigilant about requiring very substantial changes and rejecting applications that are insufficiently modified. I submitted a "near-miss" A1 with one specific aim and part of another ripped out, with a new specific aim and subaim added, but kept the strongest portion intact (with wording changes in the titles/specific aims/abstracts) and had the application removed administratively by CSR. I appealed (delineating how it was 50% changed in aims), they said "sorry" without giving me any guidance about how much more needed to be changed for the application to qualify as new (I asked, they gave the notice URL as an unhelpful response).
    The whim of CSR is a real danger here, and I think that the grant I eventually submitted was overall weaker than the earlier one that was pulled by CSR. Subsequent review confirmed my thoughts on this, so several of those removed parts are being put back in (since they won't check an A1, and the additions were requested by reviewers, as anticipated).

  • JAT says:

    My goodness. What a colossal misjustice. In my view, anything within 5 percentile or so away from payline (12% on average for RO1) would be extremely hard to revamp to look unlike the old one, especially if there are no real "major concerns". And NIH is supposedly to get the playing field even for all? In your case, it is like getting to A2 except now it takes an extra round for you. I hope you get it the next time. Maybe getting a grant writer is not such an off-beat idea.

Leave a Reply