The new NIH grant summary statement will look like this

Jun 02 2009 Published by under Careerism, Grant Review, NIH

In case you managed to miss it, this round of NIH reviews will use a changed scoring system which we've previously discussed. It will also be associated with an unfamiliar summary statement. For those of us that review we will have to put away our venerable Stock Critique Templates. Fortunately, the Center for Scientific Review has provided a replacement along with a host of other informational documents on the new review system.


The handy-dandy new Stock Critique Template provided by CSR looks like this:
Overall Impact (Please limit text to ¼ page )
Strengths

  • High potential impact in clinically important area of safe blood transfusion.
  • Highly qualified investigators with complementary expertise ensure likely success.
  • Novel application of incident reporting methods now in use in other fields could lead to improved public confidence in blood supply.
  • The study will bring a rigourous, systematic approach to the current error reporting process, which is empiric and lacking in evaluation.

Weaknesses

  • Lack of representation of non-academic transfusion medicine practitioners may make incident reporting less effective in non-academic hospital setting.
  • • Not enough time is allotted for aim one work and aims two and three too dependent on success of aim one work lessens confidence that work can be successfully completed.

SCORED REVIEW CRITERIA
1. Significance (Please limit text to ¼ page)
Strengths

  • An effective incident reporting system should greatly increase confidence of the public in safety of the blood supply.
  • Models developed for other error-critical fields have been effectively adapted in the development of an incident-reporting system for transfusion medicine.
  • Identifies and incorporates limited and appropriate range of human error patterns--will be easily transferable to practice.
  • Could be generally applicable to understanding influence of incentives/disincentives on behavior.

Weaknesses

  • Premise that human error follows limited range of patterns in diverse situations may not hold true.
  • Unclear how incident reporting system would be utilized to reduce human error.
  • Unclear whether public perception or clinical need is target of model application.

2. Investigator(s) (Please limit text to ¼ page)
Strengths

  • Investigators are well recognized for their expertise in blood banking.
  • The PI has a strong publication record in field of HIV analysis in the context of blood transfusions.
  • Collaborators have expertise in error-critical fields and are complementary in expertise to PI.

Weaknesses

  • Levels of effort propsed by investigators are unlikely to be sufficient for scope of project, raising doubts that goals will be accomplished.
  • There is not sufficient expertise in the area of transfusion medicine. Research plan would benefit from input from someone with expertise in collection & transfusion in non-academic hospital setting.

3. Innovation (Please limit text to ¼ page)
Strengths

  • Adaptation of incident reporting systems used in other error-critical fields (aviation, nuclear power, anesthesiology) to transfusion medicine is highly innovative.
  • Use of positive reinforcement to address disincentives associated with self reporting of errors.

Weaknesses
4. Approach (Please limit text to ¼ page)
Strengths

  • The experiments are well designed with appropriate controls proposed.
  • Potential problems are anticipated and alternative approaches are presented.
  • Conceptual framework for the entire project is well developed and supported.
  • Use of ubiquitous computer platform to manage information is sound and generates confidence in the potential expandability of the system across the country.

Weaknesses

  • Time alloted for aim one work is likely to be insufficient.
  • Aims two and three are less well developed because they are dependent on outcome of aim one.
  • It is unclear which test site will provide gold standard against which results from other sites are compared.
  • Test sites are all academic institutions and thus lack real world diversity; hospitals/community health centers should be included so that results are more generalizable.

5. Environment (Please limit text to ¼ page)
Strengths

  • Outstanding scientific environment in each participating institution.
  • Strong commitment of support from each institution.
  • Opinion leaders in transfusion medicine field will participate.

Weaknesses

  • Notable omission is the absence of non-academic hospital settings.
  • Better plan to ensure uniformity across sites is needed.

34 responses so far

  • dajokr says:

    Hah. I'm in the middle of doing mine right now (albeit writing during our lunchtime break - I'd never comment on a blog while the study section meeting is ongoing). I'll be interested to see if this more concise, bulleted format will make it easier for applicants to respond.
    If readers haven't looked before, I encourage them to pull up the PDF of your Stock Critique Template. I still think that you will be able to modify the template to continue to operate in a time-efficient and meaningful manner within the new format.

  • Luigi says:

    That PDF of stock critiques is hilarious. I wish I had seen it before. Although, of course, it feels like I have.

  • Yeah, that shit is motherfucking hilarious! It's like a parody of a review.

  • qaz says:

    Did anyone else notice that the Challenge grants are limited to one page of references? Apparently some people just truncated their reference pages. So I'm seeing Challenge grants with hundreds of references in the grant itself and only one page of maybe twenty citations.
    I can't figure out where in the form I should put "Authors did not follow directions."

  • Luigi says:

    qaz: Talk to the SRA* Officially, that sort of stuff should have been caught by administrators and been grounds for rejection without review. Granted, it's impractical to normally expect that, and a harsh result. But given the number of applications, I think reasonable on both counts.
    I once had a reviewer chastise me for 'not filling out the forms carefully' in the summary statement. Fair enough, except my forms were actually correct; the reviewer had taken the 'liberty' of checking up on some stuff online and made a mistake!
    *Who will probably just tell you to decide for yourself how that affects your opinion of the proposal. In which case you are back to square one.

  • qaz says:

    *Who will probably just tell you to decide for yourself how that affects your opinion of the proposal. In which case you are back to square one.

    Give the man a gold star! I'd already checked with the SRA about this. And that's exactly what he said.

  • DrugMonkey says:

    qaz, this is just exactly like the old days when people would push all the font requirements. I never heard of that being caught by receipt and referral. Everything I have ever been able to glean from fellow reviewers is that the SROs leave it up to you to make a stink about it. And who is going to do that? And if you catch it early, the worst that happened was the PI had to quickly provide a fixed version.
    I've only heard of one case, second hand, where a colleague claims another dude on the section intentionally didn't tell the SRO and brought it up at the discussion expressly so the PI could not send in a fixed, font-kosher version. Assuming this actually happened, you can imagine that this guy was seen more as a dickweed than as a great guy, saving all and sundry from apps which cheated on the rules.
    Nowadays you still see this with violations of the rules on what can be included on the biosketch (i.e., I see conference presentations and "in prep" listed quite frequently), etc.

  • DrugMonkey says:

    ...and just to belabor the obvious, as is my wont, this is yet another one of those little areas in which study section experience helps your own grant writing. You see just which of the rules can be fudged, should they seem to be of advantage to you.

  • Luigi says:

    (i.e., I see conference presentations and "in prep" listed quite frequently), etc.

    ...and just to belabor the obvious, as is my wont, this is yet another one of those little areas in which study section experience helps your own grant writing. You see just which of the rules can be fudged, should they seem to be of advantage to you.

    But seriously, what kind of moron thinks conference presentations are going to help their application? Applicants might as well list perfect attendance for grade school. Along those lines, is anyone else annoyed by zillions of teeny Honors and Awards you've never heard of, like the 'John T. and Catherine Peters Award for Research Excellence'. For all I know, that could be some bogus birthday card from the applicant's maternal grandparents. And even some of the 'real' versions of this are not that different -- cozy CV-stuffing reach arounds by a select set of (typically) ivy league wankers. Does this crap really fool anybody?

  • But seriously, what kind of moron thinks conference presentations are going to help their application?

    Not that I don't think you are a foul repulsive douchebucket, but you are right on this one. When I see conference posters or presentations on a CV, it just annoys me that I have wasted seconds of my time scanning past that garbage. Nobody gives a shit about that crap. Not that it isn't good to *do* that shit, but no one needs to read about it on you CV.

  • Mike_F says:

    "But seriously, what kind of moron thinks conference presentations are going to help their application?"
    In mathematics, computer science and some areas of physics, the more prestigious form of scientific communication is certain types of conference presentations. Typically participation in such conferences is limited and competitive by blind or double-blind review of extended abstracts (these can be 2-4 pages in some cases, so almost the length of a Nature/Science paper). When reviewing tenure/promotion files from these faculties, the candidates get more credit for such reviewed conferences than for regular published papers. Obviously this is not relevant for many NIH applications, but if you are seeing grants from systems biology people coming from a 'hard sciences' background, you might want to ask a colleague from such a field to take a closer look at the conference proceedings list in the cv.

  • JP says:

    I'm in agreement with Mike_F above. Engineering is another field where conference papers/presentations can be significant. Of course, one has to weed out the garbage conferences from the prestigious ones, but the same has to be done for journals.
    So, if you're looking at proposals in bioengineering, conference presentations might be more important than you think.

  • The RC1s and RC2 also had other special instructions like omitting background and Significance and Prelim data sections.
    Just out of curiosity, if someone did follow all the instructions, spent some time crafting a proposal to fit the requirements (including writing it and citing it such that the references would fit one page), would he/she get any consideration above proposals that ignored said guidelines (other things being relatively equal, let's say)? Or does it come down to BSDs getting a pass and stupid naive n00Bs essentially getting laughed at in review sections for coloring within the lines?

  • qaz says:

    Anonymoustache #13 -
    1. Just because you are supposed to omit certain sections, doesn't mean that the information shouldn't be in there. I've always been taught that when filling out a form (and what is a proposal but a form with a secret/unknown format), that you need to find a way to get the information you want into that form. In large part, the difference between a senior person (what's a BSD?) and a n00b is that the senior person knows how to put the important stuff into the form. (Of course, now the form has changed...)
    2. I don't know about other reviewers, but I'm certainly getting ticked off about several of the proposals that did not follow directions (like the one that has numbered citations to >150 and 15 listed on the one page of references [from Anne Aardvark to Amy Appleton]). Getting your reviewer ticked off is never going to help your proposal.

  • Luigi says:

    Ah, now I guess we know what kind of moron lists conference proceedings. Thanks Mike & JP; that's useful to know.

  • qaz #14:
    I understand that the requisite background info needs to be in there---you need to provide appropriate context for a proposal. I also understand that with the special instructions, it becomes a bit more difficult to do that---one doesn't get a couple of pages for background and a couple for prelim data. Hence my question---some of us took some time to craft it accordingly and stay within the lines. It would suck if we lost out to others who got to state their case more expansively by breaking/ignoring the rules. More words don't necessarily mean better proposals but if you're talking 12 pages versus 15, the extra three could make a world of difference.
    Now, if there are many proposals that ignored the rules, then it stands to reason that a not-insignificant portion of the reviewer community will also be composed of rule flouters. Somehow I don't think they'll be sticklers for enforcement. Throw in some percentage of wink-wink-nudge-nudge-old-chap and soon it may be that the n00bs who stayed within the lines end up looking like idiots.
    BTW, BSD is an acronym for the *ahem* well endowed faculty --- grants-wise of course, what'd you think? I believe it stands for Big Swinging Directcosts or something like that.

  • microfool says:

    With regards to conference presentations, I know of one recent example where noting such activities on one's CV played into administrative grant extensions. Such activities were used as a proxy for scientific community impact and involvement. These were relatively early career scientists nominated for their substantial scientific impact, and only about a third received the extensions. Listing these activities or not wasn't a deal maker/breaker, but may have affected who was above and below the line.
    Obviously, I defer to CPP and others who have actual experience in study sections, faculty hiring, etc but I have to wonder that if there is no space limitation, or strict rules against the information, should one include such activities on a general audience CV?

  • qaz says:

    Anonymoustache #16 - I agree completely. As someone who spent a lot of time and effort making sure that I followed the rules (one page of references, specific sections fitting into space requirements, etc), I'm less than sympathetic to rule-flouters. I suppose we'll have to see, but the last time I was on study section, we had one application that was a rule-flouter that got past the SRA and no one was sympathetic to the application.
    There's always this weird conflict in study section between "that would be a good project if funded" and "this is a well-written grant". Somehow, you have to balance the two issues. In particular, it comes into conflict when you have an obviously good project with someone who clearly just tossed off the grant. (For example, it's full of spelling errors, or the references are all jumbled up, but the science would be excellent.) I suppose that given the very very small funding percentage for challenge grants, funded proposals will be both. But we'll see.
    PS. I have two that flouted the reference rule, one by a senior "BSD" and one by a n00b, so at least with my limited observations, it's irrespective of that.

  • As long as we're on this topic, for the love of fucking god, do *not* use numbered citations in your grant applications. Use the (first author, year) citation format, and then list the references alphabetically. Reviewers want to know what you're citing, and with numbered citations they have to page back to the reference list every motherfucking time. If you use the (first author, year) citation form, most of the time the reviewer will recognize the reference and not have to page back.
    Yeah, you lose some space for more of your pellucid prose, but the annoyance you induce in the reviewer far outweighs this piddly shit benefit.

  • Qaz:
    ---"In particular, it comes into conflict when you have an obviously good project with someone who clearly just tossed off the grant."---
    That's precisely what I figured. Of course, we want to fund the best science (and I know that a bunch of criteria go into that) but what's the point in having the rules if you cannot invalidate even a good scientific project if it is presented in a way that it blatantly flouts the guidelines? It's not like there's a dearth of good projects to fund, right? Also, if the 'shoddy' proposal is indeed about great shit, then it will probably get funded next time around when the applicants can be bothered to give a shit.
    I can understand about small errors or oversight etc, but in the case of the RCs where even the organization of the research plan is different and pretty well laid out for ya, how much flouting do you take? I guess that will vary on an individual basis with respect to reviewers.
    Anyway, I figured that there was a method behind the madness for the RCs and their special instructions---which is that these are big picture, big idea awards. If you need 10 pages of arcana by way of background or references to legitimize or bolster your claim, this mechanism is probably not for you. Why not enforce that with a slightly extended philosophy too---if you can't see the program for what it is and tailor your proposal accordingly, then this program is probably not for you?
    Also, if someone couldn't be bothered make a reasonably good effort to read or follow the instructions for proposal, what's the guarantee he/she'd bother to follow other NIH instructions and guidelines if the award is made?

  • DrugMonkey says:

    First, AM, settle down. I inadvertently gave the impression that *all* experienced grantsmiths were flouting the rules regularly- not so. All I'm saying is that it happens frequently enough for me to see this as a bit of an issue.
    As microfool alludes to, sometimes the rules against manuscripts in prep and conference presentations (and I mean the bioscience kind, not the computer/physics kind) will leave an applicant looking like they have done nothing. This could be because they are very new or because their model is a lengthy longitudinal one. We handle a fair bit of human subjects stuff in my section, for example. Often times those project go on for a long time without apparent publication output but then it all comes out in a rush at the end as the final subjects are enrolled in the study.
    If you are doing a competing continuation app, or if you have been previously busted for lack of productivity...well, you have to do what you can to look like you've been working. If that information hasn't been provided in the app, then you have to rely on random luck that the reviewer in question has been following your work at meetings.
    CPP on numbered references- hellz yes! Do. Not. Use. Numbered. Refs. EVAH!!!!!!!!!
    microfool- As we've discussed before, you have multiple versions of your "CV" depending on use. I think that at least one of these should be what I call the Full Monty CV that has every little thing you have ever done related to your career on it. Eventually you are going to forget stuff. This should be your main updated one because it is simple just to take that and cut it down for various other purposes.

  • Cashmoney says:

    Also, if someone couldn't be bothered make a reasonably good effort to read or follow the instructions for proposal, what's the guarantee he/she'd bother to follow other NIH instructions and guidelines if the award is made?
    Because just about any of these that matter are done institutionally at most research Universities? With a research grant, there is very little the PI has to remember. If you spend your money out of category or try to spend it inappropriately it is the ol' Grants Office that keeps you on the strict and narrow path.

  • Luigi says:

    microfool: There was some good discussion of CV format etc a while back on this blog. Maybe one of our illustrious hosts remembers and can provide a link.
    Comrade: Interesting that you hate numbered references. I use numbered references whenever possible, because I think it aids readability. I like reading things with numbered references for the same reason. I guess this implies that I don't worry too much about what particular paper is being cited. And I don't. Generally I only check the references if a statement raises my eyebrows or I really honest-to-God want to follow up on some information. Who's-citing-whom games annoy me. I just want to hear the argument for something, which either will make sense or not. If one's whole plan revolves around resolving some microdispute between two labs or tying up some irrelevant frayed historical thread then it's already sunk before it begins, in my book. The really best stuff has almost nothing to cite, because it's all really new territory.
    Anonymoustache: The Challenge grants are a trial balloon launched in a hurry because NIH needed to give out a lot of money fast. Reviewing this first batch must be a nightmare I'm glad to not have a part in. Eventually, I think a successful style for this mechanism will soon emerge. Just like styles for other grants have emerged which do or do not exactly follow the format. If it was obvious what to write based on the forms and instructions, guides to grant writing like Comrade has posted here would have no utility. But they do, and Comrade's advice is I think on target. Success is as much about understanding the culture as anything else. Which, I guess, is the central message of this blog (besides its apparent role in indulging some of DM's more unmanly desires for interracial and gender empathy).
    But sort of back on topic -- DM have you circulated your stock critique document at study section? God that would be a bright spot to have something like that slid under the table toward me while Dr. Chamadranahandranamni droned on about his methodological concerns relative to the vastly important technical refinements that he published years ago in Arch. Soc. Neur. Tech. Symp. Abstr.

  • God that would be a bright spot to have something like that slid under the table toward me while Dr. Chamadranahandranamni droned on about his methodological concerns relative to the vastly important technical refinements that he published years ago in Arch. Soc. Neur. Tech. Symp. Abstr.

    Yeah, cause we know all about those Indian scientists with those funny names, don't we?

  • qaz says:

    Actually, the worst problem with reviewing the challenge grants is not the grants themselves or even the structure. (Although I'm still mad about the sloppiness of some of the PIs!) What's driving me crazy is this new very formulaic format for reviews. The issues with each grant are so different that it's hard to find how to put my review into the new format. And, of course, I'm completely uncalibrated on this new 1-9 scale.
    Luigi - On the references, one of the big issues with a lot of grants that I read is determining if the investigators are aware of the work outside their little circle of friends, in particular work across species, labs, family-trees, etc. It's not a "give a shout-out to another lab" but being aware that the issues your wrestling with are larger than your personal history. In my opinion, this is not a grant-sloppiness issue, but a good-science issue. If the investigators are not aware of a portion of the literature, they're not going to be doing good science.
    I've heard arguments for both numbered and author-year. I definitely prefer author-year because I know most of the key papers and can often get away without checking the reference list. But what I *really* hate is citation-order (i.e. non-alphabetical) reference lists since I can't just go into the list and make sure they know the key papers in the field. Well, ok, what I *really* *really* hate are numbered, citation-order, and missing most of the reference pages!

  • Luigi says:

    Yeah, cause we know all about those Indian scientists with those funny names, don't we?

    Huh? Where did that come from? I was trying to onomatopoeiate an old white British guy's throat-clearing mumble.

  • becca says:

    I'm getting an echo of that vague sense of unease I always feel when I watch judges score gymnastics competitions...

  • DrugMonkey says:

    What's driving me crazy is this new very formulaic format for reviews. The issues with each grant are so different that it's hard to find how to put my review into the new format. And, of course, I'm completely uncalibrated on this new 1-9 scale.
    WORD THE FUCK UP!!!!!111!!!!
    /forgetting momentarily that his SRO occasionally reads the blog...yikes!

  • DrugMonkey says:

    I'm getting an echo of that vague sense of unease I always feel when I watch judges score gymnastics competitions...
    Yes, it is exactly like that. It has never been any different and I doubt it will change in the future.

  • antipodean says:

    Finally, CPP provides some concrete scientific advice for referencing in grants (CPP, 2009). I'll remember that for next round. Cheers.
    Luigi- when you get called out on being a racist dickhead after deliberately being a racist dickhead, don't try to pretend you weren't.

  • Luigi says:

    Luigi- when you get called out on being a racist dickhead after deliberately being a racist dickhead, don't try to pretend you weren't.
    But it was a good attempt. At least give me credit for trying.

  • Luigi- when you get called out on being a racist dickhead after deliberately being a racist dickhead, don't try to pretend you weren't.

    He's not actually being a racist dickhead. What he's doing is pushing the envelope, pretending to be a racist dickhead in order to try to get a rise out of people and draw attention to himself. This is the same as when he says creepy weird borderline misogynist shit.
    Frankly, it's pathetic. Normal people who want attention write their own motherfucking blogs. This loser gets his by acting like an asshole at someone else's.

  • Luigi says:

    No, I'm a racist dickhead. I sincerely believe that my white male upbringing influences my decisions. If I wanted to pretend, I'd pretend to be a foulmouthed NIH-funded faculty member at a private medical school. Because THAT is uber kewl.

  • Luigi says:

    Crap. I'm gonna have to change my username again, aren't I?
    Annoying Comrade is getting to be real work!

Leave a Reply