NIH Review of Peer Review Process Finally Samples Current Reviewers

Colleagues who have served on NIH study sections in recent rounds are reporting being surveyed by the NIH/CSR process which has been evaluating grant review. The NIH Peer Review Advisory Committee (PRAC) links are a good place to start if you want to catch up. Or you can start with some posts on the old blog.
It is no secret that I think it idiotic that so much of this breast beating about the "broken" review system has preceded without obvious and significant input from current reviewers. Well, now they are soliciting input.


From my sources, one Don Schneider has been emailing a request for comment on study sections. (lightly edited for anonymity purposes)

You have attended recent meeting(s) of the Bunny Hopper study section. Since the NIH Center for Scientific Review believes in periodic assessment and continuous adjustment in response to changing science, we are using a Working Group Process to examine how well study sections are functioning. In this process, distinguished reviewers like you are asked to comment in confidence on study section operations. Small samplings of such opinions are pooled and summarized, and the summary report (free of reviewer identifiers) is discussed with the collective chairs. The summary report is also shared with the NIH Peer Review Advisory Committee.
Would you please help with this process by sending me your thoughts on the functioning of the study section. To guide your response, we are interested in the following areas:
Scientific
· Scope and breadth of the science reviewed
· Accommodation of new directions and emerging areas
· Appropriateness, qualifications, stature and diversity of reviewers
Operational
· Management of the meeting
· Policies and procedures
· Other emerging peer review issues
To facilitate the processing of information on our end, a reply by October 1 would be most helpful.
Thanks very much for your consideration.

Looks like a good plan to me. Hope they get enthusiastic participation from reviewers. You know, reviewers who have just received their pile of applications to review for this round. ...sigh.
So, DearReader, if you have reviewed for NIH in the past couple of rounds and have received this query, please respond. Take the few minutes and jot down some thoughts. Reviewer voices need to be heard in this process.

8 responses so far

  • PhysioProf says:

    This shit is just pure theater, and doesn't mean jack diddly shit for actual peer review.

  • Lorax says:

    I expect this is primarily for permanent members, but if I am contacted for my ad hoc duties, Ill reply. I disagree with PP, that philosophy is similar to the undergraduate class evaluations we do every year. Many students think the evaluations do "jack diddly shit" as well and then wonder why some classes never improve. Its the same rationale given for not voting too. I do not expect NIH to come to a schreeching halt when comments come in, but I do expect NIh to take the comments seriously. Case in point, when asked for comment on the proposed changes in R01 grant formats, I was happy to see several things I suggested in the final list of issues to deal with (I doubt my responses had much to do with it, but my concerns + those 100s - 1000s of similarly expressed concerns were heard).

  • BKProf says:

    Okay, I'm not sure why they're still soliciting advice. I am at a scientific conference where on Saturday, CSR head Tony Scarpa gave us a rundown on the changes in peer review to be implemented as soon as February. Apparently, NIH director Elias Zerhouni had just approved these changes the day before, so this is the latest information and hasn't yet been widely announced. Many of the changes were not a surprise. . .changing the numerical scoring system, changing reviews to focus more on impact, eventually shortening the applications (although this won't be implemented right away). But what has everyone here scratching their heads is the proposal to do away with A2 applications, so we'll all get only two chances to submit. The thinking behind this proposal is this: NIH has determined that all A0 applications that rank in the top 25% eventually have a 100% success rate. Therefore, why not fund the really strong applications earlier, instead of making the investigators essentially wait in a queue? All the attendees I spoke to here are concerned about this. I'm a regular study section member, and I have often seen grants that did not start out as high scoring A0s, but eventually became very strong after three attempts. I fear that by limiting submissions to two tries, we may be preventing funding of good science by individuals who may simply lack really strong grantsmanship skills (especially new investigators). NIH will have to be very careful in the way they implement this change. Unfortunately, they have not yet worked out the details (or at least Tony Scarpa wouldn't tell us if they have), so it's hard to know what will happen.

  • becca says:

    So, BKProf, call up NIH and tell them to not do away with A2s for young investigators (anyone who has never had independent NIH funding, or within the first 5 years of it, say).

  • BKProf says:

    Becca, I heartily agree with your idea, and I will certainly give my opinion to the NIH, but I have absolutely no influence on what they will do. I'm just a poor schlub who reads grants for them. Without going into details, I know of at least one person within NIH who changed positions because they felt they could not support the proposed changes. Even NIH staff have minimal power to alter these policies established on high.

  • neurolover says:

    "call up NIH and tell them to not do away with A2s for young investigators "
    I think the key here is that if NIH wants to keep new investigators alive, they need to get seriously into the mentoring business. They need to help the folks who "lack grantsmanship skills, who don't know how to respond to criticism, who don't know what 'I don't see how to fix this problem' really means, who don't know that you can just as your SRO a question and they'll answer, if they can". Of course, one is supposed to get that assistance from one's own mentors, at one's university, but as the system gets more competitive and tight and people, even people who really want to help, have less time to do these things, and a variety of people tugging for their help.
    This problem strikes people who are "different" more strongly -- case in point, a young woman really can't just go up to old guys in her field and ask them for drinks, or at least, it takes a pretty non-standard personality to do so (or to even accept when asked). That means women are always a little bit outside the system (interestingly, I note that some women who make the leap are partnered with people in the system (FSP comes to mind). This imposes significant barriers, but once you reach a certain mid-level, it can also help.
    So, formal mentoring -- it's a big big deal.

  • DrugMonkey says:

    They need to help the folks who "lack grantsmanship skills, who don't know how to respond to criticism, who don't know what 'I don't see how to fix this problem' really means, who don't know that you can just as your SRO a question and they'll answer, if they can".
    In my view this is not even the first problem that needs fixing. I see grants that are already great being dinged into the non-fundable zone essentially for no reason other than that the PI is younger and supposedly less proven. Great grant apps, in fact. While pretty bad grant applications from senior figures skate into the fundable zone on the strength of supposed track record. These are undoubtedly the New Investigator proposals that Program Staff have been picking up lately- just watch, I bet they do just fine with their funding.

  • neurlover says:

    "I see grants that are already great being dinged into the non-fundable zone essentially for no reason other than that the PI is younger and supposedly less proven."
    bad, I guess
    "These are undoubtedly the New Investigator proposals that Program Staff have been picking up lately- just watch, I bet they do just fine with their funding."
    good, if true -- especially if "they do just fine with their funding." The problem with these "pickup" ideas has always been if they're protecting people from the real competition early on, only to see them falter later on. But, I guess if they're "early money is like yeast" and helps means success, well that'll be great.
    But, how do the program officers know to pickup these grants? Do people actually say "I'm dinging this person because they're naive and inexperienced"? Is there conflict between reviewers?

Leave a Reply