The Great Zerhouni Announces Implementation Phase

The NIH Director Elias Zerhouni issued a recent press release on the implementation of new "enhancements" of peer review. No kidding this time.

"As we contemplated possible changes, we were guided by several fundamental principles. First, while improving the system, do no harm. That is, ensure that any changes to the peer review system bring significant value and outweigh costs," said Zerhouni. "Second, continue to maximize the freedom of scientists to pursue high-risk, high-impact research. Moreover, we want to cultivate a sense that we continuously re-evaluate the peer review system to ensure that it is the best that it can be."

Oh boy.


The Enhancing Peer Review at NIH splash page currently has a nifty timeline graphic.

The 6/6/08 report to the Advisory Council of the Director is here in pdf form.
Okay, none of this will be startling news to Readers who have been following along for the past year or so. Nevertheless, the action plan has been Issued!

The Implementation Plan Report consists of four main priorities and highlights include:
* Priority 1 -- Engage the Best Reviewers: Increase flexibility of service, formally acknowledge reviewer efforts, further compensate time and effort, and enhance and standardize training
* Priority 2 -- Improve Quality and Transparency of Reviews: Shorten and redesign applications to highlight impact and to allow alignment of the application, review and summary statement with five explicit review criteria, and modify the rating system
* Priority 3 -- Ensure Balanced and Fair Reviews Across Scientific Fields and Career Stages
o Support a minimum number of early stage investigators and investigators new to NIH, and emphasize retrospective accomplishments of experienced investigators
o Encourage and expand the Transformative Research Pathway
o Create a new investigator-initiated Transformative R01 Award program funded within the NIH Roadmap with an intended commitment of a minimum of $250 million over five years
o Continue the commitment of -- and possibly expand the use of -- the Pioneer, EUREKA, and New Innovator Awards. NIH will invest at least $750 million in these three programs over the next 5 years.
o Reduce the burden of multiple rounds of resubmission for the same application, especially for highly meritorious applications
* Priority 4 -- Develop a Permanent Process for Continuous Review of Peer Review

Sadly, I'm mired in grant reviewing stuff and preparing for a meeting so I'll have to do some diagnosis later. "Early stage investigators" is intriguing but wtf is up with "retrospective accomplishments of experience investigators"??? We're going to bloody enshrine the "gee this research plan is for garbage but dang, Prof Bluehair and Dr. Greybeard have done such great stuff that we're just going to fund them anyway"? Is THAT what this is about?
In the meantime you may visit
writedit for explication of some of the steps.

Here, the goal is to reduce resubmissions both from applicants with a high likelihood of funding based on their A0 review (hallelujah!) and from applicants with low or no likelihood of funding based on their A0 review (thank you straight talk express). This component also seeks to "rebalance success rates among A0, A1, and A2 submissions to increase system efficiency" and to include statistics on cumulative success rates as a function of score or percentile in the summary statement.

I swear there are times when I think these people are actually reading my blog!

On the issue of percent effort, an alternative approach to requiring a minimum percent effort is suggested: applicants would be required to complete a subfield in the Environment section of the application in which they indicate if they have NIH RPG (research project grant) support in excess of $1M at time of anticipated funding. If so, they must justify why additional resources are needed.

I might actually arrive at the figure of $1M in direct costs myself, if pressed to give a number.
You may also wish to visit with Jake who, in his usual naivety, notes:

I think that the bit about shortening grants and paying people to be regular attendants at study section (the meeting of peer reviewers where the assign scores to each of the grants) will certainly improve the caliber of reviewers. It would also be a great idea if you could do these reviews from afar rather than heading down to Washington. (Do they do them in places besides Washington? I don't actually know.)

Sigh. Got any evidence that this will "improve the caliber of reviewers" there buddy? Evidence there is something wrong at present and how this will address any problems? Or are you just parroting the usual lines unthinkingly?
And seriously. You blog, man. Do you really think that substituting online discussion formats for face to face meetings is a good move? Really?

On the other hand, I do understand the resistance to labeling all grants as "new." It really depends on why you think most grants aren't funded the first time. If you think that is because they weren't very good to begin with, you should be in favor of labeling all grants as new so as to more rapidly identify the best grants and get rid of the rest. On the other hand, if you think that a lot of grants aren't funded the first go because they A) weren't written well but have fundamentally good ideas or B) because funding is so limited, you need that "resubmitted" label to help your chances for the second or third go-around.

The thought is incomplete but I think he was suggesting if you think B is true, you should be against labeling all grants as "new". This overlooks the fundamental problem at hand in that B is the problem itself, not the solution to a problem. Similarly, the fact that we obsess too much over whether grants expressing fundamentally good ideas are presented in a well-written packages is the problem this is trying to address. My hope would be that this modification would be used as a tool to try to force reviewers (and Program!) to a fish-or-cut-bait strategy on proposals. Is it fundamentally of interest and sound? Will the review/revise process change anything fundamental or is it just wasting everyone's time? Those are the questions.

10 responses so far

  • PhysioProf says:

    Are you sure the $1,000,000 limit is direct only, and not total costs? I assume it is direct, but it doesn't really say. BTW, are these fuckers gonna release some kind of real written report with real implementation details, and not just a fucking Powerpoint?
    BTW, there are a few fascinating historical statistical analysis slides that I might blog on if I feel like it.

  • Neuro-conservative says:

    Maybe applications will now be reduced to 12 powerpoint slides.

  • * Priority 3 -- Ensure Balanced and Fair Reviews Across Scientific Fields and Career Stages
    The comedic element would have been significantly improved had it read "Fair and Balanced" instead.

  • M. says:

    On the other hand, I do understand the resistance to labeling all grants as "new." It really depends on why you think most grants aren't funded the first time.
    I'm wondering at the lack of the reverse question: why are grants funded the second time around? I'm also wondering if people at NIH are blind to the problems this can cause.
    This means that a starting/mid-level researcher has even less chance of keeping the funding even if they get it in the first place. Fine, one can say, if they can't cut it, it's better to get rid of them before they waste even more money.
    However, aren't we forgetting that the PI is only one person in the lab? If labs can't keep their funding, the already difficult position of postdocs becomes even less tenable, as they now have to worry even more about when they will have to job-hunt again, and whether they'll manage to get the publications out before the lab folds. Graduate students and technicians would be affected as well.
    Don't get me wrong - I would support good policies that intend to cut the gravy train of silverbacks with decades-long history of automatic renewals and without a publication history of any real interest. But these goals seem explicitly targeted at not affecting the silverbacks (the "retrospective accomplishments" clause), so the whole point there becomes moot.
    As it already is, I'm seeing many of our best postdocs and young scientists moving away from basic research, and seeking greener pastures - nursing, medicine, patent law, or in the best case industry. Making their future job prospects even less stable then they are now doesn't strike me as the smartest way to go.

  • Lab Lemming says:

    This is the sort of thing that makes me happy I work in industry now.

  • DrugMonkey says:

    taking a few more minutes to go through the pdf report....schweeee! Take a gander at pages 25-29. The pages go into some data on the whole revision thing that I've not seen or been able to generate myself yet. I think it backs up my contention about needing to get serious at the A0 stage, to realize a large number of nonfunded A0 apps will eventually get funded and figure out some way to halt the revision-churning.

  • Neuro-conservative says:

    Slide #32 is surprising -- what do you think is GZ's message or intended change?

  • bsci says:

    For those with some time on your hands, you can also watch the actual NIH Advisory Committee meeting with people discussing the plan at:
    http://videocast.nih.gov/Summary.asp?File=14542
    The peer review discussion was only part of the meeting. They didn't keep on schedule, but the relative order is at:
    http://acd.od.nih.gov/agendas/06-06-2008.pdf

  • DrugMonkey says:

    Slide #32 is surprising -- what do you think is GZ's message or intended change?
    We'd have to go to the videocast and I haven't had time yet. Some of the initial mutterings give mixed messages because each time something is said suggesting the NIH is going to punish soft-money institutions into providing some hard money support, they undercut it with weasel language. so who knows where this is really headed.
    It is interesting to see the proportion of NIH investigators who are totally soft-money. Smaller than I would have thought but perhaps a better analysis would have been to ask what fraction of total grant award dollars are held by each of those %-effort categories.
    Also, since this is only limited to NIH funds, there is a big window for NSF, DoD, state, local, foundation, etc funding to be making up the difference for many of those in less than 100% NIH support categories.

  • Neuro-conservative says:

    On the other hand, some PIs presumably have some of their effort on grants (up to even 100%) that does not include salary support. This is common for the hard money senior PIs at my institution. So there is not a 1to-1 correspondence of the data in that slide to soft money status (unless the slide was only including salaried effort, which I doubt).

Leave a Reply