On recruiting peers to review NIH grants

Jun 30 2015 Published by under NIH, NIH Careerism

May 2015 Advisory Council round for the CSR of NIH.

I'll be making observations on the Luci Roberts presentation in a little while. For now, enjoy.

UPDATE:

Okay, down to business. The part I wanted to highlight starts at 1:35 of the videocast. Dr. Nakamura introduces Luci Roberts, Director of Planning and Evaluation in the OER.

A comment from baltogirl

It is a little known fact that SROs have trouble recruiting for study section (I was told that two-thirds of people asked decline to serve). It's likely that most people are so busy writing their own grants they can't take a full month off to devote to reading the work of others.

reminded me I forgot to discuss these data.

The Division of Planning and Evaluation conducted a survey** some time ago, trying to determine the attitudes of extramural PIs that would affect their willingness to serve on study sections. I thought it applied to the above comment.
They surveyed 4,000 individuals who had submitted at least one grant as PI in the past five years and who had obtained active funding (any variety) from the NIH in the past five years. 1830 or 46% responded. Not too shabby. (They also surveyed 423 SROs of which 271/64% responded, more on that later.) Of the PIs 1,616 had served as PIs or PDs, the balance were TG directors or subproject/consortium PIs.

First up. 964 (53%) had never been asked to serve on study section in the 12 months prior to the survey. Hmm. I can almost stop right here. But no....there's much more. Still, if there is a reviewer crunch, the first order of business is to determine why over half the potential pool is not even being asked.

Ok, of the remaining 866 individuals who were asked to serve on study section, 762 agreed to do so. That's 88% saying yes. So the rumor that "two-thirds of people asked decline to serve" is falsified by this survey. Clearly, the vast majority of people who are asked step up and do their community duty.

I really, really like this. It is heartening.

The next most-interesting thing was the 10th slide which shows the ranks of PIs who are asked/not asked to review. As you would expect, the Full/Associate/Assistant Professor ranks for those asked ran 54%/32%/11% (that high for Assistants?) and 22%/23%/37% for those not asked to serve.

Again, this outcome makes it really clear what needs to be done if getting reviewers is a problem. Ask more Assistant Professors to serve***. Right? Done.

Then we get into a couple of slides on why people might say "no" when asked to review. Slide 11 present the top reasons (out of an open text box response, per Roberts' presentation) for the process being "more burdensome than it could be". The numbers are confused here because the denominator appears to be 861 when it should be 762. But in any case, 630 of the respondents who reviewed said the process was not more burdensome than necessary (huh? surprised on this one). Of the 158 who said it was too burdensome, 45% complained about the number of assigned applications. The next most common (16%) complaint was "too much time devoted to applications that will never fund". So pretty much, the most burdensome thing was review load.

This brings us to Slide 13 which pits SRO opinion versus how reviewers think. One of the biggest disconnects was in the number of in-person meetings per year that is "reasonable" since reviewers lean 1-2 and SROs were about evenly split between 1-2 and 3-4 as okay. A similar disconnect was found on application load. Half of the reviewers felt 4-6 apps was a reasonable load and only 25% felt 7-9 was reasonable. SROs leaned 7-9 (~60% of SROs) with less than 40% finding a 4-6 grant load reasonable.

Predictably.

I'm skipping Slide 12 on reviewer and SRO thoughts on reason to accept and decline invitations to review since 88% of those asked say yes anyway. Who cares what they say if most of them will do it just for the asking?

Okay, those were the things that jumped out at me.

___
*As a disclosure, Dr. Roberts was once a SRO who played a highly formative role in the early-career understanding of the NIH grant business for YHN. She was kind enough to send me the slide deck that was used in her presentation.

**I was a respondent, fwiw.

***In case of any newcomers, both YHN and CPP have advocated for this on this blog since forever.

16 responses so far

  • DJMH says:

    How is it that they can't get 95% SRO responses to their surveys? The SROs work for them..

  • Philapodia says:

    Our department can't get over 50% responses from our faculty, either.

  • profduder says:

    From Nakamura's presentation at 2:28:00 in the video
    http://i.imgur.com/gft9hWh.png

    Please explain.

  • Philapodia says:

    "From Nakamura's presentation at 2:28:00 in the video
    http://i.imgur.com/gft9hWh.png"

    The proposals that get funded (>20th percentile) are the ones that are getting publications. Because they have the money to do the work. The riff-raff who have to write 6+ grants a year to keep funded are likely less productive because they are writing 6+ grants a year and not papers. Likely the people who are getting 1st-5th percentile on the steep slope know how the system works and only submit a grant or two a year to friendly study sections. It would be interesting to correlate this with PI age or the number of post-docs in the lab.

    I'm surprised that 90th percentile grants are getting funded and can publish anything, but apparently there are a few...

  • profduder says:

    Is the graph saying that 90th percentile grants are equivalent to 10th percentile grants in terms of citations "residual number of citation"?

  • A Salty Scientist says:

    I have no idea why would there would be sharp inflection upwards at the ~60th percentile for the citations. Even if only BSDs with crappy grants were being picked up at those percentiles, I can't imagine why the 90th percentile would be better than the 70th. This seems like a problem with how they processed the data, because it does not jive with the previous figure.

  • Juan Lopez says:

    "Clearly, the vast majority of people who are asked step up and do their community duty."

    The vast majority of THOSE WHO RESPONDED TO THE SURVEY said that they would also review. Perhaps those with no interest in reviewing are also not doing surveys? Or those who are too selfish to respond might not participate as much.

  • physioprof says:

    You could have at least thanked me for parsing all this shittio out for you, you ungrateful fuckebagge.

  • drugmonkey says:

    I don't know why you imagine that was your unique insight Holmes. And if I recall you still have posting privileges.

  • drugmonkey says:

    Fair point JL. Asked by an advisory board member and Roberts begged off saying they were loathe to annoy their reviewer pool by hounding them. Which is a fair excuse.

  • girlparts says:

    I had to beg an SRO to let me on study section, and then beg my peeps on study section to beg the SRO before I finally got invited. Interestingly, though, what the SRO finally said was, "you've been on my list, but you keep submitting proposals. You have to assure me you will not submit a grant to us this cycle, and I will invite you." Probably lots of people in the same boat.

  • Dr Becca says:

    girlparts, is there really only one study section that's appropriate to your expertise? I applied through the Early Career Reviewer program and got invited not to the study section that most of my proposals had gone to, but to one with substantial overlap, so there was no need to sit out a round. It was very eye-opening, and now I know of another SS that I could send my proposals to.

  • girlparts says:

    Thanks, Dr. Becca. Once I became aware that was the problem, I did manage to target my grant to a related study section that I had been considering anyway. It just hadn't occurred to me, somehow.

  • Mid career PI says:

    Actually, 1/3 acceptance rate for reviewer invites sounds not that far off. Remember that there are probably more review panels entirely made up of ad hoc reviewers than there are standing SS.

    For example, in 2014 I was asked to serve ad hoc on a regular SS, to serve on an SEP and on an STTR/SBIR review panel. Accepted one invitation, declined two (one because it conflicted with a grant submission, one because I had prior commitments for the meeting dates) = 2/3 decline rate.

  • qaz says:

    Mid-career PI has an important point. When they are measuring accepts, is the denominator individuals or times people get asked?

    For example, I am asked to be on study sections between 3 and 6 times per year, even while I was a standing member on a study section (and thus doing 3 study sections per year). When I say asked, I mean asked *in addition to* being a standing member.

    More importantly, if they think they are having trouble getting reviewers (whether they are or not), they should (a) invite assistant professors on - as DM, CPP, and most of the commentariat has said on this and other blogs and (b) incentivize review panels better - at least give us our donuts back.

    DM - can we use this misperception of getting reviewers as a hook to get them to let the assistant professors back on study section?

  • Namesaste_Ish says:

    Wondering what those numbers look like when they include losers like me who agree to ad hoc but not to sit on standing study section. Because of prison time and what not.

Leave a Reply