NIH clumsily tries to .. [something] ... for grant reviewers

Feb 20 2015 Published by under Careerism, NIH, NIH Careerism

I noticed a funny one in the NIH Guide notices today.

NOT-OD-15-035 Reinforcing Service to the Biomedical Research Community

Yes, yes. I see. "Reinforcement" of a behavior like "Service to the Biomedical Research Community" means increasing the strength or probability of the behavior. So yes, that's good. What are they trying to do here?

This Notice gratefully acknowledges, and seeks to reinforce, service to the biomedical research community by recipients of National Institutes of Health (NIH) research funding (see NOT-OD-10-089). Obtaining input from qualified experts across the entire spectrum of the extramural research enterprise furthers diversity of scientific thought, inclusiveness, and breadth of perspectives necessary to evaluate applications in a review process that strives for integrity and fairness. The interdisciplinary, collaborative, and global nature of biomedical research today requires increasingly complex review panels that need both broad and specific expertise in countless topic areas. Thus, the NIH, the biomedical research community, and the general public benefit from the service of NIH-funded investigators and maximize the Nation's investment in biomedical research.

Yes, yes. Very nice. but what are they actually doing to reinforce the behavior?

The NIH expects principal investigators of NIH supported grants and contracts to serve on NIH peer review groups, when asked. Therefore, the NIH expects grantee institutions and R&D contract recipients to encourage their NIH-funded investigators to serve on NIH peer review and advisory groups. These groups include Scientific Review Groups (or “study sections”) in the initial peer review of grant applications and technical evaluation of R&D contract proposals, National Advisory Boards or Councils (NACs) for second-level peer review, NIH Boards of Scientific Counselors (BSCs) for intramural programs, and Program Advisory Committees (PACs) for initiative development and concept review.

emphasis added.

Okay, so any University with a pulse is already encouraging their PIs to serve on study section. Right? They know about how this will help their bottom IDC line, yes? And if they are discouraging any subset of investigators from serving I imagine it is the Assistant Professors...who the NIH / CSR isn't looking to recruit anyway.


I have a suggestion. Two actually. The first one is hey, if you want to reinforce a behavior, why don't you use the delivery of a rewarding stimulus? I mean sure, you give us reviewers a delay in the submission deadlines, that's cool and all. But obviously the NIH thinks they need something more. How about protection from budget reductions? A couple of extra percentile points on newly competing awards?


Okay, that costs you money, I realize. How about something very cheap with some motivational value? Journals often publish a list of their reviewers at the end of the calendar year and thank them for their service. It's nice. But the NIH can do this one better. Set up a website with a list of reviewers and the number of grants they've been assigned to review. Maybe do it by year too and provide permalinks.

Trust me, academics will eat this up. They will check out how many reviews their buddies are/are not doing and give them a little hell for not matching up around the conference coffee table. They will start linking to their entry from their websites and bragging about it in their P&T documentation.

I wonder. Really, NIH. Do you have anyone making policy that understands people even the tiniest little bit? I am about the opposite of a people person and it took me like two tweets to think of this.

31 responses so far

  • Comradde PhysioProffe says:

    I expect daily blow jobbes and encourage my wife to provide them. Guess how many blow jobbes I receive.

  • datahound says:

    I know you do not like to hear this but the NIH General Counsel's office expressed the opinion that any grant compensation (no reduction, additional years, etc.) based on review service was not legal. Similarly, requiring grantees to serve if asked as also deemed illegal.

    I do like the idea of sharing review service records publicly. The rosters are already available (although not in an aggregated format). One concern I can imagine is releasing information that could somehow be used to link reviewers with specific applications.

    We can try to push this idea further if you are game.

  • Philapodia says:

    Something in Richard Nakamura's post on RockTalk that struck me:

    "Looking closer at the data, if ONE DAY of peer review service per year would be considered a reasonable expectation for service, then currently fewer than half of these funded scientists (45%) achieve that level of service." (emphasis mine)

    It takes significantly more than a day to prepare for each study section. It usually takes me a solid week to review a stack of grants, especially if they're not in my direct area. With other distractions (teaching / discussing science with my students / the lab catching on fire) it's usually a 2-3 week process to get the critiques written even before traveling to and reviewing in study section. I wonder if there is a disconnect between how much time the NIH thinks it takes to review and how much time it actually takes. If you're a regular member, this is a significant chunk of your time to be a good reviewer. This is probably one major reason why not everyone jumps when asked to study section.

  • drigg says:

    Excellent idea. Scientists love metrics like this. Chairs love em. P&T committees get absolutely warm and fuzzy over them.

    It shouldn't take long after implementation for someone (ahem, Datahound) to crunch the numbers to find a positive correlation between reviewer activity and grant success.

    I disagree that the CSR is not looking for Asst Profs as reviewers, though. Not as standing members much, but definitely as ad hoc reviewers. Having a few NIH IAR reviewer sessions on an Asst Prof's P&T application is weight on the right side of balance at most institutions.

  • Rebecca Jacobson says:

    i would be happy with free coffee and snacks at study sections I am that cheap!!!

  • drugmonkey says:

    DH- yeah I was joking about grant-related direct compensation. But I'm confused how aggregating the number of reviews each person wrote per year makes them more identifiable than their name on a given study section meeting's roster. If anything it would tend to obscure matters.

  • CD0 says:

    Serving as a regular study section member certainly involves a huge amount of work and does not necessarily make you many friends in these terrible times. However, it also provides a perspective of where the field goes; teaches you how convey your ideas; keeps you sharp; puts you in touch with other experts in your field; opens to you new concepts that could help your own research; and may help you a little bit in terms of visibility. Overall, I do not see how you can get understand the peer-review process and get funded if you do not serve periodically.

    Some of the frustrations associated with the job:
    1-I find outrageous that somebody who did not bother serving in study sections for years publicly complains about the system when his/her grants don't do well. Shameful.
    2-I am also frustrated about the obsession of some institutes for controlling the process of peer-review or going against the opinion of peer reviewers. For instance, the NCI now keeps control of the review process for all R21s, for which they need to recruit a significant number of reviewers who never got a grant, because the rest are involved in regular study sections. Even worse, some institute directors are selectively funding applications that literally received the worst scores in the meeting and defund the ones that were most appreciated by the panel.
    3-It is sad to see first hand how many potentially great studies will go unfunded because less than 1% of grants can make it.
    4-It's also ridiculous to take a couple of breaks on a >10 hours meeting and find luxurious foods and beverages for corporative meetings next door, while we have to run to the nearest bar for the most expensive cup of coffee.

    I agree that increasing the visibility of reviewers would be rewarding. But perhaps also opening some of the "outstanding", "transformative", "emeritus" awards only to individuals who have served for at least an entire term.

    I do not think that forcing people to serve would help (if that was legal), because I would like to get a reviewer who does not want to be there.

  • CD0 says:

    I would NOT like to get a reviewer who does not want to be there...of course

  • Ageing PI says:

    Bring back the free coffee at study section meetings! NIH stopped paying for it and its an enormous waste of time having to search the local area for some. Its the little things that matter...

  • Physician Scientist says:

    and maybe don't give us a 1099 tax form for our per diems (transportation, food, internet, etc) - either we pay taxes on it or need to keep the receipts and put it in Schedule C come tax time.

  • Pinko Punko says:

    It is government-wide that they can't have food in the meetings.

    I do think people should get spanked for lazy/poor reviews. I know some people don't think it is worth their time to review their whole pile the way they would if they were primary, but f*ck just don't go if you are going to phone it in. I know everyone is totally demoralized, but blech.

  • drugmonkey says:

    Some SROs either assign a Discussant to some apps or SROs/culture permit/encourage Reviewer 3 to act as Discussant. That can mean just writing a summary paragraph instead of a full criticism. I always have a lot to say and have written full critiques as a Discussant but if someone chooses not to, how can we fault them?

  • Pinko Punko says:

    DM- not talking about that- I am seeing more and more R1/R2 look like R3. And am seeing R3s being emptier. This is not everyone, just increasing subset. Also, it is culture that allows that- there are never specified instructions in my experience that say if you are R3 just write a paragraph, it is allowed but not prescribed. What I have seen is that if critique is negative it is requested that it should really provide evidence of the key points, especially if a particular subscore is driving the overall. I just wish engagement were higher. I am demoralized for PIs that get hammered with empty or generic stuff. We all get some of that, but I think we need to fight against it.

    I have seen a decrease in engagement from the last two years to this year and I don't like the trend. I feel reviews can be pettier and they are less constructive. I know that you like to talk about it not being reviewers job to do that kind of stuff, just to tell NIH what is what, but to me you only really can say what is what if it is clear what the magnitude of the issue is and is it conceivably fixable.

  • datahound says:

    DM: I have no specific idea of how aggregating data could make it easier to identify a reviewer. I was just thinking through potential objections and have been surprised about how data analysis could lead to identification in other circumstances. I agree that it should not be a serious concern.

    Aging PI: These rules do not come from NIH, but from HHS and higher levels of the government. It was a reaction (IMO an over-reaction) to abuses from other agencies with apparently lavish parties at taxpayer expense.

  • Davis Sharp says:

    I wonder if there is a disconnect between how much time the NIH thinks it takes to review and how much time it actually takes.
    Philapodia , February 20, 2015 at 7:38 pm

    Many (most?) SROs are former faculty members who have reviewed. I don't think there is a general disconnect.

  • Grumble says:

    CPP: Maybe you should try rewarding your wife with a website describing her service and commending her for it.

    DM: Your suggestion to publicly acknowledge reviewers as a form of reward just reinforces that we academics are willing to accept peanuts for our services, and rancid ones at that. As someone alluded to above, even the review-related expenses we get reimbursed for get reported to the IRS as income -- in fact, this year they changed the 1099 form so that both the official payment ($200/day) and expense reimbursement amount are lumped together, so you can't even separate them out easily and list only the real income on your tax return. Thanks a lot. I'm halfway tempted to start declining review invitations. On balance they are a waste of my time. Why should I serve if my service is not remunerated at a level commensurate with its worth? -- and then be forced to pay taxes TWICE on the expenses I incur to provide that service?

  • drugmonkey says:

    PiPunk- how would you feel about a simple checkbox that said "meh".

  • Established PI says:

    Grumble: Report your travel expenses on Schedule C to offset the 1099 income so that you only end up with a tax liability for the honorarium portion of the payment.

    DH: If there are no legal means to either reward or punish PIs for their review service, then what is the point of all these postings? I can't imagine that the target group (namely well-funded PIs who don't review enough) will alter their behavior based on public shaming, as I am sure they will fire back with all the amazing things they are doing for science in lieu of NIH reviewing.

  • pinus says:

    the tax thing is really annoying. a colleague got audited because of it.

  • drugmonkey says:

    Oooh, new thought. Publish the percent of PIs who serve on study section by applicant institution. Also by funding IC to detect bunnyhopperism and subfield scofflawism. C'mon NIH, let DataHound loose on the review numbers!

  • anon says:

    Not having coffee or refreshments at review was not a NIH decision. A memo came down from the Department in response to certain scandals regarding government waste at other departments (known as the muffin memo). Among the requirements included a ban on paying for refreshments at conferences and the requirement to renegotiate any contracts for meeting space that may include them (even if it increased the price). There were many other requirements such as requiring department approval for conference grants over a dollar threshold, caps on number of nih work related travel, etc. Anyway, coffee isn't coming back.

  • dsks says:

    If any y'all happen to be reviewing one of my grants and a mysterious suitcase full of premium muffins shows up outside your hotel room door... I know absolutely nothing about it, just fyi.

    But yeah, all this "expects"... "encourage"... "Oohhhh wouldn't it be luvv-er-ly!!!!" guff isn't going to change anything. And wonders, really, whether it's meant to, or whether this is just a way of allowing the status quo to continue while displacing blame from the NIH policy makers...

    "Well, we did ask very nicely, but nobody listened to us!" &c

  • Eli Rabett says:

    It was the usual congressional fuck up. The cost of muffins and coffee was aggregated in a bill including the cost of the room, so it looked like the muffins cost $100 each, and it went downhill from there.

    OTOH, since you can claim expenses, let everyone pitch in for coffee service and have it delivered. The hotel or conference center can provide the bills.

  • Pinko Punko says:

    The "meh" checkbox would be more transparent than a pile of meaningless words. If you can't articulate your meh, you aren't doing the job, though "meh" checkbox would be a wonderful place to obscure any and all bias. :/

  • […] NIH clumsily tries to .. [something] … for grant reviewers India’s toxic air is taking away three years from your life (probably affects cognitive development too) Major study led by autistic scientist challenges long-held preconceptions about the condition High-impact journals: where newsworthiness trumps methodology (technique matters) 2014-2015 Influenza Season Week 6 ending February 14, 2015 […]

  • Philapodia says:

    @Davis Sharp

    I was talking about the upper admins at NIH, not SROs in the trenches. Richard Nakamura's post on RockTalk was implying that one day of service isn't too much to ask for if you're funded, but it is much more than that. DM posted that on the RockTalk blog, so hopefully the higher-ups at CSR either choose their words with more care next time or get a clue. Telling your volunteers that they're lazy isn't a good way to keep them working for peanuts.

    I've told a friend who is a gov't contractor in the DC area what the compensation is when I review (ie per diem, honorarium, travel expenses). He said I'm getting ripped off by normal gov't contractor standards.

  • gingerest says:

    Australia's National Health and Medical Research Council doesn't go as far as you want, but it does at least thank reviewers by name on the website:

  • TheIntronertPhD says:

    Comradde PhysioProffe, well, then promise to put her name on a list on the internet with all the other wives that give you blow jobbes. She will eat it up!

  • Grumble says:

    "He said I'm getting ripped off by normal gov't contractor standards."

    Of course, and therein lies the problem. Such an incredibly smart group of people willing to labor for days in exchange for pocket change, no coffee, no muffins, and the privilege of having that pocket change be taxed twice. Maybe we should go on strike. Or maybe we aren't as smart as we think.

  • Joe says:

    The SRO gets to pick the hotel from a list of available approved hotels. For the meeting I was just at, the hotel provides free coffee to all meetings as their policy. Panel members expressed to the SRO their desire to have meetings at that hotel again.
    The no coffee things is plain stupidity. NIH is paying hundreds of dollars per person to fly in 25 experts and put them up at a hotel, but they can't pay $200 for the group to have coffee to keep them awake through the meeting? Stupid, stupid, stupid.

Leave a Reply