Your Grant in Review: Skin in the Game

Jan 25 2016 Published by under Careerism, NIH Careerism, Tribe of Science

Should people without skin in the game be allowed to review major research grants?

I mean those who are insulated from the results of the process. HHMI stalwarts, NIH intramural, national labs, company scientists...

On one hand, I see argument that they provide needed outside opinions. To keep an insular, self-congratulating process honest.

On the other, one might observe that those who cannot be punished for bad behavior have license to be biased, jerky and driven by personal agenda.

Thoughts?

Would you prefer review by those who are subject to the funding system? Or doesn't it matter?

46 responses so far

  • Sam says:

    In my (admittedly-limited) experience, I would be horrified to see the IM NIH researchers I've met or know on my study section. There is a major disconnect to how the real (I'm saying extramural for this topic) world works.

  • Philapodia says:

    If you don't have skin in the game then how can you understand the true impact of your scoring? It's like letting a trust-fund baby pick your stock portfolio for you. Who cares if you're wrong, it won't affect you in the least!

  • drugmonkey says:

    And I forgot to mention scientists from other countries- occasional participants on NIH review, IME. Reminded by a Twitter person.

  • SidVic says:

    if you are hhmi; why in the hell would you want to review? Is this how it works? One is punished for bad behavior. Which, of course, is always in the eye of the beholder.

  • eeke says:

    Yes. Why would these people have a "personal agenda" any more or less so than those who do have "skin in the game"? And I'd be curious to know, who does get "punished" for bad behavior? I've never heard such thing except in extreme cases of racism. Would getting kicked off a study section (which some people might be thankful for) be a real "punishment"? Can you elaborate on this?

    The goal should be to bring in a balanced panel of expert scientists. If the playing field for reviewers extends world-wide, I think this would add to the review process and not in any way be a detriment. The only people I see who would have a problem with this are back-scratching NIH club members.

  • qaz says:

    This skin in the game analysis was what they used to kick assistant professors off of study sections. I think this was a terrible dis-service, both to the assistant professors who needed the learning experience, and to the study sections, because (in my experience) "Yun Gun" and their assistant professor colleagues were often the best reviewers, fair, thorough, and really taking the time to read through the grants.

    I don't care about HHMI. There are too few of them to matter. In my experience, IM people are good scientists and can do reviews as well as the rest of us. Foreigners could be pulled in for their expertise (I certainly do my share of reviewing for foreign countries), but we have a large enough pool of reviewers here. (Especially if we let the assistant professors back.) Let's get everyone to pull their share.

    (I think that someone who has not done their time on study section should not be eligible for NIH funding. That would free up a lot of money for the rest of us who are willing to do the work and would solve the culling completely then and there.)

  • zb says:

    "Why would these people have a "personal agenda" any more or less so than those who do have "skin in the game"?"

    Yeah. Maybe there are those who can try to exclude everyone who works on similar topics (who have more conflict than those who vaguely don't understand how NIH grant demands change life in a research lab), but there are many fields in which excluding all those people means that there are no experts left. In fields I'm familiar with, much progress has depended on whether the other guys in the field are "nice guys" or not. Widening the pool to those who aren't trying to draw from the same pot of money might be a benefit, not a curse.

  • DNAdrinker says:

    Many grant programs ONLY allow reviewers who don't have skin in the game. The Texas cancer research program is like that.

  • Established PI says:

    I haven't seen examples of CSR punishing bad behavior but see plenty of rewards for members of subfields that look out for one another rather than being open to fresh ideas and new areas of research. The old bias against junior unfunded investigators was that they lacked sufficient experience to do a good job of evaluating grants, not that they didn't have skin in the game. I want my grants evaluated by experts who have a broad perspective, whether or not they are funded by the particular agency or foundation to which I have submitted my grant.

  • drugmonkey says:

    qaz- but assistant profs most assuredly have skin in the game! They are submitting grants - that's being in the game.

  • Pinko Punko says:

    My only worry about IM on panel is that first few panels, they might go with flow. It is entirely different world for them (as would IM be to us). Because of this, there can be some culture shock. I would say that this just would be sub-category of situations where indistinctness of voice might dilute power of that reviewer in discussion.

    If IM reviewer lacks confidence or shows reluctance/hesitation it either positive or negative critique, I think the panel could be non-responsive. I would say that I have witnessed such. I think it would be harder for lifetime IM to gauge scope of some proposals. I think this would be likely to hurt non-BSDs because BSD science has a lot of thumbs on scale in terms of perceived impact.

    Scientifically, there are a lot of great IM people, and they can of course add value.

  • dr24hours says:

    Equally you can ask the question if people with skin in the game should be allowed to review at all. If you're submitting grants in this environment, you have a conflict of interest.

  • baltogirl says:

    The reason they call it "peer review" is that it is supposed to be done by peers. An assistant professor who has hardly published, who has received perhaps a single R21, and who has only a few years of experience in the field is not the peer of a professor with decades of R01 experience (both reviewing and receiving). There are enough of us around that an attempt should be made to match experience. This will rule out a lot of company people etc.
    IM scientists are reviewed by external reviewers, so why should they be barred from external review?
    HHMI? this is a tough one. Some probably provide great reviews, others perhaps trash everyone not in their elite universe. As mentioned above, I suspect few bother to serve, so it is likely not a significant factor.
    Some (smaller) countries ONLY use outside scientists to review their grants to avoid bias. I don't have a problem with foreign scientists- provided they are peers.

  • dr24hours says:

    The idea that an assistant professor isn't a peer and thus shouldn't be allowed to review grants for more senior scientists is absurd to the point of ridiculousness.

    Why not turn it around? "Older scientists trained too long ago to be peers of today's up and coming scientists. They're behind the times."

    What you say? All that matters is demonstrated expertise in the subject matter? Yeah that goes both directions.

  • Joni says:

    Given how difficult it is to get senior scientists to serve on study sections, I'm not much bothered where their current funding is coming from. Kudos to them for agreeing to serve when it doesn't benefit them.

  • baltogirl says:

    "Demonstrated expertise" is a good criterion. That includes papers and grants, both receipt and review: which assistant profs have much less of. I have seen a lot of grant-inexperienced assistant professors take down grants for bizarre/petty reasons; this may also happen in the other direction, but far less often.
    More time in the saddle helps a lot.

  • The Other Dave says:

    Anyone with skin in the game is competing for the same resources. That is a clear conflict of interest, especially in today's hypercompetitive funding environment. Thus, one could argue that the best reviewers are those WITHOUT skin in the game.

    I also agree 100% with qaz that anyone who gets NIH funding should be willing to review. I have actually had this discussion with POs, and they agree. The problem with mandatory reviewing, they say, is that not everyone who gets NIH funding would make a good reviewer. At NSF, if you say no to review requests unnecessarily, I *know* it gets you a black mark when it comes time for the POs to make funding recommendations.

  • dr24 says:

    Sorry, but evidence for "far less often" is necessary. And numbers of papers/grants means nothing with regard to how qualified a person is to review. Entire line of argument is paternalistic nonsense.

  • Grumble says:

    "At NSF, if you say no to review requests unnecessarily, I *know* it gets you a black mark when it comes time for the POs to make funding recommendations."

    Since I'm at a med school and have no chance of ever getting a NSF grant, this means that I will feel free to return the favor. Why should I review for them if I can't get funded by them? Or even get an honorarium (which I always get for reviewing NIH grants, and which I use at the bar to deaden the pain of having to reject 9 out of 10 grants).

    (And yes, NSF has asked me a few times, and I said yes once and never again. Bridges successfully burnt, I guess.)

  • Dave says:

    I had my grant reviewed by an SEP, and only one of them had an active NIH grant. Several of them hadn't had a grant in years. Most of them hadn't published in yonks, but they were senior profs. Do they have demonstrated expertise? I got a fair review, but it's an interesting discussion.

    I'm willing to review. I'm willing to review. I'm wiling to review. I got accepted into the ECR program, but it's been crickets ever since.........

  • MorganPhD says:

    How many HHMI profs DON'T have NIH funding? My guess is less than 10%...

  • drugmonkey says:

    FTR, this blog remains staunchly in favor of assistant professor participation on NIH study sections.

  • DrESchaffhausen says:

    Serve on a study section which reviews some drug development grant apps (U type mechs) in addition to traditional R and P mechs. About 10-20% of panel is from companies. They bring a valuable perspective and are as well behaved as the academic types, maybe even a little better. Nice to hear voices from the real world on what it actually takes to develop a drug and what is wishful thinking (ie., your siRNA against c-fos is not going to be the next treatment for heartburn). This is also common on SBIR/STTR panels.

  • Juan Lopez says:

    I have tried being reviewed by those who are not your peers. It is no fun. My department chair has never had NIH funding.

    While, in theory, they could be wonderful unbiased reviewers with fresh ideas, in practice they just make unrealistic demands from a position of power. No, thanks!

  • L Kiswa says:

    "I have tried being reviewed by those who are not your peers. It is no fun."

    How are we defining peers?

  • Jonathan Badger says:

    Have none of you ever been asked to review a grant from another country? It seems some countries (Ireland for example) actually *want* input from outsiders.

  • Baltogirl says:

    I review for other countries occasionally. Usually I never find out what happened. Some have a separate level of review on top of the foreign input that is essentially political, i.e. a super powerful Council, such that your work is really pro forma.

    Dr24h, fledgling assistant profs are not born with the magical ability to do jobs they were never trained for. Years of writing and receiving grants puts them in the position of being able to review, and if at all possible, those are the folks I want reviewing my work. Sorry, dm, we can again agree to disagree.

    Juan, I agree that the senior chair who has never or only rarely been funded is also not a true peer. I do think SROs are having trouble getting grantees to say yes to serving, so they are asking people they shouldn't, hurting the quality of peer review.

  • Grumble says:

    "I do think SROs are having trouble getting grantees to say yes to serving, so they are asking people they shouldn't, hurting the quality of peer review."

    Yeah, whenever I get asked to serve on a study section, I say I'm about to submit a grant to it, so I can't serve this time. Most of the time it's actually true. So I can see why SROs are having problems.

    OTOH, if I get asked to do an online review, I almost always accept because the workload is minimal and I don't have to travel.

  • Grumble says:

    "Have none of you ever been asked to review a grant from another country?"

    Several times. Sometimes the questions on the review form are so vaguely worded that I have a hard time understanding what they are asking. Because the ones I've been involved with have mostly not involved an in person meeting of the review panel (either face to face or online), and the agencies haven't let me see the other reviews, I can't gauge how my reviews measure up to those of other reviewers. So I find it frustrating to do foreign reviews.

  • Dave says:

    I review for several other organizations too, here and abroad. And I usually get paid. Not sure what makes the NIH so special that they can afford to exclude most ass profs en bloc.

  • drugmonkey says:

    I think that the vast majority of any real problem with assistant professors reviewing grants is chalked up to *inexperience reviewing grants*. The solution for that is to get them some experience of course. Just like anyone else.

  • drugmonkey says:

    JB, Grumble-

    I've done reviews for maybe three non-US funding agencies and I usually find the guidance documents sufficient to get me close. After that I figure that their system is used to accommodating the NIH-style review orientation or they wouldn't have asked me.....

  • drugmonkey says:

    Baltogirl-

    Step back for a second and think about how your narrow definition of a "peer" for reviewing affects the system as a whole. Don't you think there is a certain self-selecting, inherent conservatism that is bad for scientific diversity?

  • drugmonkey says:

    Btw, in some of my earliest NIH study section experiences I think I immediately ran across two or three standing members who were from Non-US institutions. This seems not to happen anymore? I'm not clear on whether Scarpa's cost-cutting ended that or if I just noticed very rare exceptions in the first place.

    Anyone else remember standing members of NIH study sections who were not US based?

  • Susan says:

    In theory, a wide variety of opinions is a good thing. Intramural? HHMI? Non-US? Great.

    In practice, you get 2* people reading and judging your grant in detail, and representing it to a murder of crows. Those two people are crucial. I think they do need skin in the extramural NIH game (and asst profs. certainly have skin in the game).

  • qaz says:

    At some point, everyone is new to study section. They therefore have little experience and are naive about grant review processes. There is no reason to think that assistant professors are going to learn any less quickly than associate professors and lots of reasons to think they will learn faster.

  • The Other Dave says:

    @DM: The membership of all the study sections relevant to me are 100% U.S.A. And mostly from the same institutions, over and over and over again. It's kind of a circle jerk, actually.

  • drugmonkey says:

    Are you saying the rules for geographical diversity are not followed? Maybe you should complain about that.

  • The Other Dave says:

    Are you saying the rules for geographical diversity are not followed?

    Geographical diversity and institutional diversity are two very different things. It's very easy to have reviewers from the East, Central, and West, but all from the same half dozen institutions. And that doesn't seem to change from year to year. I remember one year there were four people from the same institution on the same panel. There always seems to be at least two from the same place.

    Maybe you should complain about that.

    ha ha Nice try. Why don't YOU tell the SROs that they're screwing things up? Meanwhile, I'll keep sucking up to mine 😉

  • drugmonkey says:

    I was told by an SRO that ppl couldn't even serve on a single meeting panel from the same institution except as very rare exceptions. Empaneled members from same Uni were right out.

    Guess that varies across CSR.....

  • ProdigalAcademic says:

    Many researchers at National Labs are also grant supported (even if not by NIH). Same for researchers from other countries. Industrial researchers also need to compete to have their ideas funded, so it isn't like successful researchers who are not supported by NIH grants are not aware of funding pressures and proposal writing.

    I think the diversity of opinion is better served by inviting experts in the field to evaluate proposals, even if they have direct skin in the game. If anything, they are less likely to play out personal hobby horses than NIH funded panelists, because they don't have an NIH-related agenda.

    With the concentration of Federal funds going to a smaller and smaller subset of people who have trained in the same places, maintaining outside perspective will be increasingly important.

  • A Portlander says:

    Your tweet about wishing more death on the Bundy idiots is fucking stupid. This shitshow is probably gonna come to my town. Maybe you'd like to re-enact Oklahoma City where you live, I'd really rather not.

  • drugmonkey says:

    It was a poll genius. And the forces of peace are winning.

  • brainsbourbonbeer says:

    re: HHMI profs - my understanding was that HHMI profs not @ Janelia have their salary (generously) covered by Hughes, but still have funding from NIH for lab expenses (I've worked in a few HHMI labs and we still applied for RO1's and other funding mechanisms).

  • E-rook says:

    To answer your question: Yes. I think for certain areas, especially where clinical trials might be involved, or technology development, it is important to have industry (pharma, device, medical services) evaluating grants. I've had several conversations with folks in medical officers' office in companies ... my impression is that they are passionate about patient care and are optimistic about the potential of research results to lead to developments that make a difference in people's lives. I've also had several conversations with R&D folks who are very pragmatic about limitations, what is feasible to bring to market or to scale; but also will recognize great ideas. These are folks I would want on study sections. Assistant Professors should definitely be on study sections. From industry, they would have the knowledge/experience to make appropriately powerful & influential statements, and know when & how to speak up to achieve their goal. When I was an Ass. Prof., I used to write reviews for the full prof who's grants were paying half my salary and who's lab space I "shared" ... for some reason this person served on 2-3 SS's per year -- each time it was, "I absolutely CAN'T say NO." whatever.

    I do submit grants on behalf of a non-profit. A very minuscule amount of our budget is from federal government. Despite the large amount of work for a low payout, we feel it is important from a programatic perspective to submit & obtain NIH grants.

  • thorazine says:

    A bunch of you are mentioning review for non-US agencies. In a lot of cases, our (non-US) funders need to look for reviewers from other countries simply because most countries are not as big as the US. Jonathan Badger mentioned Ireland - the population of Ireland is about 1.4% the size of the population of the USA. Academic science is, if anything, a _smaller_ fraction of the workforce than in the USA. This means the domestic reviewer pool is very limited. It would be impossible to get useful scientific review relying purely on Irish reviewers. At the same time, there are obvious political reasons you can't really make a standing review panel that's 90% foreigners. The only way to do this is to rely heavily on ad hoc review and on the generosity of foreign scientists with their time and expertise. (Thank you.)

    I work in a country larger (thank God) than Ireland, but small enough that a lot of our reviews come from abroad. I try very hard to avoid naming Americans as reviewers for my grants. The expectations are too different; for example, we cannot (generally) resubmit - if you're giving comments that you think should be addressed in the revision, you've killed my grant.

Leave a Reply