Your academic society is working for (or against?) you under the NIH Grant waterline

A recent blog entry from Pascale H. Lane discusses her reasons for belonging to academic societies. Our good blog friend Dr. Isis is frequently found to be going all fangrrl about the APS (no, not the real APS, these Physiological pretenders who are well down the GoogJuice list). Pascale touched on one Golden Thought about what academic societies can do that is, or should be, of general interest to my readership:

The society maintains several grant programs for research funding, and it leads advocacy efforts to maintain adequate federal funding for kidney disease research and treatment.

Grant \($! Wooot!


Quite some time back on the original incarnation of DrugMonkey we started talking about the bunny hopper phenomenon. This was long-time commenter whimple's coinage of the term:

Say I work on the mechanics of bunny-hopping. My papers get sent for review to colleague bunny-hoppers, my grants are reviewed by the bunny-hopping study section, and there is never really an opportunity (ESPECIALLY with the study section) for a non-bunny-hopper to stand up and say, "look, other than the bunny-hoppers, nobody really cares about bunny-hopping, and I think we already know all we need to know about bunny-hopping for now," and close down the field.

Academic societies, of course, are bunny-hopper advocates. That's the whole point. To enhance, support and promulgate science in a relatively specific domain of interest. It could be disease related, basic science theory-related, biological sub-system associatedor technique-dependent. It can involve a specific research species or human subpopulation. Perhaps the bigger societies seem very broad ("Neuroscience") but that is only because you are not considering all of biology, health/medical science and/or all Federally supported research. Even these big ones are bunny hoppers by some accounting.
One of the things that academic societies are doing, or should be doing, on the behalf of the bunny-hopper scientists in their domain is to manipulate the NIH grant system. To work to ensure that as large a share of the NIH pie goes into their preferred topic domain as possible. We have seen some of this in the current (albeit waning) battle to merge the Alcohol-focused NIH Institute with the Drug-Abuse focused one. The Research Society on Alcoholism has been battling mightily to keep NIAAA intact and un-assimilated. Another way academic societies battle is by sticking their noses into the grant review process.
leapingjackalope.jpg
IFBH represent!
Today's topic comes from a long-term reader who sent me something that ties back to our prior discussion (original here) of what NIH terms "clustering" and "captive study sections". The most extreme version of a captive CSR study section would be one that exclusively reviews grant proposals that would be assigned to a given NIH IC, say, NIAAA. But study sections might also be captive to a given topic domain. Conceptually we might think of this as captive to a specific academic society such as the International Fellowship of Bunny Hoppers. If we harken back to whimple's original critique, this sort of topic insularity of a study section risks continued NIH funding of decreasingly relevant science which has long since passed it's "drink by" date. Naturally, this is the concern of those who are not in the IFBH but are rather in the Society of Badger Diggers which, as we all know, work on much more relevant topics and publish more impactful research in journals of significantly higher prestige.
For the IFBH scientists, however, it is absolutely Right and Good that the relevant domain-captive NIH study section stay just like it is and has always been. The situation passed along by a reader, however, suggests otherwise. The NIH is apparently discussing the re-organization of a study section that is highly important to the IFBH scientists because many grants are reviewed there. The communication points out something you may not initially appreciate. Competition for funding, as we know, is tight. If you have 100 applications in a study section within a given round, then only about 10 of them (for argument's sake) are going to get funded. If that study section is captive to Bunny Hopping scientists, then they are competing against each other for an essentially fixed pool allocated to that study section (this is not the way it works exactly but it is a decent way to think about it). If, however, those 100 applications are spread out across four study sections with a more general mission and the Bunny Hopper proposals are highly competitive against say the Badger Digger, Otter Slider and Squirrel Hoarder applications then the IFBH might end up with 20-30 proposals being scored within the top tenth percentile.
So while the line PIs who have been successful might fear study section re-organization, perhaps the academic society is saying "Stop panicking, we've got plenty of more broadly competitive proposals, so let's go get some of that NIH \) away from those other societies".
The trick is, of course, that only some subset of the IFBH membership is going to benefit. Some other subset is going to lose out. The question for any society, or any kvetching PI, is whether these shifts are a good or bad thing.
Back to the individual PI level I advise, as always, that devising research proposals which span the interests of a range of study sections (and ICs) is absolutely essential. Sometimes you are going to have Bunny Hopper wheelhouse proposals that would get eaten up in a broader section- so it's nice to have that internecine section available. Other times, you are going to bring new techniques and approaches to the Bunny Hopper field and it is easier to convince fans of those techniques/approaches of the importance of Bunny Hopping than it is to get the Bunny Hoppers on board with the HawtNewTechnique.

39 responses so far

  • Marty says:

    I must defend the honor of our Dr. Isis and address DrugMonkey's comment that "Dr. Isis is frequently found to be going all fangrrl about the APS (no, not the real APS,...). Who is the real APS - the American Physical Society, the American Psychological Society, the American Peptide Society? There are too many of us to keep track. I suspect that Drug Monkey is alluding to those physics folks since their URL is APS.ORG. But they weren't founded until 1899, 12 years after the founding of the society for physiologists. We are the ones who should be APS.ORG. Unfortunately, the physicists created the internet, along with Al Gore, so they got there first. We got stuck with The-APS.org instead.
    Now for the advocacy played by the "real APS" based on age. I agree that there are study sections, societies, and institutes that are captured, that are destined to focus on one disease or organ system. Because our members do research across many diseases and organs, we advocate for increased funding for NIH, not for a specific institute. We believe in the wisdom of the individual investigator initiated research project. Let creatieve minds pursue their ideas freely and believe it or not, we will discover ways to answer the questions of disease that need to be answered.
    As for Dr. Isis, we love her and she is a great inspiration for students, postdocs and faculty seeking insights and advice. I am just upset that she only features hot shoes for women. What about us guys?

  • Jen says:

    @Marty "I am just upset that she only features hot shoes for women. What about us guys?" - maybe Dr. Isis didn't want to steal PhysioProf's thunder? http://physioprof.wordpress.com/2009/01/20/comrade-physioprofs-inaugural-shoe/ [although he has been sadly remiss in keeping up with his shoe postings....]

  • Kausik Datta says:

    ...there is never really an opportunity (ESPECIALLY with the study section) for a non-bunny-hopper to stand up and say, "look, other than the bunny-hoppers, nobody really cares about bunny-hopping, and I think we already know all we need to know about bunny-hopping for now," and close down the field.

    I disagree with this one-hammer-hits-all approach. What exactly qualifies a non-bunny-hopper to pronounce a judgement on the merits of the bunny-hoppers, or the relative merits of bunny-hopping research, with such an expansive statement? And someone who says "we already know all we need to know about" anything is a fool. Knowledge evolves and progresses. What is bunny-hopping today may easily provide tomorrow some great insight into the 'musculo-skeletal dynamics of psaltatory locomotion' of the great jackalope, and solve inner city transportation issues by producing an improved pogo-stick the day after. And bring World Peace.
    That said, given the tight competition for funding, any funding agency is bound to prioritize by identifying core areas. NIH under Collins has already announced a policy shift to fund more translational research activities (I can try to find the actual quote from Collins' speech - something about NIH being the National Institutes of Health, and not National Institutes of basic science research). But I sometimes wonder how long the translational research activities will be sustained if the wellspring of basic science dries up from lack of funding. There obviously has to be a balance. I think this is a time for an evidence-based approach.

  • whimple says:

    What exactly qualifies a non-bunny-hopper to pronounce a judgement on the merits of the bunny-hoppers, or the relative merits of bunny-hopping research, with such an expansive statement?
    What is the alternative? You can't fund everything so someone has to make these kinds of decisions. The bunny-hoppers aren't objective enough to make this call so non-bunny-hopper insight is needed.

  • SR says:

    My heart-felt condolences to Marty on reading the insults thrown both at his amazing society and his Goddess. I hope it won't ruin his Easter or Passover or whatever.

  • Kausik Datta says:

    ... so someone has to make these kinds of decisions. The bunny-hoppers aren't objective enough to make this call so non-bunny-hopper insight is needed.

    I am curious indeed. Do you understand the term 'specialist'? If you, say (nothing personal intended), have a persistent gastrointestinal problem, would you go to a GI specialist, or would you rather consult a cardiothoracic surgeon or a neurologist? If you wouldn't trust a cardiologist or a neurologist to pronounce a diagnosis on a GI issue going out of their specialty, why would you trust a non-bunny-hopper to understand and appreciate the issues associated with the studies on bunny-hopping, let alone pronounce judgements upon bunny-hoppers?
    You are rather facilely assuming the absence of objectivity on part of the bunny-hoppers. I think it is unfair to make this generalization. However, even if I accept your position to be true - given the fallibility of humanity, the best course to follow IMO is, as I said earlier, an evidence-based approach. Would it be too difficult to devise a metric based on several parameters assessing viability of a particular field or line of enquiry, and applying that metric to determine and prioritize certain fields over others?

  • Would it be too difficult to devise a metric based on several parameters assessing viability of a particular field or line of enquiry, and applying that metric to determine and prioritize certain fields over others?

    Yes, it would. In fact, it is impossible to devise such a metric.

  • Jim Thomerson says:

    Some years ago, mid-80's maybe, there was an article in Bioscience looking at growth of science as a logistic growth curve. As resources doubled, amount of scientific effort increased exponentially. However, funding for science was projected to never again double, so science is at carrying capacity. With no, or little, growth of resources, there will be increased competition among science areas for support. Some areas would increase while others would lose ground, or even become extinct. For example, a study revising a group of animals using DNA morphology is much more likely to be funded than the same study using only classical morphology.

  • ...the insults thrown both at his amazing society and his Goddess.

    Let it be known that I am the property of no man. Still, ping.

  • Kausik Datta says:

    Yes, it would. In fact, it is impossible to devise such a metric.

    Yet, CPP, we must strive for it. The alternative that whimple proposes is too arbitrary and fraught with problems to consider.

  • Yet, CPP, we must strive for it.

    Striving for the demonstrably impossible is fucking stupid.

  • Science is no different than any other industry. People don't invest in your shit unless they see that your product satisfies some need.

  • Kausik Datta says:

    CPP:

    Striving for the demonstrably impossible is fucking stupid.

    Yes, I presumed you'd say that. Do you have an alternative - as an NIH-funded researcher? Whatever your flavor of bunny-hopping is, would you be comfortable leaving the fate of your bunny-hopping to a random non-bunny-hopper, a la whimple?
    @Isis: True. But while one may not see a need in bunny-hopping, another may. Who's to judge? What's the goal, short-term or long-term? How to prioritize?

  • Whatever your flavor of bunny-hopping is, would you be comfortable leaving the fate of your bunny-hopping to a random non-bunny-hopper, a la whimple?

    Unless you're completely incompetent, your fate wouldn't be left to "a random" non-bunny-hopper".
    First, you would tailor your grant application to a *particular* study section that you have scoped out as being a suitable context for its review, and you would discuss this with the SRO so that you could ensure that it would be assigned to that study section. Second, as part of this tailoring, you would do a good job of justifying bunny-hopping to the expected likings, dislikings, and other inclinations of this study section. Third, you would impress upon the SRO well in advance of the study section meeting that it would be important for at least one individual with the right sort of expertise in bunny-hopping to be invited to serve ad hoc.
    If these things are not apparent to you, then you need some serious remedial mentoring in how to navigate the NIH system.

  • Kausik Datta says:

    CPP, thank you for providing a scope into the workings of the NIH study section. Yes, this is what I would do, and this is how it is done.
    As a researcher, you do your part by scoping out the particular study section that you'd want your application to be reviewed by, and by providing justification for your proposed work, making it dovetail with the goals of that study section.
    It stands to reason, then, that if your study is on bunny-hopping, you would try to find a study section stuffed with bunny-hoppers, so that it is easier to sell the merits of bunny-hopping to them. But isn't that what whimple's criticism is all about? As noted by Drugmonkey in the original post above:

    If we harken back to whimple's original critique, this sort of topic insularity of a study section risks continued NIH funding of decreasingly relevant science which has long since passed it's (sic) "drink by" date.

    We seem to have come full circle, because it is this characterization of the NIH study section in whimple's remark that I was objecting to.
    To my understanding, objective evaluation of a study proposal or grant application is the goal here. Whimple presupposes that bunny-hoppers are by default incapable of objectively assessing the merits of a proposed bunny-hopping study, and requires that non-bunny-hoppers be allowed to call the shots. I submit that inclusion of non-bunny-hoppers is no guarantee of an objective evaluation on their part, for they may be biased against a bunny-hopping study as equally as the bunny-hoppers are assumed to be biased for.
    Hence the need to develop a newer metric, a more objectively assessable criteria for evaluating grant applications. It won't be easy, but it is doable. I am basing my assumption on the principle of 'unmet medical needs assessment' that pharmaceutical companies often do before investing into research and development in a particular area.

  • To my understanding, objective evaluation of a study proposal or grant application is the goal here.

    This is root of your delusion. There is no such thing as "objective evaluation of a study proposal or grant application", and thus that cannot be the goal of the NIH peer review system. Rather, the goal of the NIH peer review system must be fair, balanced, transparent, subjective evaluation. Titlting after fantasmagorical nonsense like "objective evaluation" is counterproductive.

  • Kausik Datta says:

    Rather, the goal of the NIH peer review system must be fair, balanced, transparent, subjective evaluation.

    Well, you have just turned my world upside down, CPP. I was using the word 'objective' to mean 'unbiased', 'merit-based'. If that is not what the NIH peer review system is doing, then... crap!

  • Dude, you're fucking deranged. The NIH peer review scoring system explicitly embraces the concept that different reviewers will weight different factors differently in arriving at priority scores. They are *inviting* reviewers to allow their biases--i.e., scientific preferences--to influence their scores.

  • DrugMonkey says:

    KD,
    The NIH system features many levels of letting competing biases fight out where the grant money goes. The idea seems to be that in aggregate the individual biases will wash out *NIH wide*. But this is not quite the same as insisting that the disposition of each and every application is unbiased or can be placed on a single merit dimension with perfectly objective ranking.

  • Just to bring this back to the bunny-hoppping scenario, the NIH-mandated reviewer instructions for assigning criterion scores and an overall impact score *explicitly instruct you* that if you think the study of bunny-hopping is completely useless irretrievably boring shit fit only for douchebags, then--regardless of any of the other features of the grant application--you should give a bad score for significance and a bad score for the overall impact. They are *instructing* you to embrace your subjective bias against bunny hopping, and have it dominate your entire score.

  • Kausik Datta says:

    CPP and DM,
    It seems that I still have a lot to learn about the NIH peer review system. Thanks for that interesting primer.
    As to the impact of the subjective bias dominating the scoring system, I shall admit frankly that I still don't understand how that helps science and progress of scientific research, or whether that even makes sense. For instance, if I belong to the Society for Racing Turtles, what is stopping me from giving bad scores for significance and for overall impact to any and all applications proposing to study bunny-hopping?
    But I continue to learn.

  • whimple says:

    For instance, if I belong to the Society for Racing Turtles, what is stopping me from giving bad scores for significance and for overall impact to any and all applications proposing to study bunny-hopping?
    Nothing is stopping you, except the wily bunny-hoppers will be damn certain you never go near their grant as a reviewer. This is one of the reasons why a poor choice of study section dooms applications out of the gate. Unfortunately this phenomenon can also stifle innovation as "orphan applications" have trouble finding a good study section home.

  • DrugMonkey says:

    KD, if you mean the level of homer interest that operates *within* study section, flagrant bias does not convince anyone. Plus, most reviewers are trying to do their best to be objective. ( Within the limits of their scientific domain and professional viewpoints of course.)
    Nothing is stopping a reviewer from spiking particular kinds of applications but it has to be well grounded or first, nobody on the panel will vote with the crank and second, the SRO will stop assigning those apps to that person.

  • Kausik Datta says:

    DM, I spoke to my boss and confirmed what you told me here. It appears that there is a certain opportunity for self-censorship (or self-restraint, if you will); what I mean is that if you are a dedicated Turtle Racer, you may decline to be a part of a study section evaluating Bunny-Hopping. The SRO appears to wield a lot of power in that respect.
    However, I am still trying to understand the entire process. Recently, a friend of mine got a very low score (or, more precisely, too high a score, in this bizarro world, where a high score is a low score!) to her grant, and when the comments came in, they seemed to be rather frivolous, and seemed to show that the most of the reviewers did not quite understand what was intended in the proposal. I haven't been able to make any sense of that.

  • [quote][W]hen the comments came in, they seemed to be rather frivolous, and seemed to show that the most of the reviewers did not quite understand what was intended in the proposal.[/quote]
    It means the proposal was written improperly.

  • 24
    DM, I spoke to my boss and confirmed what you told me here.

    HA HA HA

  • DrugMonkey says:

    they seemed to be rather frivolous
    without knowing more about how much you've read up on what the NIH grant review process is supposed to actually be doing, as opposed to what you imagine it is doing, it is hard to address this.
    what are the general nature of the frivolities, if you can get more specific without revealing too much about your friend's identity?

  • Kausik Datta says:

    @Isis: I deserve the HAs, because I wrote in a hurry while leaving the lab for the day, and what I wrote did not come out quite right. What I meant was that I spoke to my boss about the study sections and the peer review process, and what she said was very similar to what DM (and CPP, colorfully) indicated. I was not doubting DM's words. Now move along, people! Nothing to see here.
    @DM: Thank you for offering to demystify this for me. However, if I try to specific, I am afraid I would reveal personal information. Let me just say that this was an R21 grant application. My friend got a bad score in innovation (which would have been understandable, but there was no comment in 'weakness'), approach (the comment in 'weakness' made it seem that the reviewer did not understand the nature of the "high-risk" application), and investigator (the "strengths" were all praises, while the "weaknesses" were restricted to one sentence, 'Dr. So-and-so has only one bench to work on', when nowhere in the application it was mentioned that Dr. So-and-so would be shackled to that one bench, or Dr. So-and-so cannot use the easily accessible central core facilities of the institution - which, incidentally, were listed in the grant application).
    I understand that this short description may be too nebulous to make any sense; I appreciate DM's offer to help nonetheless.
    @CPP: Sigh. Perhaps. I am not experienced enough to judge that proposal on its stylistic merit. I was just going by - rather simplistically - what seemed to make sense, to me. But as I said before, I continue to learn.

  • whimple says:

    'Dr. So-and-so has only one bench to work on'
    To me this is code for, "no way we fund anyone without a tenure-track appointment".

  • DrugMonkey says:

    KD, my apologies.
    We occasionally get a little rowdy around here taking the piss out of each other about differences in informed opinion about the NIH grant process. Occasionally we forget that there are trainees in the audience/commentariat who really are just learning all this stuff and might be expected to be confused when the reality doesn't match a golden ideal.
    You are doing just exactly the right thing, btw, in running stuff you think you are learning from random assclowns on the internet past your local mentor(s). (Isis was laughing at me, not at you, I trust.) My mantra is that you should get as many perspectives on this grant and career stuff as possible and sift through it actively to see what makes sense for your own situation. Nobody has the OneTrueAnswer to success. Your goal is to tilt the odds as much in your favor as you can at any given career stage.
    Reading between the lines of your friend's situation it sounds very much like a junior person with nebulous independence is taking shots for this situation. Totally typical. And also typical that if a reviewer has made up his/her mind to assign a relatively bad score based on gestalt impression or fatal flaw that not a lot of effort is taken to "fix" the application for the PI. The failure to properly credit the High Risk / High Reward of an R21 could be down to any number of things. The application may not have made the case. There may be no case to be made, despite what your buddy thinks. Also, sadly, this is an area where IME study sections struggle. People have difficulty reviewing R21s differently from R01s. Difficulty taking the exploratory/developmental and risk/reward aspects seriously.
    we have had a lot of conversations over the years about reading grant review tea leaves, transition for jr scientists, independence, StockCritiques, etc. You can find these by going to the Archive link in the title bar and selecting NIH related and grant related categories.

  • I am not experienced enough to judge that proposal on its stylistic merit. I was just going by - rather simplistically - what seemed to make sense, to me.

    It doesn't matter what makes sense to you or to the applicant. The only thing that matters is what makes sense to the study section. If a grant application is being misunderstood by assigned reviewers, it means that the grant is not written properly for its audience.

  • Kausik Datta says:

    @DM: Thank you. [Bows]
    @CPP:

    If a grant application is being misunderstood by assigned reviewers, it means that the grant is not written properly for its audience.

    As everyone who has ever given a presentation knows that it is impossible to tailor any thing for an audience 100%. This is why I am having some difficulty accepting your statement about writing a grant application in a manner targeting a 'special' audience. On the other hand, I wonder how far it is possible for the reviewee to be able to choose his/her reviewers in the NIH system. Can one even suggest a larger set of reviewers from which the SRO may select a subset, or is the reviewer assignment totally dependent on the whims and fancies of the SRO, or is it done by trial-and-error at the expense of one or many applicants?

  • You are doing just exactly the right thing, btw, in running stuff you think you are learning from random assclowns on the internet past your local mentor(s). (Isis was laughing at me, not at you, I trust.)

    Indeed. I was laughing at the nature of pseudonymity on the internet and the fact that, for all we all know, DrugMonkey could literally be a trained monkey, spouting out crap about the NIH and the image of a commenter going to his boss and asking, "Are these dudes full of shit?" Or, better, the idea that DrugMonkey could actually be your boss and you going to her to ask her if she is full of shit. The statement lent itself to high comedy on a fucktillion different levels.

  • whimple says:

    The application may not have made the case. There may be no case to be made, despite what your buddy thinks.
    Or alternatively, the science of the application might be perfectly fine. Maybe the same application submitted by a different person, or to a different study section, or both, would have done really well. "Fixing" the science in the application may be irrelevant to obtaining funding, because the NIH funds grant applications, not science. In any event, a poor score and obvious disinterest from the reviewers all says you-can't-get-there-from-here to me. Maybe try a different study section, or a different funding mechanism, or a completely different project. The NIH thought about implementing a "will never be funded no matter how much you revise" checkbox to applications last year, but decided not to, but this is still the message you're getting back from the sound of it.

  • DrugMonkey says:

    KD, the study section audience is about 20-30 people, most of whom you know specifically because they are permanent members. The rest you can phenotype based on the past few rounds of ad hocs. This is not a crap shoot when it comes to audience

  • Kausik Datta says:

    Quick question.

    the study section audience is about 20-30 people, most of whom you know specifically because they are permanent members. The rest you can phenotype based on the past few rounds of ad hocs.

    Is this true for all the different study sections?
    Also, if a prospective applicant asks for it, can s/he get a list of the permanent members of that study section?
    If yes, can one lobby them? (Or, send in a dead horse's head, as the case may be?)

  • whimple says:

    Also, if a prospective applicant asks for it, can s/he get a list of the permanent members of that study section?
    You don't have to ask; they give you a list and these are publicly available on the web.
    If yes, can one lobby them? (Or, send in a dead horse's head, as the case may be?)
    Officially, direct contact for lobbying purposes with study section members is totally forbidden. Unofficially, I've seen this happen a lot anyway. We have a policy of inviting in regular study section members to give seminars here, so relevant people can "discuss their science with them" *nudge*nudge*wink*wink*. I've also heard (but not directly experienced) people have other people contact study section members off-the-record on their behalf to get the inside scoop on what the review was about, sticking points, personality problems etc.

  • (1) http://www.csr.nih.gov/Roster_proto/sectionI.asp
    (2) If you attempt to "lobby" study section members, your application will be withdrawn from consideration and you will be fucked.

  • DrugMonkey says:

    What CPP said.
    Also, see this repost:
    http://scientopia.org/blogs/drugmonkey/2010/04/repost-emnih-basics-the-study-sectionem
    With respect to whimple's last point, you are absolutely not supposed to discuss any specifics regarding any particular application. I am not naive and I understand that people are people, so yes there is slippage.
    However, once upon a time I sought information from an excellent SRO and was informed that it is okay to talk in generalities about how my study section tended to operate. Biases, expectations, memes.....you know, the culture. Even to provide somewhat targeted comments about what a given PI might ask me (again, without revealing the specific discussion at study section*). Just so long as I would extend my comments to anyone who asked.
    __
    *ok, I'm having trouble drawing the distinction. Think of it as if a newbie asked me about how their grant would fare if submitted to my section. Ora request to make general interpretive comments about how my section probably behaved if I didn't happen to be there for the given discussion. Definitely not OK to say "oh, yeah, Professor Jack Hole from U Whassat was grinding his motherfucking axe on your application....".

Leave a Reply