Grantsmack: The logic of hypothesis testing

Aug 26 2015 Published by under Grant Review, NIH, NIH Careerism

NIH grant review obsesses over testing hypotheses. Everyone knows this.

If there is a Stock Critique that is a more reliable way to kill a grant's chances than "There is no discernible hypothesis under investigation in this fishing expedition", I'd like to know what it is.

The trouble, of course, is that once you've been lured into committing to a hypothesis then your grant can be attacked for whether your hypothesis is likely to be valid or not.

A special case of this is when some aspect of the preliminary data that you have included even dares to suggest that perhaps your hypothesis is wrong.

Here's what bothers me. It is one thing if you have Preliminary Data suggesting some major methodological approach won't work. That is, that your planned experiment cannot result in anything like interpretable data that bears on the ability to falsify the hypothesis. This I would agree is a serious problem for funding a grant.

But any decent research plan will have experiments that converge to provide different levels and aspects of testing for the hypothesis. It shouldn't rest on one single experiment or it is a prediction, not a real hypothesis. Some data may tend to support and some other data may tend to falsify the hypothesis. Generally speaking, in science you are not going to get really clean answers every time for every single experiment. If you do.....well, let's just say those Golden Scientist types have a disproportionate rate of being busted for faking data.

So.

If you have one little bit of Preliminary Data in your NIH Grant application that maybe, perhaps is tending to reject your hypothesis, why is this of any different value than if it had happened to support your hypothesis?

What influence should this have on whether it is a good idea to do the experiments to fully test the hypothesis that has been advanced?

Because that is what grant review should be deciding, correct? Whether it is a good idea to do the experiments. Not whether or not the outcome is likely to be A or B. Because we cannot predict that.

If we could, it wouldn't be science.

45 responses so far

  • Grumble says:

    "If we could, it wouldn't be science."

    Yet this is precisely what grant reviewers seem to demand. Hence the old hack about doing all the experiments first, then writing the grant. That's one way to almost guarantee success: you can pick and choose the experiments to include as preliminary data, such that it looks very promising but leaves out the key experiments that need to be done (which is why you need the grant). Except the experiments are already done and the manuscript is sitting on your desk, ready to be submitted as soon as the NOGA arrives.

    I guarantee you that precisely this strategy is being followed this very minute. Do you STILL claim that funding-people-not-projects would be a worse system that this charade of bullshit?

  • physioprof says:

    Dude, you could have at least h/t me for sharing my motherfucken summary statements with you, one of which was all "this hypothesis can't possibly be correct, but the experiments will provide valuable high-impact knowledge, so fund this fucker", and the other was all "some of these preliminary data seem potentially inconsistent with this hypothesis, so fucke this fucker".

  • drugmonkey says:

    I totally forgot about that. I was thinking about some other situation.

  • drugmonkey says:

    Do you STILL claim that funding-people-not-projects would be a worse system that this charade of bullshit?

    Yes I do.

  • physioprof says:

    Sure you were, fucken dicke.

  • drugmonkey says:

    luckily you corrected the record.

  • Dave says:

    Yet this is precisely what grant reviewers seem to demand. Hence the old hack about doing all the experiments first, then writing the grant. That's one way to almost guarantee success: you can pick and choose the experiments to include as preliminary data, such that it looks very promising but leaves out the key experiments that need to be done (which is why you need the grant). Except the experiments are already done and the manuscript is sitting on your desk, ready to be submitted as soon as the NOGA arrives.

    This has always been my interpretation of 'grantsmanship', honestly. The rest is just formatting, isn't it?

    I like my chances of competing in a 'projects' based system rather than a 'people' based system. At least it gives one hope....

  • physioprof says:

    Did you correct your shitty attribution practice in your fucken post asshole?

  • Drugmonkey says:

    There is no attribution required if I forgot about your summary statement. ( Perhaps you can appeal to the management if you are unsatisfied. )

  • qaz says:

    This is all part of a deeper problem in science today - that many people no longer think it is OK to be wrong. This is why study section wants data suggesting a positive outcome. But also many scientists are so associated with a specific hypothesis that if that hypothesis turns out to be wrong, they worry (perhaps correctly?) that the rest of science will no longer listen to them.

    This, of course, is completely backwards.

  • potnia theron says:

    'If there is a Stock Critique that is a more reliable way to kill a grant's chances than "There is no discernible hypothesis under investigation in this fishing expedition", I'd like to know what it is. '

    Up there is "Concerns exist as to the feasibility of the experimental portion of this project given the PI's _____ (youth, age, lab, experience, model system). "

  • […] lose-lose situation of NIH grant applications with preliminary […]

  • crystaldoc says:

    "Except the experiments are already done and the manuscript is sitting on your desk, ready to be submitted as soon as the NOGA arrives."

    Yeah I've heard this nonsense but I think it is silliness; it can't possibly work in common practice. Academic urban legend. Given the relatively low chance of a given application being funded on first swing, you are seriously going to have a complete manuscript sitting on your desk for 18+ months, gathering dust and getting stale, waiting to be scooped? And you think the opportunity to so heavily curate your prelim data, having sekrit finished study up your sleeve, is going to help you more than just publishing the thing already,improving your documented productivity and success with the methods, models, etc, and then proposing the next step? yeah right.

  • jmz4 says:

    Probably more of a mix. You put in the application, start doing the experiments, but by the time you've gone through the requisite 3 cycles, you've probably got the paper pretty much done.

  • Philapodia says:

    @JMZ4
    "You put in the application, start doing the experiments, but by the time you've gone through the requisite 3 cycles, you've probably got the paper pretty much done."

    You should be thanking the reviewers for not funding your applications as they are giving you strong incentive to get papers published that otherwise might not be. You're welcome.

  • Grumble says:

    It's not quite an urban legend. I've done it myself - OK, more like the hybrid jmz4 described. Basically, by the time of the 2nd submission, I had everything I needed for a good paper. I didn't hold off on submission on purpose, but rather because it takes quite a bit of time to put a paper together. So by the time it was all ready to submit, the NOGA came through. The bottom line is that I got the grant because the project was already at a stage where the "preliminary data" was the vast majority of a really solid data-heavy paper.

  • Mikka says:

    And with how long papers are taking for peer review these days, by the time it's accepted you can put the grant number in the acknowledgements and it will even look like the grant helped you get the data.

  • jmz4gtu says:

    "You should be thanking the reviewers for not funding your applications as they are giving you strong incentive to get papers published that otherwise might not be. You're welcome.
    -It's like a donkey being led by the carrot. I should just be happy I'm not getting the stick, huh?

    Oh, btw, what's a NOGA?

  • drugmonkey says:

    Notice of grant award. I tend to use NGA, personally. It ain't real until this document is issued.

  • drugmonkey says:

    jmz4 is right that many research programs just forge forward simultaneously on both agendas. Grants and papers. As you do this, yes, sometimes the story that is strong for a proposal is mature enough to be published at the time the award funds. Should o e strategically delay the publication process by a few months here or there to assist grant chances? That is going to be a one by one decision. You have to use your best guess about grant review.

    As far as waiting to be able to put the funded grant number on it....yeah, do this if you know the grant is gong to be awarded. Sure. Why not. Not every reviewer of a continuation application complains about the grant funding date and how silly it is to attribute a paper published two months later to that award.....

  • crystaldoc says:

    As a grant reviewer, I find that often the submissions that do well as A1 (especially for relatively more junior PIs) did not have the paper published at time of 1st submission, got some critiques possibly relating to feasibility and/or inadequate evidence in support of hypothesis and/or inadequate experience in subfield or with model or whatever, and then come back in the A1 pointing to the paper that they have just had published that establishes their new paradigm as a thing. Best for them is when it got into a relatively high impact journal. This tends to play really well with reviewers -- yeah, they don't ultimately get to claim it as productivity on the grant, but hey, at least they get the grant. They adjust the aims a little bit to say they completed this part already and now moving on to the next step. I am confident those A1 apps would NOT do as well with the ms still sitting on the desk. This is especially true for the more junior investigators-- BSDs with long track record are much less dependent on a single paper published. If you are a new investigator, or midcareer but moving into a field, model, disease without a personal track record of publications, I think it would be a terrible strategy to delay publishing and hope to get a grant.

  • Grumble says:

    Write "NGA" and pronounce the abbreviation (e.g., as you would for OPEC or NAFTA). Now do the same for "NOGA." Now you see why "NOGA" is better.

  • Joe says:

    As a reviewer, I would like to say Please, please include a stated hypothesis in your grant proposal. Sometimes I can't tell why the applicant wants to do the proposed experiments, and a hypothesis helps clarify that.
    As someone who writes grants, I think that if your preliminary data suggests your hypothesis might be wrong, then you need to work on how you state your hypothesis. Surely there is a way to state your hypothesis such that it encompasses the possible outcomes of your experiments while still making clear what you are doing and why.

  • […] “If there is a Stock Critique that is a more reliable way to kill a grant’s chances than ‘There is no discernible hypothesis under investigation in this fishing expedition’, I’d like to know what it is.” DrugMonkey angsts over NIH grant review. […]

  • The Other Dave says:

    You're right, DM, that falsifying one's hypothesis is still good science. But if it looks like your hypothesis is going to be disproven, then a lot of taxpayer money will have been spent on learning nothing.

    Reviewers want to pay for *new knowledge*. Your proposal needs to promise new knowledge, not just good science.

    Got it? Good.

  • Philapodia says:

    "But if it looks like your hypothesis is going to be disproven, then a lot of taxpayer money will have been spent on learning nothing."

    @TOD I wholeheartedly agree. This is what a lot of people forget, that we're asking for taxpayer money to do our research. The mission of the NIH is to improve human health. Work funded by the NIH should provide a service to the country and to the NIH's mission, not just be used to let scientists follow their lark. When we purchase something from the store we expect something of value in return for our money. Why should it be any different when the NIH is the customer and we're selling them our science? You want value for your money, and so does the NIH. If you want to study squirrel thermoregulation, go talk to the NSF since their mission is to expand science.

  • Juan Lopez says:

    TOD, it's not true that disproving a hypothesis teaches us nothing. Most often we could formulate the hypothesis in the opposite way. So proving or disproving the hypothesis is far less different than you make it look. I agree on the importance of generating knowledge and in supporting the NIH mission.
    Philapodia, what you said is a Palinism. Thermoregulation is critical to human life. I wouldn't assume that squirrel thermoregulation is useless. I would have to be more familiar with the field before discounting it as useless for NIHs mission.

  • Philapodia says:

    "Philapodia, what you said is a Palinism. Thermoregulation is critical to human life. I wouldn't assume that squirrel thermoregulation is useless. I would have to be more familiar with the field before discounting it as useless for NIHs mission."

    I didn't say that it was useless, just that it didn't have a direct relationship to human health. This was in reference to McKnight's ASBMB article last month, where squirrel thermoregulation was touted as something interesting in and of its self and not as a model system for human thermoregulation. Obviously thermoregulation is critical to human life, but NIH's mission isn't to protect squirrels. We can almost always make an argument for why some esoteric thing we want to study has some relevance to human health, but the question is are we getting good value for the taxpayer's dollar? This is why funding projects is important, since it allows outside reviewers to call BS on fairy dust projects with little relevance to the NIH mission.

  • As far as waiting to be able to put the funded grant number on it....yeah, do this if you know the grant is gong to be awarded. Sure. Why not. Not every reviewer of a continuation application complains about the grant funding date and how silly it is to attribute a paper published two months later to that award.....

    PRE-AWARD COSTS!!!!!!!!!!!!!!!

  • Juan Lopez says:

    Exactly, Philapodia. You dismissed the topic as esotheric and fairy dust without knowing the rationale for studying it. Because it does not look important TO YOU on the surface, you conclude that the argument of relevance must be a stretch. Hence, it is not good value for taxpayers to study this. It is a textbook example of a Palinism.

    Palin mocked the study of fruit flies using exactly the same arguments you do. NIH's mission is not to protect fruit flies... I can see Russia from my window. I am so glad she didn't win. Things are difficult enough without that.

    Note, I have no idea what McKnight thinks about Thermoregulation. I didn't see that discussion. I also agree on the need to fund applied work with NIH dollars. My argument us that we cannot dismiss something as fairy dust because WE don't know how it is related to human health.

    I also note that I really like reading your comments. Other than this one, where you are clearly just wrong 🙂

  • Philapodia says:

    @JL I don't think I'm wrong, I just think that we're talking about different things here. I'm advocating for rigorous review of projects (ie fund projects not people) to make sure that the funding will go towards something that is applicable to human health. The McKnight article was about funding vertically ascending scientists without direct review of what the project would be, with the assumption that letting elite scientists follow their lark obviously will result in groundbreaking research (people-not-projects). If your science is not reviewed, how can we know the NIH is not getting fairy dust for its money? That's all I'm saying.

    Please don't think that I'm hating on squirrels. I love squirrels. I have binders full of squirrels.

  • qaz says:

    Squirrell thermoregulation is the bunny hopping of our day...

    Philapodia don't go all "translational" on us. The whole point of NIH was (and is) to do research important to the fields that impact human health... eventually.... Basic science on stuff like squirrell thermoregulation is important BECAUSE we don't know where it will lead. All of the impact I have had on clinical practice (and there has been some) is due completely to esoteric research on other issues that turned out to have surprising outcomes.

    Science has a 30 year translational process. It has been ever thus, at least since the late 1800s, and probably earlier. This is true across all of the sciences, from physics to biology to the information sciences. The translational breakthroughs that are happening now are due to the esoteric research done in 1985. If we want any translational results in 2045, we have to be working on the esoteric stuff now. This is why NIH exists. It is not worth it to profit-driven companies to invest in basic (esoteric) research because it doesn't tend to come back to them quickly enough to be profitable. Therefore basic research is a public good and requires public investment, otherwise known as taxpayer dollars.

  • drugmonkey says:

    Cling to that tired mantra dude. Grasp it hard.

  • drugmonkey says:

    A solid year's worth of pre-award costs, PP? Mmmhmmmm.

  • Philapodia says:

    @Qaz
    "If we want any translational results in 2045, we have to be working on the esoteric stuff now."

    To what end? How do we pick and choose which esoteric area of research is the best bet when dollars are limited? The NIH mission as stated is "NIH’s mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability." (emphasis mine). All I'm asking for is a modicum, a bare minimum, even a smidge, of effort to relate why a particular project is relevant to human health. People-not-project funding circumvents this, whereas funding projects allows some oversight.

    Also, how has "translational research" become a dirty phrase? There is an implication that if you want to use scientific knowledge to make something that will be of direct use to the public, your work isn't pure enough. I work with a lot of engineers, so perhaps I have a more practical point of view in terms of what I think is valuable.

  • physioprof says:

    "A solid year's worth of pre-award costs, PP? Mmmhmmmm."

    What is the relevance of the quantity of pre-award costs expended to whether one can list a grant on a publication as having supported the research reported in that publication? One dollar is sufficient.

  • A Salty Scientist says:

    "To seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability." My bias as a basic researcher is to emphasize the first part. Certainly the goal of NIH is to support both basic and translational research. What should be the relative proportion, and how basic is too far outside the NIH mission? I will emphasize that I'm much more worried about basic research in our current political climate, with its particularly nasty anti-intellectual streak.

  • jmz4 says:

    The biggest danger to the future of basic research comes from the fact that the younger generation of scientists are shying away from it precisely because of the current funding environment. Going into basic research means you will have limited funding opportunities and no applicable skills if you want to move into industry.
    I imagine basic researchers are having trouble getting experienced help, as well as grants. And it only snowballs as this next generation matriculates into study sections, since they'll naturally be less disposed to fund those kinds of projects.

  • drugmonkey says:

    So all y'all basic researchers parse the Second Amandment how? Looking at you, PP, especially.

  • Juan Lopez says:

    Philapodia, on the one hand you ask for a minimum of application, but on the other, you argued that almost anything can be claimed to be related to application. So, essentially, you are asking for people to make the exercise to twist things into applicability, which you will later not believe when reviewing their grant applications.

  • Philapodia says:

    @JL

    I ask applicants to think about how what they are doing could be applied to human health, which is the mission of the NIH that we are reviewing grants for. Please see the stated NIH mission above. I'm not seeing why this is a bad thing. You are trying to make sure that the applicant is giving the NIH what they want, which is knowledge that can help human health. If you can come up for a compelling and logical reason that studying squirrel thermoregulation has some bearing on human health, then I'm willing to consider it. For example:

    "Squirrels are a primary vector for bubonic plague on ILAF campuses, primarily transmitting the pathogen while dropping acorns on BSDs lost in vertically ascending thought. The transmission of plague to BSDs is inversely correlated (p > 0.49999) with squirrel internal temperature, suggesting that squirrel thermoregulation is a critical variable in the recent epidemic of plague in the BSD population."

    This is something that I would look at and think "Huh?! That's an angle that I hadn't thought of. Let's read more and see if we can save the BSDs!" I didn't see how squirrel thermoregulation could have a link to human health (because who would, really), but here is a good (fake) reason! Booyah!! I now know of this link because the applicant educated me as a reviewer about the significance of the topic. The reviewer is not obligated to teach themselves about why your topic is important, that's the job of the applicant. Therefore, if you want money to study squirrel thermoregulation, you need to convince me why.

    If the applicant cannot come up with some even tortured, twisted reasoning for why squirrel thermoregulation is somehow tangentially related to human health (which is again the mission of the NIH that we are reviewing grants for), then I will feel that the project isn't targeted to the right funding agency and won't score well. If it's not related to human health, then it should go to NSF, a private foundation, or go for crowdsource funding.

    I don't have anything at all against basic research. I just think that the research should be in line with the mission of the funding agency. And when it's something like squirrel thermoregulation, which could be used as a political weapon by the Palin types (which I assure you I am not), it needs to have a strong and logical rationale for funding so that Palin-types have a harder time using it. The "because it's interesting" argument won't cut it.

  • Grumble says:

    "So all y'all basic researchers parse the Second Amandment how?"

    I believe the Founders' intention was something like, "Shoot all applied researchers."

  • Juan Lopez says:

    ok. Looks like we do agree on several things.

  • qaz says:

    Philopodia - Yes, we need to find the best research, but I would argue that the criterion for basic science needs to be about whether the research is likely to move the basic knowledge of the human condition forward the most, NOT the stuff that's going to cure the latest disease.

    You want a justification for squirrel thermoregulation, convince me that we don't completely understand thermoregulation (in general, which is obviously important to human health) and that there's something you can ask about squirrel thermoregulation that you can't ask using some other system. Then fund it.

    Translation is important. But we need stuff aimed at the 5 year timeline (make the treatment really work safely), the 10 year timeline (develop the treatment), the 15 to 20 year timeline (what sorts of treatment are there), AND the 30 year timeline (esoteric stuff that has no obvious treatment importance).

    The problem is that as funding gets tight, people start demanding (at least publicly*) that every research question has to be in the 10-year timeline or less (develop the treatment, study the treatment, make the treatment work). Part of the point of government is to fund public goods that are too valuable to lose but are not going to help the immediate funder directly. So we need NIH to be funding that esoteric 30-year stuff.

    * The funny thing about all of this discussion, is that the basic-science study sections I've been on funds lots of BSDs and lots of newbies on really phenomenal basic science work, the justification of which is often "Disease X affects brain structure Z. I have a new way of looking at how brain structure Z works." And then disease X never appears again. So I don't see this translation-mania as actually affecting the study sections I sit on.

  • drugmonkey says:

    I think what you mean qaz is that when funding gets tight, self-interested PIs start screaming that their type of research is under specific attack by Program and study sections. Without any evidence for that, and with plenty of evidence suggesting everyone is hurting.

Leave a Reply