Grantsmack: The logic of hypothesis testing

(by drugmonkey) Aug 26 2015

NIH grant review obsesses over testing hypotheses. Everyone knows this.

If there is a Stock Critique that is a more reliable way to kill a grant's chances than "There is no discernible hypothesis under investigation in this fishing expedition", I'd like to know what it is.

The trouble, of course, is that once you've been lured into committing to a hypothesis then your grant can be attacked for whether your hypothesis is likely to be valid or not.

A special case of this is when some aspect of the preliminary data that you have included even dares to suggest that perhaps your hypothesis is wrong.

Here's what bothers me. It is one thing if you have Preliminary Data suggesting some major methodological approach won't work. That is, that your planned experiment cannot result in anything like interpretable data that bears on the ability to falsify the hypothesis. This I would agree is a serious problem for funding a grant.

But any decent research plan will have experiments that converge to provide different levels and aspects of testing for the hypothesis. It shouldn't rest on one single experiment or it is a prediction, not a real hypothesis. Some data may tend to support and some other data may tend to falsify the hypothesis. Generally speaking, in science you are not going to get really clean answers every time for every single experiment. If you do.....well, let's just say those Golden Scientist types have a disproportionate rate of being busted for faking data.

So.

If you have one little bit of Preliminary Data in your NIH Grant application that maybe, perhaps is tending to reject your hypothesis, why is this of any different value than if it had happened to support your hypothesis?

What influence should this have on whether it is a good idea to do the experiments to fully test the hypothesis that has been advanced?

Because that is what grant review should be deciding, correct? Whether it is a good idea to do the experiments. Not whether or not the outcome is likely to be A or B. Because we cannot predict that.

If we could, it wouldn't be science.

23 responses so far

Grantsmack: Overambitious

(by drugmonkey) Aug 25 2015

If we are entering a period of enthusiasm for "person, not project" style review of NIH grants, then it is time to retire the criticism of "the research plan is overambitious".

Updated:
There was a comment on the Twitters to the effect that this Stock Critique of "overambitious" is a lazy dismissal of an application. This can use some breakdown because to simply dismiss stock criticisms as "lazy" review will fail to address the real problem at hand.

First, it is always better to think of Stock Critique statements as shorthand rather than lazy.

Using the term "lazy" seems to imply that the applicant thinks that his or her grant application deserves a full and meticulous point-by-point review no matter if the reviewer is inclined to award it a clearly-triagable or a clearly-borderline or clearly-fundable score. Not so.

The primary job of the NIH Grant panel reviewer is most emphatically not to help the PI to funding nor to improve the science. The reviewer's job is to assist the Program staff of the I or C which has been assigned for potential funding decide whether or not to fund this particular application. Consequently if the reviewer is able to succinctly communicate the strengths and weaknesses of the application to the other reviewers, and eventually Program staff, this is efficiency, not laziness.

The applicant is not owed a meticulous review.

With this understood, we move on to my second point. The use of a Stock Criticism is an efficient communicative tool when the majority of the review panel agrees that the substance underlying this review consideration is valid. That is, that the notion of a grant application being overambitious is relevant and, most typically, a deficiency in the application. This is, to my understanding, a point of substantial agreement on NIH review panels.

Note: This is entirely orthogonal to whether or not "overambitious" is being applied fairly to a given application. So you need to be clear about what you see as the real problem at hand that needs to be addressed.

Is it the notion of over-ambition being any sort of demerit? Or is your complaint about the idea that your specific plan is in fact over-ambitious?

Or are you concerned that it is unfair if the exact same plan is considered "over-ambitious" for you and "amazingly comprehensive vertically ascending and exciting" when someone else's name is in the PI slot?

Relatedly, are you concerned that this Stock Critique is being applied unjustifiably to certain suspect classes of PI?

Personally, I think "over-ambitious" is a valid critique, given my pronounced affection for the NIH system as project-based, not person-based. In this I am less concerned about whether everything the applicant has been poured into this application will actually get done. I trust PIs (and more importantly, I trust the contingencies at work upon a PI) of any stage/age to do interesting science and publish some results. If you like all of it, and would give a favorable score to a subset that does not trigger the Stock Critique, who cares that only a subset will be accomplished*?

The concerning issue is that a reviewer cannot easily tell what is going to get done. And, circling back to the project-based idea, if you cannot determine what will be done as a subset of the overambitious plan, you can't really determine what the project is about. And in my experience, for any given application, there are going to usually be parts that really enthuse you as a reviewer and parts that leave you cold.

So what does that mean in terms of my review being influenced by these considerations? Well, I suppose the more a plan creates an impression of priority and choice points, the less concern I will have. If I am excited by the vast majority of the experiments, the less concern I will have-if only 50% of this is actually going to happen, odds are good if I am fired up about 90% of what has been described.

*Now, what about those grants where the whole thing needs to be accomplished or the entire point is lost? Yes, I recognize those exist. Human patient studies where you need to get enough subjects in all the groups to have any shot at any result would be one example. If you just can't collect and run that many subjects within the scope of time/$$ requested, well.....sorry. But these are only a small subset of the applications that trigger the "overambitious" criticism.

41 responses so far

Completely uncontroversial graph preferences

(by drugmonkey) Aug 24 2015

I am sure that nobody has any opinions whatsoever on using the placement of significance symbols to...err....emphasize..... the magnitude of the effect.

9 responses so far

Tales from the search committee

(by drugmonkey) Aug 21 2015

Prof Booty has written about chairing a recent search committee.

Starting a little over a year ago, I served as chair of my department’s search committee, which concluded in the spring with a successful hire. With that experience still relatively fresh, I hope I can share some important insights into how our top candidates caught our eye, as well as the behind-the-scenes process of selecting those candidates.

Go read.

No responses yet

NIH grant applications are not competing with the reviewers!

(by drugmonkey) Aug 18 2015

So misguided. Understandable frustration...but misguided.

Think of it this way- do you dismiss Olympic judging of diving or figure skating because the judges can't do that themselves? What about the scoring of boxing?

Your competition is not the judge. It is the other participants in the event that stand between you and glory.

In NIH grant review, that means the other applications that have been submitted.

31 responses so far

Brief thought on GenX scientists

(by drugmonkey) Aug 18 2015

I detailed some of the ways that my generation of scientist had been screwed in a well received prior post.

Today I thought about another factor. Scientific impact of a scientist is captured by paper citations, which is related to the number of people working within a sphere of investigation. A given scientist's reputation can be burnished by the number of publishing scientists that he or she is respected by and viewed by as a thought leader. 
Scientific progeny are a key factor. The trainees that exit out labs, gain faculty positions and start up vigorous publication trains very frequently boost our own reputations.
When the odds of trainees becoming traditional, independent, academic research scientists are lower for a generation of mentoring scientists, this will cripple the apparent importance and influence of that generation. 
How convenient for the Boomers.

9 responses so far

Runts of the Litter

(by drugmonkey) Aug 18 2015

Sometimes, I page back through my Web of Science list of pubs to the minimal citations range. 

I love all of my papers of course, and feel a little sorry for the ones that never garnered much appreciation. 

19 responses so far

Seriously? Payment for citations?

(by drugmonkey) Aug 14 2015

A Reader submitted this gem of a spam email:

We are giving away $100 or more in rewards for citing us in your publication! Earn $100 or more based on the journal’s impact factor (IF). This voucher can be redeemed your next order at [Company] and can be used in conjunction with our ongoing promotions!

How do we determine your reward?
If you published a paper in Science (IF = 30) and cite [Company], you will be entitled to a voucher with a face value of $3,000 upon notification of the publication (PMID).

This is a new one on me.

36 responses so far

Repost: An Honor Codes' Second Component and Research Science

(by drugmonkey) Aug 12 2015

This was originally posted October 4, 2007.


Many academic honor codes boil down to two essential statements, namely "I will not cheat and I will not tolerate those who do". For "cheat" you may read any number of disreputable activities including plagiarism and research fraud. My alma mater had this sort of thing, I know the US military academies have this. Interestingly a random Google brings up some which include both components (Davidson College, Notre Dames, Florida State Univ (which as been in the academic cheating news lately), and some which do not (CU Boulder, Baylor); Wikipedia entry has a bunch of snippet Honor Codes. The first component, i.e. "don't cheat" is easily comprehended and followed. The second component, the " I will not tolerate those who do" part is the tricky one. Continue Reading »

12 responses so far

Repost: On submitting a grant application from a University you plan to leave

(by drugmonkey) Aug 11 2015

This entry was originally posted 2/9/2011.


  • A bit of confusion has arisen on the Twitts over who can serve as the PI of a grant application submitted to the NIH, who "owns" the award and what the implications are for moving the award to another University.

    For a highly related topic I recommend you re-read my old post Routes to Independence: Beyond Ye Olde Skool Tenure Track Assistant Professorships (original).

    To distill it to a few simple points for the current discussion:

    • The University (or Research Institution, company, etc) submits the grant to the NIH and receives the award from the NIH.
    • Anyone who the submitting institution deems to be a PI can serve as the PI. Job title or status is immaterial as far as the NIH is concerned.
    • Postdocs, Research Scientists, Staff Scientists, etc can be the listed PI on most broad NIH mechanisms (there may be the occasional special case like MD-required or something).
    • The submitting institutions, for the most part, permit anyone of tenure track professorial appointment to prepare NIH grants for them to submit but it gets highly variable (across institutions, across their respective non-professorial and/or non tenure track...and across time) after that.
    • The question of how study sections view applications submitted by those of other than tenure track professorial rank is a whole 'nother question, but you would be making a mistake to think there are hard and fast exclusive principles.

    The second issue has to do with moving the award to another institution, given that a PI on an NIH award decides to go somewhere else. Although technically the University owns the award, in the vast majority of cases that institution will relinquish the award and permit it to travel with the PI. Likewise, in the vast majority of cases, the NIH will permit the move. In all cases I am aware of this move will occur at the anniversary of funding. That is because the award is in yearly increments (maximum of 5 unless you win a PECASE or MERIT extension* of the non-competing interval). Each progress report you submit? That's the "application" for the next year of funding. Noncompeting application, of course, because it does not go back to study section for review. At any rate it makes it less painful for all concerned to do the accounting if the move is at the anniversary.

    Soooooo.....

    Point being that if you are a postdoc or non tenure track scientist who wants to write and submit a grant, you need to start snooping around your local University about their policies. Sometimes they will only let you put in a R21 or R03 or some other nonrenewable mechanism. Sometimes they'll let you throw down the R01. Just depends. Most of the time it will require a letter of exception to be generated within the University- Chair or Dean level stuff. Which requires the approval of your current lab head or supervisor, generally. You need to start talking to all these people.

    Since these types of deals are frequently case-by-case and the rules are unwritten, don't assume that everyone (i.e., your PI) knows about them. Snoop around on RePORTER for awards to your institution and see if anyone with non-TT professorial appointment has ever received an award from the NIH. Follow up on that rumour that Research Scientist Lee once had an award.

    If you are really eager, be prepared to push the envelope and ask the Chair/Dean type person "Well why not? University of State1 and State University2 and IvyUni3 and Research Institute4 all permit it, why can't we?". This may require doing some background surveying of your best buddies spread around the country/world.

    Final point:
    Obviously I wouldn't be bringing up these theoretical possibilities if I hadn't seen it work, and with some frequency. As a reviewer on a study section I saw several applications come through from people who had the title of something below tenure track assistant professor. Instructor, Research Scientist and yes, even Postdoc. I myself submitted at least two R01 applications prior to being able to include the word "Professor" on my Biosketch. I have many peers that were in a similar circumstance at their early stage of grant writing/submitting and, yes, winning.

    No, you will not be treated just like an Assistant Professor by the study sections. You will be beat up for Independence issues and with doubts about whether this is just the BigCheeze trying to evade perceptions of overfunding. You will have "helpful" reviewers busting on your appointment as evidence of a lack of institutional commitment that the reviewer really thinks will get the Dean or Chair to cough up a better title**.

    In all of this however there is a chance. A chance that you will receive an award. This would have very good implications for your transition. (Assuming, of course, that you manage to get the grant written and submitted without too big of a hit to your scientific productivity, never forget that part.) And even if you do not manage to obtain a fundable score, I argue that you get valuable experience. In preparing and submitting a half-decent proposal. In getting some degree of study section feedback. In taking a shot across the bow of the study section that you have ideas and you plan to have them review them in the coming few years. In getting the PO familiar with your name. In wrangling local bureaucracy.

    All of this without your own tenure clock running.
    __
    *there may be other extensions I am unaware of.

    **One of the first questions I asked an experienced reviewer about after joining a study section. Sigh.

  • 18 responses so far

    Older posts »