Archive for the 'NIH Careerism' category

Grantsmack: The logic of hypothesis testing

Aug 26 2015 Published by under Grant Review, NIH, NIH Careerism

NIH grant review obsesses over testing hypotheses. Everyone knows this.

If there is a Stock Critique that is a more reliable way to kill a grant's chances than "There is no discernible hypothesis under investigation in this fishing expedition", I'd like to know what it is.

The trouble, of course, is that once you've been lured into committing to a hypothesis then your grant can be attacked for whether your hypothesis is likely to be valid or not.

A special case of this is when some aspect of the preliminary data that you have included even dares to suggest that perhaps your hypothesis is wrong.

Here's what bothers me. It is one thing if you have Preliminary Data suggesting some major methodological approach won't work. That is, that your planned experiment cannot result in anything like interpretable data that bears on the ability to falsify the hypothesis. This I would agree is a serious problem for funding a grant.

But any decent research plan will have experiments that converge to provide different levels and aspects of testing for the hypothesis. It shouldn't rest on one single experiment or it is a prediction, not a real hypothesis. Some data may tend to support and some other data may tend to falsify the hypothesis. Generally speaking, in science you are not going to get really clean answers every time for every single experiment. If you do.....well, let's just say those Golden Scientist types have a disproportionate rate of being busted for faking data.

So.

If you have one little bit of Preliminary Data in your NIH Grant application that maybe, perhaps is tending to reject your hypothesis, why is this of any different value than if it had happened to support your hypothesis?

What influence should this have on whether it is a good idea to do the experiments to fully test the hypothesis that has been advanced?

Because that is what grant review should be deciding, correct? Whether it is a good idea to do the experiments. Not whether or not the outcome is likely to be A or B. Because we cannot predict that.

If we could, it wouldn't be science.

12 responses so far

Grantsmack: Overambitious

Aug 25 2015 Published by under Grant Review, NIH, NIH Careerism, NIH funding

If we are entering a period of enthusiasm for "person, not project" style review of NIH grants, then it is time to retire the criticism of "the research plan is overambitious".

Updated:
There was a comment on the Twitters to the effect that this Stock Critique of "overambitious" is a lazy dismissal of an application. This can use some breakdown because to simply dismiss stock criticisms as "lazy" review will fail to address the real problem at hand.

First, it is always better to think of Stock Critique statements as shorthand rather than lazy.

Using the term "lazy" seems to imply that the applicant thinks that his or her grant application deserves a full and meticulous point-by-point review no matter if the reviewer is inclined to award it a clearly-triagable or a clearly-borderline or clearly-fundable score. Not so.

The primary job of the NIH Grant panel reviewer is most emphatically not to help the PI to funding nor to improve the science. The reviewer's job is to assist the Program staff of the I or C which has been assigned for potential funding decide whether or not to fund this particular application. Consequently if the reviewer is able to succinctly communicate the strengths and weaknesses of the application to the other reviewers, and eventually Program staff, this is efficiency, not laziness.

The applicant is not owed a meticulous review.

With this understood, we move on to my second point. The use of a Stock Criticism is an efficient communicative tool when the majority of the review panel agrees that the substance underlying this review consideration is valid. That is, that the notion of a grant application being overambitious is relevant and, most typically, a deficiency in the application. This is, to my understanding, a point of substantial agreement on NIH review panels.

Note: This is entirely orthogonal to whether or not "overambitious" is being applied fairly to a given application. So you need to be clear about what you see as the real problem at hand that needs to be addressed.

Is it the notion of over-ambition being any sort of demerit? Or is your complaint about the idea that your specific plan is in fact over-ambitious?

Or are you concerned that it is unfair if the exact same plan is considered "over-ambitious" for you and "amazingly comprehensive vertically ascending and exciting" when someone else's name is in the PI slot?

Relatedly, are you concerned that this Stock Critique is being applied unjustifiably to certain suspect classes of PI?

Personally, I think "over-ambitious" is a valid critique, given my pronounced affection for the NIH system as project-based, not person-based. In this I am less concerned about whether everything the applicant has been poured into this application will actually get done. I trust PIs (and more importantly, I trust the contingencies at work upon a PI) of any stage/age to do interesting science and publish some results. If you like all of it, and would give a favorable score to a subset that does not trigger the Stock Critique, who cares that only a subset will be accomplished*?

The concerning issue is that a reviewer cannot easily tell what is going to get done. And, circling back to the project-based idea, if you cannot determine what will be done as a subset of the overambitious plan, you can't really determine what the project is about. And in my experience, for any given application, there are going to usually be parts that really enthuse you as a reviewer and parts that leave you cold.

So what does that mean in terms of my review being influenced by these considerations? Well, I suppose the more a plan creates an impression of priority and choice points, the less concern I will have. If I am excited by the vast majority of the experiments, the less concern I will have-if only 50% of this is actually going to happen, odds are good if I am fired up about 90% of what has been described.

*Now, what about those grants where the whole thing needs to be accomplished or the entire point is lost? Yes, I recognize those exist. Human patient studies where you need to get enough subjects in all the groups to have any shot at any result would be one example. If you just can't collect and run that many subjects within the scope of time/$$ requested, well.....sorry. But these are only a small subset of the applications that trigger the "overambitious" criticism.

37 responses so far

NIH grant applications are not competing with the reviewers!

Aug 18 2015 Published by under NIH Careerism

So misguided. Understandable frustration...but misguided.

Think of it this way- do you dismiss Olympic judging of diving or figure skating because the judges can't do that themselves? What about the scoring of boxing?

Your competition is not the judge. It is the other participants in the event that stand between you and glory.

In NIH grant review, that means the other applications that have been submitted.

31 responses so far

Repost: Don't tense up

Aug 07 2015 Published by under Careerism, Grant Review, Grantsmanship, NIH, NIH Careerism

I've been in need of this reminder myself in the past year or so. This originally went up on the blog 25 September, 2011.


If you've been going through a run of disappointing grant reviews punctuated by nasty Third Reviewer comments, you tend to tense up.

Your next proposals are stiff...and jam packed with what is supposed to be ammunition to ward off the criticisms you've been receiving lately. Excessive citation of the lit to defend your hypotheses...and buffer concentrations. Review paper level exposition of your logical chain. Kitchen sink of preliminary data. Exhaustive detail of your alternate approaches.

The trouble is, then your grant is wall to wall text and nearly unreadable.

Also, all that nitpicky stuff? Sometimes it is just post hoc justification by reviewers who don't like the whole thing for reasons only tangentially related to the nits they are picking.

So your defensive crouch isn't actually helping. If you hook the reviewer hard with your big picture stuff they will often put up with a lot of seeming StockCritique bait.

25 responses so far

Will Obama's revamp of the overtime rules mean postdocs are paid more?

Jul 06 2015 Published by under Careerism, NIH Careerism, Postdoctoral Training

Justin Kiggins has launched the discussion at The Spectroscope.

Those postdocs who are salaried employees, however, are currently "exempt" from overtime pay if they make more than $23,660. The new rules mean that they will need a salary of at least $50,400. So if their institution is following the NIH standard, which sets a minimim of $42,840, it looks like they'll either need to get paid overtime for any work done over 40 hours or get a raise to meet the exemption requirements

Go on over and read, especially for the links to places you can comment on this rule prior to implementation.

184 responses so far

Grumble of the Day

Jul 06 2015 Published by under NIH, NIH Careerism, NIH funding

I still get irritated every time a PO gives me some grant advice or guidance that is discordant with my best understanding of the process. It's not so much that I take it seriously for my own strategy...I've been around this block once or twice. 

What kills me is thinking that there are poor newcomer applicants who get this advice and may think it is Gospel. This would then lead them into making suboptimal strategic or tactical decisions.

Related Reading:
POs do not understand...

28 responses so far

On recruiting peers to review NIH grants

Jun 30 2015 Published by under NIH, NIH Careerism

May 2015 Advisory Council round for the CSR of NIH.

I'll be making observations on the Luci Roberts presentation in a little while. For now, enjoy.

UPDATE:

Okay, down to business. The part I wanted to highlight starts at 1:35 of the videocast. Dr. Nakamura introduces Luci Roberts, Director of Planning and Evaluation in the OER.

A comment from baltogirl

It is a little known fact that SROs have trouble recruiting for study section (I was told that two-thirds of people asked decline to serve). It's likely that most people are so busy writing their own grants they can't take a full month off to devote to reading the work of others.

reminded me I forgot to discuss these data.

The Division of Planning and Evaluation conducted a survey** some time ago, trying to determine the attitudes of extramural PIs that would affect their willingness to serve on study sections. I thought it applied to the above comment.
They surveyed 4,000 individuals who had submitted at least one grant as PI in the past five years and who had obtained active funding (any variety) from the NIH in the past five years. 1830 or 46% responded. Not too shabby. (They also surveyed 423 SROs of which 271/64% responded, more on that later.) Of the PIs 1,616 had served as PIs or PDs, the balance were TG directors or subproject/consortium PIs.

First up. 964 (53%) had never been asked to serve on study section in the 12 months prior to the survey. Hmm. I can almost stop right here. But no....there's much more. Still, if there is a reviewer crunch, the first order of business is to determine why over half the potential pool is not even being asked.

Ok, of the remaining 866 individuals who were asked to serve on study section, 762 agreed to do so. That's 88% saying yes. So the rumor that "two-thirds of people asked decline to serve" is falsified by this survey. Clearly, the vast majority of people who are asked step up and do their community duty.

I really, really like this. It is heartening.

The next most-interesting thing was the 10th slide which shows the ranks of PIs who are asked/not asked to review. As you would expect, the Full/Associate/Assistant Professor ranks for those asked ran 54%/32%/11% (that high for Assistants?) and 22%/23%/37% for those not asked to serve.

Again, this outcome makes it really clear what needs to be done if getting reviewers is a problem. Ask more Assistant Professors to serve***. Right? Done.

Then we get into a couple of slides on why people might say "no" when asked to review. Slide 11 present the top reasons (out of an open text box response, per Roberts' presentation) for the process being "more burdensome than it could be". The numbers are confused here because the denominator appears to be 861 when it should be 762. But in any case, 630 of the respondents who reviewed said the process was not more burdensome than necessary (huh? surprised on this one). Of the 158 who said it was too burdensome, 45% complained about the number of assigned applications. The next most common (16%) complaint was "too much time devoted to applications that will never fund". So pretty much, the most burdensome thing was review load.

This brings us to Slide 13 which pits SRO opinion versus how reviewers think. One of the biggest disconnects was in the number of in-person meetings per year that is "reasonable" since reviewers lean 1-2 and SROs were about evenly split between 1-2 and 3-4 as okay. A similar disconnect was found on application load. Half of the reviewers felt 4-6 apps was a reasonable load and only 25% felt 7-9 was reasonable. SROs leaned 7-9 (~60% of SROs) with less than 40% finding a 4-6 grant load reasonable.

Predictably.

I'm skipping Slide 12 on reviewer and SRO thoughts on reason to accept and decline invitations to review since 88% of those asked say yes anyway. Who cares what they say if most of them will do it just for the asking?

Okay, those were the things that jumped out at me.

___
*As a disclosure, Dr. Roberts was once a SRO who played a highly formative role in the early-career understanding of the NIH grant business for YHN. She was kind enough to send me the slide deck that was used in her presentation.

**I was a respondent, fwiw.

***In case of any newcomers, both YHN and CPP have advocated for this on this blog since forever.

16 responses so far

Question of the day

Jun 29 2015 Published by under Fixing the NIH, NIH, NIH Careerism

What percentage of K99 should fail to transition to the R00 phase in a healthy system?
What percentage of those that go to R00 should fail to ever gain major independent funding as a PI? 

38 responses so far

A Tweet which captures the problem with NIH's "pipeline" response to Ginther

Jun 29 2015 Published by under NIH, NIH Careerism

14 responses so far

Peer review and the death sentence

Jun 25 2015 Published by under NIH, NIH Careerism

It is relatively easy to kill grant applications because the reviewer knows the applicant can always apply again.

Same thing for hiring decisions because surely some lesser University will hire the three other candidates on the short list.

In many tenure cases, the Department knows that this person will get a professorial rank job elsewhere*.

Germain's scheme is going to require peers in the field to pass a death sentence on the career. And to make the numbers add up, there will be a LOT of this.

Those peers know that they themselves will be up for chopping in the next 5-7 years.

I do not foresee much enthusiasm for scoring progress as deficient and this will only grow more intense with each successive 5 year review interval.

__
*and anyway these are so personal at this point that it is a different matter.

15 responses so far

Older posts »