Archive for the 'NIH' category

Grantsmack: The logic of hypothesis testing

Aug 26 2015 Published by under Grant Review, NIH, NIH Careerism

NIH grant review obsesses over testing hypotheses. Everyone knows this.

If there is a Stock Critique that is a more reliable way to kill a grant's chances than "There is no discernible hypothesis under investigation in this fishing expedition", I'd like to know what it is.

The trouble, of course, is that once you've been lured into committing to a hypothesis then your grant can be attacked for whether your hypothesis is likely to be valid or not.

A special case of this is when some aspect of the preliminary data that you have included even dares to suggest that perhaps your hypothesis is wrong.

Here's what bothers me. It is one thing if you have Preliminary Data suggesting some major methodological approach won't work. That is, that your planned experiment cannot result in anything like interpretable data that bears on the ability to falsify the hypothesis. This I would agree is a serious problem for funding a grant.

But any decent research plan will have experiments that converge to provide different levels and aspects of testing for the hypothesis. It shouldn't rest on one single experiment or it is a prediction, not a real hypothesis. Some data may tend to support and some other data may tend to falsify the hypothesis. Generally speaking, in science you are not going to get really clean answers every time for every single experiment. If you do.....well, let's just say those Golden Scientist types have a disproportionate rate of being busted for faking data.


If you have one little bit of Preliminary Data in your NIH Grant application that maybe, perhaps is tending to reject your hypothesis, why is this of any different value than if it had happened to support your hypothesis?

What influence should this have on whether it is a good idea to do the experiments to fully test the hypothesis that has been advanced?

Because that is what grant review should be deciding, correct? Whether it is a good idea to do the experiments. Not whether or not the outcome is likely to be A or B. Because we cannot predict that.

If we could, it wouldn't be science.

41 responses so far

Grantsmack: Overambitious

Aug 25 2015 Published by under Grant Review, NIH, NIH Careerism, NIH funding

If we are entering a period of enthusiasm for "person, not project" style review of NIH grants, then it is time to retire the criticism of "the research plan is overambitious".

There was a comment on the Twitters to the effect that this Stock Critique of "overambitious" is a lazy dismissal of an application. This can use some breakdown because to simply dismiss stock criticisms as "lazy" review will fail to address the real problem at hand.

First, it is always better to think of Stock Critique statements as shorthand rather than lazy.

Using the term "lazy" seems to imply that the applicant thinks that his or her grant application deserves a full and meticulous point-by-point review no matter if the reviewer is inclined to award it a clearly-triagable or a clearly-borderline or clearly-fundable score. Not so.

The primary job of the NIH Grant panel reviewer is most emphatically not to help the PI to funding nor to improve the science. The reviewer's job is to assist the Program staff of the I or C which has been assigned for potential funding decide whether or not to fund this particular application. Consequently if the reviewer is able to succinctly communicate the strengths and weaknesses of the application to the other reviewers, and eventually Program staff, this is efficiency, not laziness.

The applicant is not owed a meticulous review.

With this understood, we move on to my second point. The use of a Stock Criticism is an efficient communicative tool when the majority of the review panel agrees that the substance underlying this review consideration is valid. That is, that the notion of a grant application being overambitious is relevant and, most typically, a deficiency in the application. This is, to my understanding, a point of substantial agreement on NIH review panels.

Note: This is entirely orthogonal to whether or not "overambitious" is being applied fairly to a given application. So you need to be clear about what you see as the real problem at hand that needs to be addressed.

Is it the notion of over-ambition being any sort of demerit? Or is your complaint about the idea that your specific plan is in fact over-ambitious?

Or are you concerned that it is unfair if the exact same plan is considered "over-ambitious" for you and "amazingly comprehensive vertically ascending and exciting" when someone else's name is in the PI slot?

Relatedly, are you concerned that this Stock Critique is being applied unjustifiably to certain suspect classes of PI?

Personally, I think "over-ambitious" is a valid critique, given my pronounced affection for the NIH system as project-based, not person-based. In this I am less concerned about whether everything the applicant has been poured into this application will actually get done. I trust PIs (and more importantly, I trust the contingencies at work upon a PI) of any stage/age to do interesting science and publish some results. If you like all of it, and would give a favorable score to a subset that does not trigger the Stock Critique, who cares that only a subset will be accomplished*?

The concerning issue is that a reviewer cannot easily tell what is going to get done. And, circling back to the project-based idea, if you cannot determine what will be done as a subset of the overambitious plan, you can't really determine what the project is about. And in my experience, for any given application, there are going to usually be parts that really enthuse you as a reviewer and parts that leave you cold.

So what does that mean in terms of my review being influenced by these considerations? Well, I suppose the more a plan creates an impression of priority and choice points, the less concern I will have. If I am excited by the vast majority of the experiments, the less concern I will have-if only 50% of this is actually going to happen, odds are good if I am fired up about 90% of what has been described.

*Now, what about those grants where the whole thing needs to be accomplished or the entire point is lost? Yes, I recognize those exist. Human patient studies where you need to get enough subjects in all the groups to have any shot at any result would be one example. If you just can't collect and run that many subjects within the scope of time/$$ requested, well.....sorry. But these are only a small subset of the applications that trigger the "overambitious" criticism.

41 responses so far

NIH grant applications are not competing with the reviewers!

Aug 18 2015 Published by under NIH Careerism

So misguided. Understandable frustration...but misguided.

Think of it this way- do you dismiss Olympic judging of diving or figure skating because the judges can't do that themselves? What about the scoring of boxing?

Your competition is not the judge. It is the other participants in the event that stand between you and glory.

In NIH grant review, that means the other applications that have been submitted.

31 responses so far

Repost: Don't tense up

Aug 07 2015 Published by under Careerism, Grant Review, Grantsmanship, NIH, NIH Careerism

I've been in need of this reminder myself in the past year or so. This originally went up on the blog 25 September, 2011.

If you've been going through a run of disappointing grant reviews punctuated by nasty Third Reviewer comments, you tend to tense up.

Your next proposals are stiff...and jam packed with what is supposed to be ammunition to ward off the criticisms you've been receiving lately. Excessive citation of the lit to defend your hypotheses...and buffer concentrations. Review paper level exposition of your logical chain. Kitchen sink of preliminary data. Exhaustive detail of your alternate approaches.

The trouble is, then your grant is wall to wall text and nearly unreadable.

Also, all that nitpicky stuff? Sometimes it is just post hoc justification by reviewers who don't like the whole thing for reasons only tangentially related to the nits they are picking.

So your defensive crouch isn't actually helping. If you hook the reviewer hard with your big picture stuff they will often put up with a lot of seeming StockCritique bait.

25 responses so far

Salary Cap and the BRDPI Inflation Estimate

Jul 16 2015 Published by under Fixing the NIH, NIH Budgets and Economics

Wow. I last used the BRDPI estimate of inflation in the cost of biomedical research to illustrate how the full modular grant ($250K direct) had not changed and therefore purchasing power had eroded.

Jeff Mervis at Science has a blockbuster observation.

To remind you, the BRDPI is this:

The annual change in the Biomedical Research and Development Price Index (BRDPI) indicates how much the NIH budget must change to maintain purchasing power. The BRDPI was developed and is updated annually by the Bureau of Economic Analysis (BEA), Department of Commerce under an interagency agreement with the NIH.

That link also leads you to the data tables where you find the annual rates stretching back to the 50s.

The Mervis article highlights the historical low for FY2012 and reminds us of the cut in the salary cap (maximum amount of an Investigator's salary that can be charged to NIH grants)

Congress passed a spending bill in December 2011 that lowered the salary ceiling for investigators on a standard NIH grant from $199,700 to $179,700.

and concludes with a caution:

NIH enjoys strong support in Congress, and the realization that biomedical inflation largely tracks salary trends, not the sticker price of essential lab equipment and supplies, is unlikely to have a major impact on policy debates. Still, it may behoove biomedical lobbyists to think twice before citing the cost of high-tech science as a rationale for pumping up NIH's budget.

Yeah, I hear that. Let's peer a little closer though.

The NIH Office of the Budget January 2015 overview [PDF] futher anticipates that salary cap is a major driver of the inflation index.

The modest BRDPI growth rate of 2.0 percent for FY 2014 reflects the effect of the NIH extramural investigator salary limitation (“cap”) of $181,500 and an increase on salaries of Federal civilian employees of 0.75 percent for that fiscal year.
The projected 2.2 percent growth for FY 2015 assumes a one percent increase for Federal salaries starting in January 2015, as well as an increase on the extramural investigator salary cap to $183,300.

OK, let's route ourselves back to the NIH Office of the Budget report from January 2013 [PDF] which indeed draws an explicit link.

The modest BRDPI growth rate of 1.4 percent for FY 2012 reflects the effect of the reduction of the NIH extramural investigator salary limitation (“cap”) from $199,700 to $179,700 for that year and the continued freeze on salaries of Federal civilian employees.

However, it also goes on to note other contributions:

The BRDPI growth rate was also adjusted for the growth of stipends and related expenses on fellowships and training awards. In addition, the FY 2012 BRDPI growth is lower than the growth for FY 2011 because the rate of growth of prices for several input categories slowed down in FY 2012 compared with the growth during FY 2011. For inside NIH activities, the categories with slower growth in prices include travel, transportation, printing and reproduction, ADP and other IT services, instruments and apparatus, laboratory supplies, office supplies, utilities, repairs and alterations of facilities, compensation rates for consultants and support contracts. For extramural activities, fringe benefits, travel, equipment, supplies, patient care alterations and indirect costs each showed slower price growth during FY 2012 compared with FY 2011.

Wait. Sooooo, everything contributes a little bit? This seems out of step with Mervis' column. Wait, wait....the 2013 overview continues....

Primarily because of the freeze on Federal civilian employee salaries and the cap on compensation of extramural investigators, the rate of growth of the BRDPI during the years FY 2011 through FY 2013 has been relatively low compared with its historical relationship with general inflation as represented by the growth of the GDP Price Index.

Primarily. So that circles us right back to the reduction in the extramural cap and elimination of Federal civilian salary raises. [You might ask why the Federal civilians did not also suffer salary reductions, merely freezes, eh?] But if we take this as a valid and intended connotation then it would seem Mervis has it right. The salary issues are huge.

I wonder why they didn't just find the dollar figure. How many Investigators funded by the NIH in a given year are paying up to the cap? Multiply that by you favored reduction or increase and boom, you can translate that into new R01s.

The annual Salary Cap numbers can be found here. It seemed to steadily increase from 2005-2010, including that 2008-2009 interval that produced the most immediately prior reduction in the BRDPI. So why didn't salary cap drive the BRDPI that year?

30 responses so far


Jul 10 2015 Published by under Careerism, Fixing the NIH, Postdoctoral Training


Lastly, there IS one way to experimentally test how much a PD is worth, at least to PIs, and you are promoting that test, whether you know it or not. And that is a PD strike. I would love to see you discussing how spoiled PDs are in between gel runs because your PD is really not there. I think 50K/yr wouldn't sound that much then.

Go ahead dudes, you have nothing to lose but your chains.

22 responses so far

Will Obama's revamp of the overtime rules mean postdocs are paid more?

Jul 06 2015 Published by under Careerism, NIH Careerism, Postdoctoral Training

Justin Kiggins has launched the discussion at The Spectroscope.

Those postdocs who are salaried employees, however, are currently "exempt" from overtime pay if they make more than $23,660. The new rules mean that they will need a salary of at least $50,400. So if their institution is following the NIH standard, which sets a minimim of $42,840, it looks like they'll either need to get paid overtime for any work done over 40 hours or get a raise to meet the exemption requirements

Go on over and read, especially for the links to places you can comment on this rule prior to implementation.

184 responses so far

Grumble of the Day

Jul 06 2015 Published by under NIH, NIH Careerism, NIH funding

I still get irritated every time a PO gives me some grant advice or guidance that is discordant with my best understanding of the process. It's not so much that I take it seriously for my own strategy...I've been around this block once or twice. 

What kills me is thinking that there are poor newcomer applicants who get this advice and may think it is Gospel. This would then lead them into making suboptimal strategic or tactical decisions.

Related Reading:
POs do not understand...

28 responses so far

On recruiting peers to review NIH grants

Jun 30 2015 Published by under NIH, NIH Careerism

May 2015 Advisory Council round for the CSR of NIH.

I'll be making observations on the Luci Roberts presentation in a little while. For now, enjoy.


Okay, down to business. The part I wanted to highlight starts at 1:35 of the videocast. Dr. Nakamura introduces Luci Roberts, Director of Planning and Evaluation in the OER.

A comment from baltogirl

It is a little known fact that SROs have trouble recruiting for study section (I was told that two-thirds of people asked decline to serve). It's likely that most people are so busy writing their own grants they can't take a full month off to devote to reading the work of others.

reminded me I forgot to discuss these data.

The Division of Planning and Evaluation conducted a survey** some time ago, trying to determine the attitudes of extramural PIs that would affect their willingness to serve on study sections. I thought it applied to the above comment.
They surveyed 4,000 individuals who had submitted at least one grant as PI in the past five years and who had obtained active funding (any variety) from the NIH in the past five years. 1830 or 46% responded. Not too shabby. (They also surveyed 423 SROs of which 271/64% responded, more on that later.) Of the PIs 1,616 had served as PIs or PDs, the balance were TG directors or subproject/consortium PIs.

First up. 964 (53%) had never been asked to serve on study section in the 12 months prior to the survey. Hmm. I can almost stop right here. But no....there's much more. Still, if there is a reviewer crunch, the first order of business is to determine why over half the potential pool is not even being asked.

Ok, of the remaining 866 individuals who were asked to serve on study section, 762 agreed to do so. That's 88% saying yes. So the rumor that "two-thirds of people asked decline to serve" is falsified by this survey. Clearly, the vast majority of people who are asked step up and do their community duty.

I really, really like this. It is heartening.

The next most-interesting thing was the 10th slide which shows the ranks of PIs who are asked/not asked to review. As you would expect, the Full/Associate/Assistant Professor ranks for those asked ran 54%/32%/11% (that high for Assistants?) and 22%/23%/37% for those not asked to serve.

Again, this outcome makes it really clear what needs to be done if getting reviewers is a problem. Ask more Assistant Professors to serve***. Right? Done.

Then we get into a couple of slides on why people might say "no" when asked to review. Slide 11 present the top reasons (out of an open text box response, per Roberts' presentation) for the process being "more burdensome than it could be". The numbers are confused here because the denominator appears to be 861 when it should be 762. But in any case, 630 of the respondents who reviewed said the process was not more burdensome than necessary (huh? surprised on this one). Of the 158 who said it was too burdensome, 45% complained about the number of assigned applications. The next most common (16%) complaint was "too much time devoted to applications that will never fund". So pretty much, the most burdensome thing was review load.

This brings us to Slide 13 which pits SRO opinion versus how reviewers think. One of the biggest disconnects was in the number of in-person meetings per year that is "reasonable" since reviewers lean 1-2 and SROs were about evenly split between 1-2 and 3-4 as okay. A similar disconnect was found on application load. Half of the reviewers felt 4-6 apps was a reasonable load and only 25% felt 7-9 was reasonable. SROs leaned 7-9 (~60% of SROs) with less than 40% finding a 4-6 grant load reasonable.


I'm skipping Slide 12 on reviewer and SRO thoughts on reason to accept and decline invitations to review since 88% of those asked say yes anyway. Who cares what they say if most of them will do it just for the asking?

Okay, those were the things that jumped out at me.

*As a disclosure, Dr. Roberts was once a SRO who played a highly formative role in the early-career understanding of the NIH grant business for YHN. She was kind enough to send me the slide deck that was used in her presentation.

**I was a respondent, fwiw.

***In case of any newcomers, both YHN and CPP have advocated for this on this blog since forever.

16 responses so far

Question of the day

Jun 29 2015 Published by under Fixing the NIH, NIH, NIH Careerism

What percentage of K99 should fail to transition to the R00 phase in a healthy system?
What percentage of those that go to R00 should fail to ever gain major independent funding as a PI? 

38 responses so far

Older posts »