Archive for the 'NIH funding' category

Your Grant in Review: Investigator Independence

May 27 2016 Published by under NIH, NIH Careerism, NIH funding

superkash started it:


and amplified it:

there was a bit of chatter and then eventually AListScientist asserted:

First, I addressed Independence of a NIH Grant PI to some extent way back in 2007, reposted in 2009.

I, as well as several other colleagues who review grants, have noticed a seemingly sharp uptick in the number of applications coming in from PIs who are more "transitioning" than "transitioned". PIs whose job titles might be something other than "Assistant Professor" and ones who are still in or around the same laboratory or research group in which they have done a big chunk of postdoctoral work. In extreme cases the PI might still be titled "Postdoc" or have trained in the same place essentially since graduate school!

Readers of this blog might conclude that this trend, which I've been noticing for at least the past 3-4 rounds, delights me. And to the extent that it represents a recognition of the problems with junior scientist's making the career transition to independence this does appear a positive step. To the extent that it opens up artificial barriers blocking the next generation of scientists- great.

The slightly more cynical view expressed by colleagues and, admittedly, myself is that this trend has been motivated by IC Program behavior both in capping per-PI award levels and in promoting grant success for New Investigators. In other words that the well-established PIs with very large research groups are thinking that grants for which they would otherwise be the PI will now be more successful with some convenient patsy long-term postdoc at the helm. The science, however, is just the same old stuff of the established group and PI.

I surmise that the tweeting of @superkash was related to this conundrum. I would suggest to newcomers to the NIH system that these issues are still alive and well and contribute in various ways to grant review outcome. We see very clearly in various grant/career related discussion on twitter, this blog and commentary to various NIH outlets that peer scientists have strong ideas on categories of PI that deserve or don't deserve funding. For example in the recent version on CSR's Peer Review website, comments suggest we should keel the yuge labs, keel the ESIs, keel the riffraff noobs and save the politically disconnected. The point being that peer reviewers come with biases for and against the PI(s) (and to lesser extent the other investigators).

The fact that the Investigator criterion is one of the five biggies (and there is no official suggestion that it is of any lesser importance than Approach, Significance or Innovation) permits (and one might say requires) the reviwers to exercise these biases. It also shows that AListScientist's apparent belief that Investigators are not to be evaluated because the applicant University has certified them is incorrect.

The official CSR Guidance on review of the Investigator criterion is posed as a series of questions:

Are the PD/PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?

"well suited"
"appropriate experience"
Right there you can see where the independence of the PI might be of interest to the reviewer.

"have they demonstrated an ongoing record of accomplishments"
We what to know what they personally have accomplished. Or caused to have accomplished if you want to natter about PIs not really doing hands on science. The point is, can this PI make the proposed studies happen? Is there evidence that she has done so before? Or is there merely evidence that he has existed as glorified hands in the PIs lab up to this point in time?

"are their leadership approach, governance and organizational structure appropriate for the project?"

Can they lead? Can they boot all the tails hard enough to get this project accomplished? I say that this is an entirely appropriate consideration.

I hope you do as well and I would be interested to hear a counter argument.

I suspect that most of the pushback on this comes from the position of thinking about the Research Assistant Professor who IS good enough. Who HAS operated more or less independently and led projects in the SuperLab.

The question for grant review is, how are we to know? From the record and application in front of us.

__
I am unable to leave this part off: If you are a RAP or heading to be one as a mid to late stage postdoc, the exhortation to you is to lay down evidence of your independence as best you are able. Ask Associate Professor peers that you know about what possible steps you can take to enhance the optics of you-as-PI on this.

68 responses so far

The proper number of Specific Aims for your NIH Grant

May 10 2016 Published by under NIH funding

The Aims shall be Three, and Three shall be the number of Aims.

Four shalt there not be, nor Two except as they precede the Third Aim.

Five is right out.

20 responses so far

Reviewing SABV grant applications for the NIH

May 09 2016 Published by under NIH, NIH funding, Sex Differences

The first study section rounds that are obliged to grapple with the new SABV policy are upon us.
SROs are instructing panels and issuing grant assignments to reviewers.

If you are reviewing, what are your thoughts?

Me, I see more of the entirely predictable ahead- people ignoring it (or accepting thin excuses for not studying both sexes, in reality) or brandishing it as a cudgel in highly variable fashion. I'm cynical, perhaps unduly so, eh?

The opportunity to beat a panel into better agreement will come far too late for most applications. There is no way that guilt over consistency will drag the triaged apps up for discussion. 

I still seek consistency. In my own reviewing and in any panels I serve. I think this a virtue to strive for.

And I think that consideration and discussion of the approach to tricky review issues is the way to advance toward that goal. 

I also think that when you accept a reviewer position, you are agreeing to give the NIH what it is requesting, to the best of your ability. If you fight against the SABV push, you are doing it very wrong, IMO.

So....what do you think? How are you approaching the SABV mandate? Now that you have a few examples of how applicants have dealt with it, have you learned anything useful for us to consider?

--
Open Mike blog on SABV mandate

11 responses so far

NIH policy on SABV and realistic review

May 06 2016 Published by under NIH, NIH funding

My prediction is that the grants that will do best in the next few rounds are those that successfully excuse themselves from including both male and female subjects. 

The grants that try to respond to the spirit of the new NIH SABV initiative will get comparatively hammered in review. 

28 responses so far

Entitled to a Grant: What is fair?

May 02 2016 Published by under Fixing the NIH, NIH, NIH Careerism, NIH funding

I am genuinely curious as to how you people see this. Is there any particular difference between people arguing that that acquisition of the first major grant award should be protected versus multiple award and the people arguing that acquisition of the first and third concurrent awards should be on an equal footing?

If we agree that NIH (or NSF or CIHR or whatever) grants are competitively awarded, it follows that nobody is actually entitled to a grant. And as far as I am aware, all major funding agencies operate in a way that states and demonstrates the truth of this statement.

Specifically in the NIH system, it is possible for the NIH officials to choose not to fund a grant proposal that gets the best possible score and glowing reviews during peer review. Heck, this could happen repeatedly for approximately the same project and the NIH could still choose not to fund it.

Nobody is entitled to a grant from the NIH. Nobody.

It is also the case that the NIH works very hard to ensure a certain amount of equal representation in their awarded grants. By geography (State and Congressional district), by PI characteristics of sex and prior NIH PIness, by topic domain (see the 28 ICs) or subdomain (see Division, Branches of the ICs. also RFAs), etc.

Does a lean to prioritize the award of a grant to those with no other major NIH support (and we're not just talking the newcomers- plenty of well-experienced folks are getting special treatment because they have run out of other NIH grant support) have a justification?

Does the following graph, posted by Sally Rockey, the previous head of Extramural Research at the NIH make a difference?

This shows the percentage of all PIs in the NIH system for Fiscal Years 1986, 1998, 2004 (end of doubling) and 2009 who serve as PI on 1-8 Research Project Grants. In the latest data, 72.3% had only one R01 and 93% had 1 or 2 concurrent RPGs. There were 5.4% of the PIs that held 3 grants and 1.2% that held 4 grants. I just don't see where shifting the 7% of 3+ concurrent awards into the 1-2 grant population is going to budge the needle on the perceived grant chances of those without any major NIH award. Yes, obviously there will be some folks funded who would otherwise not have been. Obviously. But if this is put through in a systematic way*, the first thing the current 3+ grant holders are going to do is stop putting in modular grants and max out their allowable 2 at $499,999 direct costs. Maybe some will even get Program permission to breach the $500,000 DC / y threshold. So there won't be a direct shift of 7% of grants back into the 1-2 grant PI population.

There has been a small trend for PIs holding more grants concurrently from 1986 to the late naughties but this is undoubtedly down to the decreasing purchasing power of the modular-budget grant.

BRDPI.
I"ve taken their table of yearly adjustments and used those to calculate the increase necessary to keep pace with inflation (black bars) and the decrement in purchasing power (red bars). The starting point was the 2001 fiscal year (and the BRDPI spreadsheet is older so the 2011 BRDPI adjustment is predicted, rather than actual). As you can see, a full modular $250,000 year in 2011 has 69% of the purchasing power of that same award in 2001.

Without that factor, I'd say the relative proportions of PIs holding 1, 2, 3 etc grants would be even more similar across time than it already is.

So I come back to my original question. What is fair? What policies should the NIH or any broad governmental funding body adopt when it comes to distributing the grant wealth across laboratories? On what basis should they do this?

Fairness? Diversity of grant effort? PR/optics?

__
*and let us face it, it is hugely unlikely that the entire NIH will put through a 2-grant cap without any exceptions. Even with considerable force and authority behind it, any such initiative is likely to be only partially successful in preventing 3+ grant PIs.

DISCLAIMER: As always, I am an interested party in these discussions. My lab's grant fortunes are affected by broad sweeping policies that the NIH might choose to adopt or fail to adopt. You should always read my comments about the NIH grant game with this in mind.

106 responses so far

NIA paylines and anti-ESI bias of review

Apr 20 2016 Published by under NIH, NIH funding

MillerLab noted on the twitters that the NIA has released it's new paylines for FY2016. If your grant proposal scores within the 9%ile zone, congrats! Unless you happen to be an Early Stage Investigator in which case you only have to score within the top 19% of applications, woot!

I was just discussing the continuing nature of the ESI bias in a comment exchange with Ferric Fang on another thread. He thinks

The problem that new investigators have in obtaining funding is not necessarily a result of bias but rather that it is more challenging for new investigators to write applications that are competitive with those of established investigators because as newcomers, they have less data and fewer accomplishments to cite.

and I disagree, viewing this as assuredly a bias in review. The push to equalize success rates of ESI applicants with those of established investigators (generational screw-job that it is) started back in 2007 with prior NIH Director Elias Zerhouni. The mechanism to accomplish this goal was, and continues to be, naked quota based affirmative action. NIH will fund ESI applications out of the order of review until they reach approximately the same success percentages as is enjoyed by the established investigator applications. Some ICs are able to game this out predictively by using different paylines- the percentile ranks within which almost all grants will be funded.

NIA-fundingpolicyAs mentioned, NIA has to use a 19%ile cutoff for ESI applications to equal a 9%ile cutoff for established investigator applications. This got me thinking about the origin of the ESI policies in 2007 and the ensuing trends. Luckily, the NIA publishes its funding policy on the website here. The formal ESI policy at NIA apparently didn't kick in until 2009, from what I can tell. What I am graphing here are the paylines used by NIA by fiscal year to select Exp(erienced), ESI and New Investigator (NI) applications for funding.

It's pretty obvious that the review bias against ESI applications continues essentially unabated*. All the talk about "eating our seed corn", the hand wringing about a lost generation, the clear signal that NIH wanted to fund the noobs at equivalent rates as the older folks....all fell on deaf ears as far as the reviewers are concerned. The quotas for the ESI affirmative action are still needed to accomplish the goal of equalizing success rates.

I find this interesting.

__
*Zerhouni noted right away [PDF] that study sections were fighting back against the affirmative action policy for ESI applications.

Told about the quotas, study sections began “punishing the young investigators with bad scores,” says Zerhouni.

Note: It is probably only a coincidence that CSR reduced the number of first time reviewers in FY2014, FY2015 relative to the three prior FYs.

16 responses so far

On removing grant deadlines

Apr 20 2016 Published by under NIH, NIH Careerism, NIH funding

Eric Hand reported in Science that one NSF pilot program found that allowing for any-time submission reduced applications numbers.

Assistant Director for Geosciences Roger Wakimoto revealed the preliminary results from a pilot program that got rid of grant proposal deadlines in favor of an anytime submission. The numbers were staggering. Across four grant programs, proposals dropped by 59% after deadlines were eliminated.

I have been bombarded with links to this article/finding and queries as to what I think.

Pretty much nothing.

I do know that NIH has been increasingly liberal with allowing past-deadline submissions from PIs who have served on study section. So there is probably a data source to draw upon inside CSR if they care to examine it.

I do not know if this would do anything similar if applied to the NIH.

The NSF pilot was for

geobiology and low-temperature geochemistry, geomorphology and land-use dynamics, hydrological sciences, and sedimentary geology and paleobiology.

According to the article these are fields in which

"many scientists do field work, having no deadline makes it easier for collaborators to schedule time when they can work on a proposal".

This field work bit is not generally true of the NIH extramural community. I think it obvious that continual-submission helps to schedule time but I would note that it also eliminates a stick for the more proactive members of a collaboration to beat the slaggards into line. As a guy who hits his deadlines for grant submission, it's probably in my interest to further lower the encouragements the lower-energy folks require.

According to a geologist familiar with reviewing these grants

The switch is “going to filter for the most highly motivated people, and the ideas for which you feel the most passion,” he predicts. When he sits on merit review panels, he finds that he can usually reject half of the proposals right away as being hasty or ill-considered. “My hope is that this has taken off the bottom 50%,” he says. “Those are the ones you read and say, ‘Did they have their heart in this?’”

Personally I see very few NIH grant proposals that appear to me to be "hasty or ill-considered" or cause me to doubt the PI has her heart in it. And you know how I feel about the proposition that the RealProblem with NIH grant success hinges on whether or not PIs refine and hone and polish their applications into some shining gem of a document. Applications are down so therefore success rates go up is the only thing we need to take away from this pilot, if you ask me. Any method by which you could decrease NIH applications would likewise seem to improve success rates.

Would it work for NIH types? I tend to doubt it. That program at NSF started with only two submission rounds per year. NIH has three rounds for funding per year, but this results from a multitude of deadlines including new R01, new R21/R03, two more for the revised apps, special ones for AIDS-related, RFAs and assorted other mechanisms. As I mentioned above, if you review for the NIH (including Advisory Council service) you get an extra extension to submit for a given decision round.

The pressure for most of us to hit any specific NIH deadline during the year is, I would argue, much lower at baseline. So if the theory is that NSF types were pressured to submit junky applications because their next opportunity was so far away....this doesn't apply to NIH folks.

6 responses so far

Bias at work

A piece in Vox summarizes a study from Nextions showing that lawyers are more critical of a brief written by an African-American. 

I immediately thought of scientific manuscript review and the not-unusual request to have a revision "thoroughly edited by a native English speaker". My confirmation bias suggests that this is way more common when the first author has an apparently Asian surname.

It would be interesting to see a similar balanced test for scientific writing and review, wouldn't it?

My second thought was.... Ginther. Is this not another one of the thousand cuts contributing to African-American PIs' lower success rates and need to revise the proposal extra times? Seems as though it might be. 

22 responses so far

On sending trainees to conferences that lack gender balance

Neuroscientist Bita Moghaddam asked a very interesting question on Twitter but it didn't get much discussion yet. I thought I'd raise it up for the blog audience.

My immediate thought was that we should first talk about the R13 Support for Scientific Conferences mechanism. These are often used to provide some funding for Gordon Research Conference meetings, for the smaller society meetings and even some very small local(ish) conferences. Examples from NIDA, NIMH, NIGMS. I say first because this would seem to be the very easy case.

NIH should absolutely keep a tight eye on gender distribution of the meetings supported by such grant awards.The FOA reads, in part:

Additionally, the Conference Plan should describe strategies for:

Involving the appropriate representation of women, minorities, and persons with disabilities in the planning and implementation of, and participation in, the proposed conference.
Identifying and publicizing resources for child care and other types of family care at the conference site to allow individuals with family care responsibilities to attend.

so it is a no-brainer there, although as we know from other aspects of NIH the actual review can depart from the FOA. I don't have any experience with these mechanisms personally so I can't say how well this particular aspect is respected when it comes to awarding good (fundable) scores.

Obviously, I think any failure to address representation should be a huge demerit. Any failure to achieve representation at the same, or similar meeting ("The application should identify related conferences held on the subject during the past 3 years and describe how the proposed conference is similar to, and/or different from these."), should also be a huge demerit.

At least as far as this FOA for this scientific conference support mechanism goes, the NIH would appear to be firmly behind the idea that scientific meetings should be diverse.

By extension, we can move on to the actual question from Professor Moghaddam. Should we use the additional power of travel funds to address diversity?

Of course, right off, I think of the ACNP annual meeting because it is hands down the least diverse meeting I have ever attended. By some significant margin. Perhaps not in gender representation but hey, let us not stand only on our pet issue of representation, eh?

As far as trainees go, I think heck no. If my trainee wants to go to any particular meeting because it will help her or him in their careers, I can't say no just to advance my own agenda with respect to diversity. Like it or not, I can't expect any of them to pay any sort of price for my tender sensibilities.

Myself? Maybe. But probably not. See the aforementioned ACNP. When I attend that meeting it is because I think it will be advantageous for me, my lab or my understanding of science. I may carp and complain to certain ears that may matter about representation at the ACNP, but I'm not going on strike about it.

Other, smaller meetings? Like a GRC? I don't know. I really don't.

I thank Professor Moghaddam for making me think about it though. This is the start of a ponder for me and I hope it is for you as well.

16 responses so far

Your Grant in Review: Power analysis and the Vertebrate Animals Section

Feb 11 2016 Published by under Grant Review, Grantsmanship, NIH funding

As a reminder, the NIH issued warning on upcoming Simplification of the Vertebrate Animals Section of NIH Grant Applications and Contract Proposals.

Simplification! Cool, right?

There's a landmine here.

For years the statistical power analysis was something that I included in the General Methods at the end of my Research Strategy section. In more recent times, a growing insistence on the part of the OLAW that a proper Vertebrate Animals Section include the power analysis has influenced me to drop the power analysis from the Research Strategy. It became a word for word duplication so it seemed worth the risk to regain the page space.

The notice says:

Summary of Changes
The VAS criteria are simplified by the following changes:

  • A description of veterinary care is no longer required.

  • Justification for the number of animals has been eliminated.

  • A description of the method of euthanasia is required only if the method is not consistent with AVMA guidelines.

 

This means that if I continue with my current strategy, I'm going to start seeing complaints about "where is the power analysis" and "hey buddy, stop trying to evade page limits by putting it in the VAS".

So back to the old way we must go. Leave space for your power analysis, folks.
__
If you don't know much about doing a power analysis, this website is helpful: http://homepage.stat.uiowa.edu/~rlenth/Power/

17 responses so far

Older posts »