Archive for the 'NIH' category

Repost: More data on historical success rates for NIH grants

Feb 17 2015 Published by under Fixing the NIH, NIH, NIH Careerism, NIH funding

Our recent discussion of topics related to the Emeritus Award being considered by the NIH powers that be has been robust. I, of course, have been reminding one of the target demographic scientists that she and her generation have had a pretty good run under the NIH system. It seemed like a good moment to remind everyone that there are data upon which to base our understanding of how difficult it has and has not been for various scientific generations. Time to repost an older blog entry.

This was first posted 11 July 2012.


Thanks to a query from a reader off the blog and a resulting request from me, our blog-friend microfool pointed us to some data. Since I don't like Tables, and the figure on the excel file stinks, here is a different graphical depiction:

The red trace depicts success rates from 1962 to 2008 for R01 equivalents (R01, R23, R29, R37). Note that they are not broken down by experienced/new investigators status, nor are new applications distinguished from competing continuation applications. The blue line shows total number of applications reviewed...which may or may not be of interest to you. [update 7/12/12: I forgot to mention that the data in the 60s are listed as "estimated" success rates.]

The bottom line here is that looking at the actual numbers can be handy when playing the latest round of "We had it tougher than you did" at the w(h)ine and cheese hour after departmental seminar. Success rates end at an unusually low point...and these numbers stop in 2008. We're seeing 15% for R01s (only) in FY2011.

Things are worse than they've ever been and these dismal patterns have bee sustained for much longer. If we look at the ~30% success rates that ruled the day from 1980-2003, the divergence from the trend from about 1989 to 1996 was interrupted in the middle and, of course, saw steady improvement in the latter half. The badness that started in FY2004 has been 8 unrelieved Fiscal Years and shows no sign of abatement. Plus, the nadir (to date) is much lower.

Anyone who tries to tell you they had it as hard or harder at any time in the past versus now is high as a kite. Period.

Now, of course, it IS true that someone may have had it more difficult in the past than they do now, simply because it has always been harder for the inexperienced PIs to win their funding.

RPGsuccessbyYear.png
source
As we know from prior posts, career-stage differences matter a LOT. In the 80s when the overall success rate was 30%, you can see that newcomers were at about 20% and established investigators were enjoying at least a 17%age point advantage (I think these data also conflate competing continuation with new applications so there's another important factor buried in the "Experienced" trace.) Nevertheless, since the Experienced/New gap was similar from 1980 to 2006, we can probably assume it held true prior to that interval as well.

3 responses so far

A tiny bias goes a long way when it comes to grant review

Feb 11 2015 Published by under Fixing the NIH, NIH

From ScienceInsider:

Now, a new computer simulation explores just how sensitive the process might be to bias and randomness. Its answer: very. Small biases can have big consequences, concludes Eugene Day, a health care systems engineer at the Children's Hospital of Philadelphia, in Research Policy. He found that bias that skews scores by just 3% can result in noticeable disparities in funding rates.

T. E. Day, The big consequences of small biases: A simulation of peer review, 2015, Research Policy [epub ahead of print 28 Jan] [Publisher Site]

from the paper Abstract:

When total review bias exceeds 1.9% of grant score, statistically significant variation in scores between PC and NPC investigators is discernable in a pool of 2000 grant applications. When total review bias exceeds 2.8% of total grant score, statistically significant discrepancies in funding rates between PC and NPC investigators are detectable in a simulation of grant review.

Day generated a Preferred Class of applications and a NonPreferred Class of applications and ran a bunch of 3-reviewer scenarios with and without reviewer bias against the NPC applications. As far as I can tell the takeaway conclusion about funding here refers to a situation in which the effective payline is 10%. You will immediately grasp that NIH grant review was a strong contributor to the model parameters.

I will admit I am only able to grasp the main points here and I am in no way able to evaluate the nitty gritty.

But it appears to have a very strong message. Namely, that our introspections that "well, if there is bias it is very tiny so we don't have to be worried about it" needs to change.

There is something even scarier in this paper. From the Discussion:

The threshold level of bias in this environment seems to be 2.8% of the total possible score of the grant; this is the level at which the 95% CI of the odds ratio “kisses” 1.00. This represents a single reviewer with a bias of 0.75 points (or three reviewers each with biases of 0.25 points), which is less than half (44.4%) of the standard deviation in a single reviewer’s score. What this suggests is that levels of bias which are sub-noise – that is, that are dramatically less detectable than normal variation in reviewer scores – are sufficient to substantially bias the number of funded applications in favor of preferred investigators.

RIGHT???? The bias can be of smaller effect size than many "normal" sources of variability in scoring that we accept as the resolution of the system. And it still leads to a statistically significant bias in funding outcome.

We are talking in recent days about bias in favor of highly established, older scientists. It has been longer but the Ginther report indicating disparity of grant review outcome for African-American PIs is clearly relevant here.

What this simulation cannot do, of course, is to model the cumulative, iterative effects of review bias. Namely, the way that selection of PC applications for funding has a tendency to increase the bias in the reviewer pool, since those beneficiaries become the next reviewers. Also, the way that over the long haul, disparity of the first award can lead to actual quality differences in the subsequent applications because PI #1 had the money to pursue her science and PI #2 did not have as easy of a time generating data, publishing papers and recruiting postdoc talent.

5 responses so far

Thought of the Day

Feb 11 2015 Published by under Grant Review, Grantsmanship, NIH Careerism

I hate when I review grant proposals that are good, but clearly have been made pedestrian and conservative through the school of hard knocks. There is so much awesome that could be done by these people. It is so clear to me what the really high impact version of this grant should look like. (Not having any illusions about my own unique brilliance, I assume they could see it too. )

But the review realities batter PIs down into a defensive crouch, worried that if they step too far past their Preliminary Data or established expertise they will get crushed.

Because, of course, they would get crushed.

Sometimes I wish I were the Boss of Science more than other times.

28 responses so far

Perspectives from senior scientists on the Emeritus award discussion

Feb 09 2015 Published by under #FWDAOTI, NIH, NIH Careerism

Tthe comments just keep coming over at RockTalking.

8581+ year old guy:

In 2012-13 my NIH renewal proposal with 4 specific aims was turned down 2X by the GM, NCSD Panel, with 35%+ priority scores. ...I appealed the grant reviews to the GM Council and they awarded the grant to me for 3 years at somewhat reduced funding. This funding will finance my lab until Sept 2016, after which I will close down. I am NOT closing down because I have lost my energy for, or interest in, research: [blah, blah we published and showed them!] I am closing down my lab because I can no longer put up with the aggravation of having my grants turned down

Cry me a river.

another senior investigator is on fire:

So far, all the comments fully support age discrimination. How sad! -

Age limits are silly and discriminatory. Merit worked well UNTIL there was no more money in the NIH bank.


There should be COMPETITIVE opportunities for scientists at all stages of their careers (notice I said competitive) . So it’s not a handout at all and it would be merit-based….just like those for newly trained scientists.

There are wonderful examples of some amazing senior scientists. ONE special initiative is not unreasonable for this group!

Finally, I don’t believe I said that I “deserve” anything… except not to be discriminated against for age, gender or whatever characteristic you wish to select.

emphasis added.

updated: omg, the old guy again!

At the last American Society for Cell Biology meeting in Philadelphia (December, 2014) I stood up at the membership meeting ( only ca. 30 people) attended by some of the senior wheels in cell biology. I asked that the society appoint a new standing committee that did
nothing but try to re-design the NIH extramural grant system in its entirety, eg. the grant applications, who can apply, how review panels are chosen, requirements for NIH supported investigators to serve on panels, and how key personnel of the NIH bureaucracy are chosen, and what type of grants should be awarded. My suggestion was not greeted with great enthusiasm and, in fact, elicited some negative comments.

Bashir points out something that is also important about this proposal.

After all that determination to do something after Ginther Report, with mentoring groups and other round-a-bout approaches aimed at eventually addressing racial disparities in grant awards, suddenly we may have a very direct new mechanism that, regardless of the underlying logic, is essentially un-diversity.

56 responses so far

Closeout funding

Feb 05 2015 Published by under Fixing the NIH, NIH, NIH Careerism, NIH funding

Within the past thre years or so I had a Program Officer mention the idea of "closeout funding" to me.

One of my top few flabbergasting moments as an extramural scientist.

It referred, of course, to them using program discretion to give a softer landing to one of their favored long-time PIs who had failed to get a fundable score on a competing renewal. It was said in a context that made it clear it was a regular thing in their decision space.

This explains an awful lot of strange R56 BRIDGE (to nowhere) awards, I thought to myself.

I bring this up because I think it relates to this week's discussion of the proposed "emeritus award" concept.

13 responses so far

Wow, the NIH seeks input on an amazing new initiative!

From NOT-OD-15-064 we learn:

The strategies for sustaining a vibrant biomedical research enterprise are complex and multi-faceted.  NIH strives to support not only the biomedical research workforce, but to support the foundation of research programs that our workforce has created. 

An important issue for NIH is the long term success  for the research we support.  Over the years, NIH has been persistent and creative in efforts to support early career investigators through policy changes and new programs.  But we must also consider the needs of our mid-career investigators and how NIH can assist with the continuation of their well-established research programs, since evidence has shown that the most innovative and productive years of work come from PIs in the 40-59 year old age range.  While these highly productive investigators are happy pursuing their research questions in the laboratory, current funding stresses have hindered the current generation relative to past generations of mid-career scientist.  Our most vibrant investigators have invested their careers to establish the intellectual and technical infrastructure needed to pursue their research questions, and it is in our interest to facilitate progress in their established programs.

Yes, I agree! Totally true.

Therefore, NIH would like to explore potential mechanisms to facilitate the needs of the most productive members of our biomedical workforce. We would like to gauge community interest in a new type of award that could allow established investigators to maximize their output under funding from NIH research grants, while greatly advancing our scientific knowledge and resources. Such an award could permit an established investigator to form partnerships with other faculty members in order to facilitate research inquiry in an efficient and cost-effective way under P-mechanisms as with prior generations. The established investigator would, of course, be expected to train and equip junior colleagues to contribute to mutual interests and research projects while working with them in a mentoring role. If such a collaboration is not feasible, a mid-career award might allow some established investigators to complete expansive projects within their own laboratories.

Wow. Really good stuff here NIH. Glad to see you finally recognizing what brung ya and what you need to bring to the race to keep on winning.

Request for Information This Request for Information (RFI) seeks input from the research community, including scientists from all career stages; research administration professionals; departmental chairs; deans; professional societies; and other interested stakeholders. Public comment is sought for the following:

  • Community interest in an award that allows a mid-career investigator to flourish without being dependent on submitting so many NIH research grant applications
  • Ideas for how one would utilize a mid-career award (e.g., to facilitate laboratory sustainability; to promote novel research inquiry; to provide opportunities for expansion of larger collaborative research projects)
  • Suggestions for the specific characteristics for a mid-career award (e.g., number of years of support; amount of support; mechanisms of evaluation)
  • Ways in which NIH could incentivize the use of a mid-career award, from the perspectives of both mid-career investigators and institutions
  • Impediments to the participation in such an award program, from the perspectives of both mid career investigators and institutions
  • Any additional comments you would like to offer to NIH on this topic

Oh, for sure. I'm going to run, not walk, right on over to the form to submit my approval.

 

Oooooooohhhhh.

 

Wait.

 

This is for EMERITUS faculty? Such as those past the age of 65 who keep on submitting copious numbers of research grants? And the NIH wants to somehow use this to persuade the unwilling* to wind down their lab in good order?

 

What a disaster.

 

Additional Reading:

DataHound

RockTalking

 

__

*This mechanism for winding down a PI's career while sustaining his** "legacy" laboratory and program already exists and is in current practice. A senior PI simply steps down from the PI position and the University nominates a junior person to take over. Maybe with continued Co-investigator status for the Emeritizing person. It works to serve this goal. It is proven.

 

**yeah, "his". that's who these people are. For the most part.

73 responses so far

Your Grant in Review: Credible

Jan 30 2015 Published by under NIH, NIH Careerism, NIH funding, Peer Review

I am motivated to once again point something out.

In ALL of my advice to submit grant applications to the NIH frequently and on a diversity of topic angles, there is one fundamental assumption.

That you always, always, always send in a credible application.

That is all.

17 responses so far

Data faker happily employed by the US Patent Office

via retraction watch we learn:

A Bijan Ahvazi has been working at the USPTO since at least 2008, and today a source confirmed that it was the same person who was the subject of last October’s ORI report. Ahvazi was found to have faked five different images in three different papers, two of which have been retracted.

The Notice of ORI finding appeared in October of 2014.

Based on the report of an investigation conducted by the National Institutes of Health (NIH) and additional analysis by ORI in its oversight review, ORI found that Dr. Bijan Ahvazi, former Director of the Laboratory of X-ray Crystallography, National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS), NIH, engaged in research misconduct in research supported by the Intramural Program at NIAMS, NIH.

The Notice shows that the offenses for which Ahvazi was convicted date to 2004 and 2006. One doesn't have to assume that much to figure out that he was busted and then had to look for a new job somewhere between 2006 and 2008. It took until 2014 for his fraud to come to light via the official ORI mechanisms. Presumably, although we don't know for sure, the investigation was confidential up until it reached its formal conclusions which may have permitted him to avoid telling the US Patent and Trade Office about his little whoopsie? I dunno, do you think the USPTO would hire a data fraud as a patent examiner if they knew about it? One thinks not.

__
p.s. apparently a co-author of this data faker died under bizarre circumstances in 2003.

11 responses so far

These ILAF types just can't help sounding selfishly elitist, can they?

Good Gravy.

One David Korn of the Massachusetts General Hospital and Harvard Medical School has written a letter to Nature defending the indirect cost (IDC; "overhead") rates associated with NIH grants. It was submitted in response to a prior piece in Nature on IDC which was, to my eye, actually fairly good and tended to support the notion that IDC rates are not exorbitant.

But overall, the data support administrators’ assertions that their actual recovery of indirect costs often falls well below their negotiated rates. Overall, the average negotiated rate is 53%, and the average reimbursed rate is 34%.

The original article also pointed out why the larger private Universities have been heard from loudly, while the frequent punching-bag smaller research institutes with larger IDC rates are silent.

Although non-profit institutes command high rates, together they got just $611 million of the NIH’s money for indirect costs. The higher-learning institutes for which Nature obtained data received $3.9 billion, with more than $1 billion of that going to just nine institutions, including Johns Hopkins University in Baltimore, Maryland, and Stanford (see ‘Top 10 earners’).

Clearly Dr. Korn felt that this piece needed correction:

Aspects of your report on US federal funding of direct research costs and the indirect costs of facilities and administration are misleading (Nature 515, 326–329; 2014).

Contrary to your claim, no one is benefiting from federal largesse. Rather, the US government is partially reimbursing research universities for audit-verified indirect costs that they have already incurred.

Ok, ok. Fair enough. At the very least it is fine to underline this point if it doesn't come across in the original Nature article to every reader.

The biomedical sciences depend on powerful technologies that require special housing, considerable energy consumption, and maintenance. Administration is being bloated by federal regulations, many of which dictate how scientists conduct and disseminate their research. It is therefore all the more remarkable that the share of extramural research spending on indirect costs by the US National Institutes of Health (NIH) has been stable at around 30% for several decades.

Pretty good point.

But then Korn goes on to step right in a pile.

Negotiated and actual recovery rates for indirect costs vary across the academic community because federal research funding is merit-based, not a welfare programme.

You will recognize this theme from a prior complaint from Boston-area institutions.

“There’s a battle between merit and egalitarianism,” said Dr. David Page, director of the Whitehead Institute, a prestigious research institution in Cambridge affiliated with MIT.

Tone deaf, guys. Totally tone deaf. Absolutely counter-productive to the effort to get a majority of Congress Critters on board with support for the NIH mission. Hint: Your various Massachusetts Critters get to vote once, just like the Critters from North and South Dakota, Alabama and everywhere else that doesn't have a huge NIH-funded research enterprise.

And why Korn chooses to use a comment about IDC rates to advance this agenda is baffling. The takeaway message is that he thinks that higher IDC rates are awarded because His Awesome University deserves it due to the merit of their research. This totally undercuts the point he is trying to make, which is presumably "institutions may be private or public, urban or rural, with different structures, sizes, missions and financial anatomies.".

I just don't understand people who are this clueless and selfish when it comes to basic politics.

23 responses so far

Excellent observation on only funding the absolutely most amazing science

Jan 21 2015 Published by under Fixing the NIH, NIH

Over at Rock Talk, by a Joel MacAuslan:

It isn’t about whether to fund only the “best” science: I really DON’T want only Isaac Newtons and Louis Pasteurs to be competitive, and to be able to spend their careers on this research. That’s because I don’t want to wait 200 years for all that “great” science to trickle through society. Fund lots and lots of very good science, and cure heart disease in 40 years, instead!

5 responses so far

« Newer posts Older posts »