Archive for the 'NIH' category

NIH clumsily tries to .. [something] ... for grant reviewers

Feb 20 2015 Published by under Careerism, NIH, NIH Careerism

I noticed a funny one in the NIH Guide notices today.

NOT-OD-15-035 Reinforcing Service to the Biomedical Research Community

Yes, yes. I see. "Reinforcement" of a behavior like "Service to the Biomedical Research Community" means increasing the strength or probability of the behavior. So yes, that's good. What are they trying to do here?


Purpose
This Notice gratefully acknowledges, and seeks to reinforce, service to the biomedical research community by recipients of National Institutes of Health (NIH) research funding (see NOT-OD-10-089). Obtaining input from qualified experts across the entire spectrum of the extramural research enterprise furthers diversity of scientific thought, inclusiveness, and breadth of perspectives necessary to evaluate applications in a review process that strives for integrity and fairness. The interdisciplinary, collaborative, and global nature of biomedical research today requires increasingly complex review panels that need both broad and specific expertise in countless topic areas. Thus, the NIH, the biomedical research community, and the general public benefit from the service of NIH-funded investigators and maximize the Nation's investment in biomedical research.

Yes, yes. Very nice. but what are they actually doing to reinforce the behavior?


Policy
The NIH expects principal investigators of NIH supported grants and contracts to serve on NIH peer review groups, when asked. Therefore, the NIH expects grantee institutions and R&D contract recipients to encourage their NIH-funded investigators to serve on NIH peer review and advisory groups. These groups include Scientific Review Groups (or “study sections”) in the initial peer review of grant applications and technical evaluation of R&D contract proposals, National Advisory Boards or Councils (NACs) for second-level peer review, NIH Boards of Scientific Counselors (BSCs) for intramural programs, and Program Advisory Committees (PACs) for initiative development and concept review.

emphasis added.

Okay, so any University with a pulse is already encouraging their PIs to serve on study section. Right? They know about how this will help their bottom IDC line, yes? And if they are discouraging any subset of investigators from serving I imagine it is the Assistant Professors...who the NIH / CSR isn't looking to recruit anyway.

Hmm.

I have a suggestion. Two actually. The first one is hey, if you want to reinforce a behavior, why don't you use the delivery of a rewarding stimulus? I mean sure, you give us reviewers a delay in the submission deadlines, that's cool and all. But obviously the NIH thinks they need something more. How about protection from budget reductions? A couple of extra percentile points on newly competing awards?

No?

Okay, that costs you money, I realize. How about something very cheap with some motivational value? Journals often publish a list of their reviewers at the end of the calendar year and thank them for their service. It's nice. But the NIH can do this one better. Set up a website with a list of reviewers and the number of grants they've been assigned to review. Maybe do it by year too and provide permalinks.

Trust me, academics will eat this up. They will check out how many reviews their buddies are/are not doing and give them a little hell for not matching up around the conference coffee table. They will start linking to their entry from their websites and bragging about it in their P&T documentation.

I wonder. Really, NIH. Do you have anyone making policy that understands people even the tiniest little bit? I am about the opposite of a people person and it took me like two tweets to think of this.

31 responses so far

If you care about the Emeritus NIH Grant...

Feb 20 2015 Published by under Fixing the NIH, NIH Careerism

...you need to take action on the official form.

Click through and get that done.

3 responses so far

"We've seen it all before, this is just a cyclical downturn for the NIH funding"

Popular thought. But it is nonsense.

A close collaborator was recently experiencing this common denial trope from one of the more established type of scientists. The thinking is that

"...sure, things are tough for younger scientists right now but hey, things have been tough before. It's all just a cycle and oh, stop complaining kiddos. We had it hard too."

Here is why it is in error to argue this- the magnitude of the downturn was lesser and it lasted for a shorter duration in those prior "cycles". Let us refer to the infamous Undoubling graph.

Heinig07-NIHbudget-trend.jpeg.jpg

Figure 1. NIH Appropriations (Adjusted for Inflation in Biomedical Research) from 1965 through 2007, the President's Request for 2008, and Projected Historical Trends through 2010.
All values have been adjusted according to the Biomedical Research and Development Price Index on the basis of a standard set of relevant goods and services (with 1998 as the base year). The trend line indicates average real annual growth between fiscal years 1971 and 1998 (3.34%), with projected growth (dashed line) at the same rate. The red square indicates the president's proposed NIH budget for fiscal year 2008, also adjusted for inflation in biomedical research.

The previous downturns in the NIH funding (and you can verify the scientist complaining by looking through old Science magazines, btw) occurred approximately in the late 1960s, the early 1980s* and the early 1990s. I happened to join this career path right around the 1990s downturn and I remember the whining about grant funding quite clearly. That 1990s downturn was what led to the infamous NIH Doubling. The late 1960s downturn led to Congressional action as well. In both cases you can see where the lapse in Congressional interest led to the following episode of downturn. It is here that we should also review the subsequent update on the Undoubling graph, the even more sinister Defunding Graph.
NIHBudget-MAW-edit-497x400
Via Michael White, presumably via John F Sargent, Jr.

It should be emphatically clear to even the casual observer that the magnitude of the decline in the NIH budget and the duration of the downturn prior to the next Congressional rescue differs. Dramatically. Make sure you check the corresponding longitudinal trends in grant success rates. In case you are wondering about the most recent numbers, according to Sally Rockey, the overall RPG success rates for FY 2012-2014 are 17.6%, 16.8% and 18.1%, respectively. Things are most emphatically not good for the kids these days.

These are the facts. We can argue until the cows come home over how and why various up and down cycles have occurred. We can dispute whether Congressional appropriations intended to rescue the NIH extramural community do harm, good, a balance of the two and what this means for the future.

It is not optional, however, to act like the present downturn is of the same magnitude or impact as the prior ones.
__
*"I remember multiple study section rounds in which nothing ended up getting funded" --a senior colleague

98 responses so far

Generational privilege under the NIH system in one easy video

Feb 17 2015 Published by under Fixing the NIH, NIH, NIH Careerism

Another key bit of information to which I frequently refer when describing generational privilege is depicted in this video from the NIH.

Facts matter.

16 responses so far

Repost: More data on historical success rates for NIH grants

Feb 17 2015 Published by under Fixing the NIH, NIH, NIH Careerism, NIH funding

Our recent discussion of topics related to the Emeritus Award being considered by the NIH powers that be has been robust. I, of course, have been reminding one of the target demographic scientists that she and her generation have had a pretty good run under the NIH system. It seemed like a good moment to remind everyone that there are data upon which to base our understanding of how difficult it has and has not been for various scientific generations. Time to repost an older blog entry.

This was first posted 11 July 2012.


Thanks to a query from a reader off the blog and a resulting request from me, our blog-friend microfool pointed us to some data. Since I don't like Tables, and the figure on the excel file stinks, here is a different graphical depiction:

The red trace depicts success rates from 1962 to 2008 for R01 equivalents (R01, R23, R29, R37). Note that they are not broken down by experienced/new investigators status, nor are new applications distinguished from competing continuation applications. The blue line shows total number of applications reviewed...which may or may not be of interest to you. [update 7/12/12: I forgot to mention that the data in the 60s are listed as "estimated" success rates.]

The bottom line here is that looking at the actual numbers can be handy when playing the latest round of "We had it tougher than you did" at the w(h)ine and cheese hour after departmental seminar. Success rates end at an unusually low point...and these numbers stop in 2008. We're seeing 15% for R01s (only) in FY2011.

Things are worse than they've ever been and these dismal patterns have bee sustained for much longer. If we look at the ~30% success rates that ruled the day from 1980-2003, the divergence from the trend from about 1989 to 1996 was interrupted in the middle and, of course, saw steady improvement in the latter half. The badness that started in FY2004 has been 8 unrelieved Fiscal Years and shows no sign of abatement. Plus, the nadir (to date) is much lower.

Anyone who tries to tell you they had it as hard or harder at any time in the past versus now is high as a kite. Period.

Now, of course, it IS true that someone may have had it more difficult in the past than they do now, simply because it has always been harder for the inexperienced PIs to win their funding.

RPGsuccessbyYear.png
source
As we know from prior posts, career-stage differences matter a LOT. In the 80s when the overall success rate was 30%, you can see that newcomers were at about 20% and established investigators were enjoying at least a 17%age point advantage (I think these data also conflate competing continuation with new applications so there's another important factor buried in the "Experienced" trace.) Nevertheless, since the Experienced/New gap was similar from 1980 to 2006, we can probably assume it held true prior to that interval as well.

3 responses so far

A tiny bias goes a long way when it comes to grant review

Feb 11 2015 Published by under Fixing the NIH, NIH

From ScienceInsider:

Now, a new computer simulation explores just how sensitive the process might be to bias and randomness. Its answer: very. Small biases can have big consequences, concludes Eugene Day, a health care systems engineer at the Children's Hospital of Philadelphia, in Research Policy. He found that bias that skews scores by just 3% can result in noticeable disparities in funding rates.

T. E. Day, The big consequences of small biases: A simulation of peer review, 2015, Research Policy [epub ahead of print 28 Jan] [Publisher Site]

from the paper Abstract:

When total review bias exceeds 1.9% of grant score, statistically significant variation in scores between PC and NPC investigators is discernable in a pool of 2000 grant applications. When total review bias exceeds 2.8% of total grant score, statistically significant discrepancies in funding rates between PC and NPC investigators are detectable in a simulation of grant review.

Day generated a Preferred Class of applications and a NonPreferred Class of applications and ran a bunch of 3-reviewer scenarios with and without reviewer bias against the NPC applications. As far as I can tell the takeaway conclusion about funding here refers to a situation in which the effective payline is 10%. You will immediately grasp that NIH grant review was a strong contributor to the model parameters.

I will admit I am only able to grasp the main points here and I am in no way able to evaluate the nitty gritty.

But it appears to have a very strong message. Namely, that our introspections that "well, if there is bias it is very tiny so we don't have to be worried about it" needs to change.

There is something even scarier in this paper. From the Discussion:

The threshold level of bias in this environment seems to be 2.8% of the total possible score of the grant; this is the level at which the 95% CI of the odds ratio “kisses” 1.00. This represents a single reviewer with a bias of 0.75 points (or three reviewers each with biases of 0.25 points), which is less than half (44.4%) of the standard deviation in a single reviewer’s score. What this suggests is that levels of bias which are sub-noise – that is, that are dramatically less detectable than normal variation in reviewer scores – are sufficient to substantially bias the number of funded applications in favor of preferred investigators.

RIGHT???? The bias can be of smaller effect size than many "normal" sources of variability in scoring that we accept as the resolution of the system. And it still leads to a statistically significant bias in funding outcome.

We are talking in recent days about bias in favor of highly established, older scientists. It has been longer but the Ginther report indicating disparity of grant review outcome for African-American PIs is clearly relevant here.

What this simulation cannot do, of course, is to model the cumulative, iterative effects of review bias. Namely, the way that selection of PC applications for funding has a tendency to increase the bias in the reviewer pool, since those beneficiaries become the next reviewers. Also, the way that over the long haul, disparity of the first award can lead to actual quality differences in the subsequent applications because PI #1 had the money to pursue her science and PI #2 did not have as easy of a time generating data, publishing papers and recruiting postdoc talent.

5 responses so far

Thought of the Day

Feb 11 2015 Published by under Grant Review, Grantsmanship, NIH Careerism

I hate when I review grant proposals that are good, but clearly have been made pedestrian and conservative through the school of hard knocks. There is so much awesome that could be done by these people. It is so clear to me what the really high impact version of this grant should look like. (Not having any illusions about my own unique brilliance, I assume they could see it too. )

But the review realities batter PIs down into a defensive crouch, worried that if they step too far past their Preliminary Data or established expertise they will get crushed.

Because, of course, they would get crushed.

Sometimes I wish I were the Boss of Science more than other times.

28 responses so far

Perspectives from senior scientists on the Emeritus award discussion

Feb 09 2015 Published by under #FWDAOTI, NIH, NIH Careerism

Tthe comments just keep coming over at RockTalking.

8581+ year old guy:

In 2012-13 my NIH renewal proposal with 4 specific aims was turned down 2X by the GM, NCSD Panel, with 35%+ priority scores. ...I appealed the grant reviews to the GM Council and they awarded the grant to me for 3 years at somewhat reduced funding. This funding will finance my lab until Sept 2016, after which I will close down. I am NOT closing down because I have lost my energy for, or interest in, research: [blah, blah we published and showed them!] I am closing down my lab because I can no longer put up with the aggravation of having my grants turned down

Cry me a river.

another senior investigator is on fire:

So far, all the comments fully support age discrimination. How sad! -

Age limits are silly and discriminatory. Merit worked well UNTIL there was no more money in the NIH bank.


There should be COMPETITIVE opportunities for scientists at all stages of their careers (notice I said competitive) . So it’s not a handout at all and it would be merit-based….just like those for newly trained scientists.

There are wonderful examples of some amazing senior scientists. ONE special initiative is not unreasonable for this group!

Finally, I don’t believe I said that I “deserve” anything… except not to be discriminated against for age, gender or whatever characteristic you wish to select.

emphasis added.

updated: omg, the old guy again!

At the last American Society for Cell Biology meeting in Philadelphia (December, 2014) I stood up at the membership meeting ( only ca. 30 people) attended by some of the senior wheels in cell biology. I asked that the society appoint a new standing committee that did
nothing but try to re-design the NIH extramural grant system in its entirety, eg. the grant applications, who can apply, how review panels are chosen, requirements for NIH supported investigators to serve on panels, and how key personnel of the NIH bureaucracy are chosen, and what type of grants should be awarded. My suggestion was not greeted with great enthusiasm and, in fact, elicited some negative comments.

Bashir points out something that is also important about this proposal.

After all that determination to do something after Ginther Report, with mentoring groups and other round-a-bout approaches aimed at eventually addressing racial disparities in grant awards, suddenly we may have a very direct new mechanism that, regardless of the underlying logic, is essentially un-diversity.

56 responses so far

Closeout funding

Feb 05 2015 Published by under Fixing the NIH, NIH, NIH Careerism, NIH funding

Within the past thre years or so I had a Program Officer mention the idea of "closeout funding" to me.

One of my top few flabbergasting moments as an extramural scientist.

It referred, of course, to them using program discretion to give a softer landing to one of their favored long-time PIs who had failed to get a fundable score on a competing renewal. It was said in a context that made it clear it was a regular thing in their decision space.

This explains an awful lot of strange R56 BRIDGE (to nowhere) awards, I thought to myself.

I bring this up because I think it relates to this week's discussion of the proposed "emeritus award" concept.

13 responses so far

Wow, the NIH seeks input on an amazing new initiative!

From NOT-OD-15-064 we learn:

The strategies for sustaining a vibrant biomedical research enterprise are complex and multi-faceted.  NIH strives to support not only the biomedical research workforce, but to support the foundation of research programs that our workforce has created. 

An important issue for NIH is the long term success  for the research we support.  Over the years, NIH has been persistent and creative in efforts to support early career investigators through policy changes and new programs.  But we must also consider the needs of our mid-career investigators and how NIH can assist with the continuation of their well-established research programs, since evidence has shown that the most innovative and productive years of work come from PIs in the 40-59 year old age range.  While these highly productive investigators are happy pursuing their research questions in the laboratory, current funding stresses have hindered the current generation relative to past generations of mid-career scientist.  Our most vibrant investigators have invested their careers to establish the intellectual and technical infrastructure needed to pursue their research questions, and it is in our interest to facilitate progress in their established programs.

Yes, I agree! Totally true.

Therefore, NIH would like to explore potential mechanisms to facilitate the needs of the most productive members of our biomedical workforce. We would like to gauge community interest in a new type of award that could allow established investigators to maximize their output under funding from NIH research grants, while greatly advancing our scientific knowledge and resources. Such an award could permit an established investigator to form partnerships with other faculty members in order to facilitate research inquiry in an efficient and cost-effective way under P-mechanisms as with prior generations. The established investigator would, of course, be expected to train and equip junior colleagues to contribute to mutual interests and research projects while working with them in a mentoring role. If such a collaboration is not feasible, a mid-career award might allow some established investigators to complete expansive projects within their own laboratories.

Wow. Really good stuff here NIH. Glad to see you finally recognizing what brung ya and what you need to bring to the race to keep on winning.

Request for Information This Request for Information (RFI) seeks input from the research community, including scientists from all career stages; research administration professionals; departmental chairs; deans; professional societies; and other interested stakeholders. Public comment is sought for the following:

  • Community interest in an award that allows a mid-career investigator to flourish without being dependent on submitting so many NIH research grant applications
  • Ideas for how one would utilize a mid-career award (e.g., to facilitate laboratory sustainability; to promote novel research inquiry; to provide opportunities for expansion of larger collaborative research projects)
  • Suggestions for the specific characteristics for a mid-career award (e.g., number of years of support; amount of support; mechanisms of evaluation)
  • Ways in which NIH could incentivize the use of a mid-career award, from the perspectives of both mid-career investigators and institutions
  • Impediments to the participation in such an award program, from the perspectives of both mid career investigators and institutions
  • Any additional comments you would like to offer to NIH on this topic

Oh, for sure. I'm going to run, not walk, right on over to the form to submit my approval.

 

Oooooooohhhhh.

 

Wait.

 

This is for EMERITUS faculty? Such as those past the age of 65 who keep on submitting copious numbers of research grants? And the NIH wants to somehow use this to persuade the unwilling* to wind down their lab in good order?

 

What a disaster.

 

Additional Reading:

DataHound

RockTalking

 

__

*This mechanism for winding down a PI's career while sustaining his** "legacy" laboratory and program already exists and is in current practice. A senior PI simply steps down from the PI position and the University nominates a junior person to take over. Maybe with continued Co-investigator status for the Emeritizing person. It works to serve this goal. It is proven.

 

**yeah, "his". that's who these people are. For the most part.

73 responses so far

« Newer posts Older posts »