Archive for the 'NIH' category

On recruiting peers to review NIH grants

Jun 30 2015 Published by under NIH, NIH Careerism

May 2015 Advisory Council round for the CSR of NIH.

I'll be making observations on the Luci Roberts presentation in a little while. For now, enjoy.

UPDATE:

Okay, down to business. The part I wanted to highlight starts at 1:35 of the videocast. Dr. Nakamura introduces Luci Roberts, Director of Planning and Evaluation in the OER.

A comment from baltogirl

It is a little known fact that SROs have trouble recruiting for study section (I was told that two-thirds of people asked decline to serve). It's likely that most people are so busy writing their own grants they can't take a full month off to devote to reading the work of others.

reminded me I forgot to discuss these data.

The Division of Planning and Evaluation conducted a survey** some time ago, trying to determine the attitudes of extramural PIs that would affect their willingness to serve on study sections. I thought it applied to the above comment.
They surveyed 4,000 individuals who had submitted at least one grant as PI in the past five years and who had obtained active funding (any variety) from the NIH in the past five years. 1830 or 46% responded. Not too shabby. (They also surveyed 423 SROs of which 271/64% responded, more on that later.) Of the PIs 1,616 had served as PIs or PDs, the balance were TG directors or subproject/consortium PIs.

First up. 964 (53%) had never been asked to serve on study section in the 12 months prior to the survey. Hmm. I can almost stop right here. But no....there's much more. Still, if there is a reviewer crunch, the first order of business is to determine why over half the potential pool is not even being asked.

Ok, of the remaining 866 individuals who were asked to serve on study section, 762 agreed to do so. That's 88% saying yes. So the rumor that "two-thirds of people asked decline to serve" is falsified by this survey. Clearly, the vast majority of people who are asked step up and do their community duty.

I really, really like this. It is heartening.

The next most-interesting thing was the 10th slide which shows the ranks of PIs who are asked/not asked to review. As you would expect, the Full/Associate/Assistant Professor ranks for those asked ran 54%/32%/11% (that high for Assistants?) and 22%/23%/37% for those not asked to serve.

Again, this outcome makes it really clear what needs to be done if getting reviewers is a problem. Ask more Assistant Professors to serve***. Right? Done.

Then we get into a couple of slides on why people might say "no" when asked to review. Slide 11 present the top reasons (out of an open text box response, per Roberts' presentation) for the process being "more burdensome than it could be". The numbers are confused here because the denominator appears to be 861 when it should be 762. But in any case, 630 of the respondents who reviewed said the process was not more burdensome than necessary (huh? surprised on this one). Of the 158 who said it was too burdensome, 45% complained about the number of assigned applications. The next most common (16%) complaint was "too much time devoted to applications that will never fund". So pretty much, the most burdensome thing was review load.

This brings us to Slide 13 which pits SRO opinion versus how reviewers think. One of the biggest disconnects was in the number of in-person meetings per year that is "reasonable" since reviewers lean 1-2 and SROs were about evenly split between 1-2 and 3-4 as okay. A similar disconnect was found on application load. Half of the reviewers felt 4-6 apps was a reasonable load and only 25% felt 7-9 was reasonable. SROs leaned 7-9 (~60% of SROs) with less than 40% finding a 4-6 grant load reasonable.

Predictably.

I'm skipping Slide 12 on reviewer and SRO thoughts on reason to accept and decline invitations to review since 88% of those asked say yes anyway. Who cares what they say if most of them will do it just for the asking?

Okay, those were the things that jumped out at me.

___
*As a disclosure, Dr. Roberts was once a SRO who played a highly formative role in the early-career understanding of the NIH grant business for YHN. She was kind enough to send me the slide deck that was used in her presentation.

**I was a respondent, fwiw.

***In case of any newcomers, both YHN and CPP have advocated for this on this blog since forever.

16 responses so far

Question of the day

Jun 29 2015 Published by under Fixing the NIH, NIH, NIH Careerism

What percentage of K99 should fail to transition to the R00 phase in a healthy system?
What percentage of those that go to R00 should fail to ever gain major independent funding as a PI? 

38 responses so far

A Tweet which captures the problem with NIH's "pipeline" response to Ginther

Jun 29 2015 Published by under NIH, NIH Careerism

14 responses so far

PhysioProffe on the conduct of science

Jun 25 2015 Published by under Careerism, Conduct of Science, NIH

go read:

Self-interested nepotistic shittebagges constantly assert this parade of horribles that if we don’t fund the right subset of scientists in today’s tight scientific funding environment (coincidentally them, their friends, their trainees, and their family members), then we are going to destroy scientific progress. This is because they are delusional......

No responses yet

Peer review and the death sentence

Jun 25 2015 Published by under NIH, NIH Careerism

It is relatively easy to kill grant applications because the reviewer knows the applicant can always apply again.

Same thing for hiring decisions because surely some lesser University will hire the three other candidates on the short list.

In many tenure cases, the Department knows that this person will get a professorial rank job elsewhere*.

Germain's scheme is going to require peers in the field to pass a death sentence on the career. And to make the numbers add up, there will be a LOT of this.

Those peers know that they themselves will be up for chopping in the next 5-7 years.

I do not foresee much enthusiasm for scoring progress as deficient and this will only grow more intense with each successive 5 year review interval.

__
*and anyway these are so personal at this point that it is a different matter.

15 responses so far

Republicans of Science

Jun 25 2015 Published by under NIH, Tribe of Science

10 responses so far

Predicting the future

Jun 24 2015 Published by under NIH, NIH Careerism

One of the biggest whoppers told by Ronald Germain in his manifesto on fixing the NIH is this:

it is widely accepted that past performance, not a detailed research plan, is the best predictor of future success. So why stay with the fiction that R01 grant proposals are the best method for determining support of the individual scientist,

As I often say there is nothing so a-scientific and illogical as a scientist on the business of science.

The way he states this plays the usual sleight of hand with the all-important, unmentioned variable.

Namely, the means to do the research. Grant funding.

There is no bigger predictor for the success of a given research plan submitted to the NIH than whether or not the PI receives the funding to do the work. Funnily enough, Germain actually recognizes this and totally undercuts his argument in one of his caveats:

with 5–7 years of support per round and 1–2 years of bridge funding available, I think it is unlikely that a highly competent investigator will fail to produce enough during 6–9 years of research to warrant a “passing grade” without further extensions, except in extenuating circumstances.

Right? He sees right here that all that matters is funding. Most competent investigators will succeed if they but have the funding! Which makes his idea that this will cut down on competition and the "stochastic" nature of getting the grant funding look as silly as it is.

It's just another way to say "We'll pick our favored winners in advance of any independent accomplishment based on who they trained with (i.e., us!) They will keep right on winning because they will be the only ones with the means to accomplish anything. All others can stay the heck away from our effortless stream of moola, no matter how good their ideas might be".

This is important and it is why basing funding on accomplishment, rather than great ideas and the capacity to fulfill them is recipe for a death spiral of the Extramural NIH productivity as a whole.

This plan will self-reinforce and harden a silo around a limited set of brains, doing science in the way they see fit. Good ideas from outside this silo will not be given a chance to compete....unless they happen to occur to someone inside the silo. And on the whole, that person will not represent a diversity of ideas, approaches and interests. This will, across the enterprise of NIH-funded science, reduce the rate of discovery.

Those who manage to accomplish will continue to have a stranglehold on the means to accomplish. Means leads to accomplishment leads to more means in the Germain scheme.

So what gets accomplished will be narrowed, iteratively, with each 5-7 year review. Only to be refreshed, minimally, with each squeezed down cohort of new hires who manage to make it into his starter, block-grant scenario. Those, of course, will be selected by Universities on the basis of seeming like the people who are already most successful since the review will be anticipated to be on the basis of the person. Naturally, the trainees of the insider club will be most highly sought after. (Take a look at the way HHMIers, espcially the Early Career ones have been trained folks. ...talk about the past predicting the future and all, right?)

So when you hear someone talking about "the best predictor of future performance is past performance", make sure to ask whether that is with or without the funding and how they know this.

The second truthy whopper Germain tells follows soon after.

true creativity is often cause for lower scores?

Personally I have yet to see a well-prepared truly creative grant get killed just for being creative and new. Maybe wackaloon geniuses who have great ideas but simply refuse to write an actual grant proposal struggle in some sections. I guess. But here's a secret for Germain. (A "secret" known to just about anyone who has served on 2-3 traditional standing study sections.) People that he is talking about, those who have demonstrated a high level of accomplishment in the past 5-7 years, get away with utterly crappy proposals and still get their funding based on their record of accomplishment.

That's right. We ALREADY have a system in place that HUGELY benefits and prioritizes the funding of people with a track record of accomplishment. The "creativity" in their proposal does not prevent them from getting funded. Nor, btw, does diverging substantially from the plan they got the money for hurt them in the next round of evaluation.

Given this, there is no conceivable way that switching to Germain's plan changes the ability to be creative.

Now, for those outsiders or people with a brand new idea absent a track record....yeah, they may take it on the chin under the current NIH system. But they would ALSO fail to gain support under Germain's. It isn't like we're inventing up some new peers to do the reviewing here. No matter if it were McKnight's panel of NAS members or Germain's ideas of the deserving elite or traditional NIH-style panels judging the "track record"....there is no way that we can assume that genius PI behind every PCR or gene knockout technology or whatever Nobel-worthy breakthrough will be immediately recognized as awesome and funded.

28 responses so far

Ronald Germain Explains How To Fix The NIH

Continue Reading »

99 responses so far

Gender smog in grant review

Jun 19 2015 Published by under Gender, Grant Review, NIH, NIH Careerism

I noticed something really weird and totally unnecessary.

When you are asked to review grants for the NIH you are frequently sent a Word document review template that has the Five Criteria nicely outlined and a box for you to start writing your bullet points. At the header to each section it sometimes includes some of the wording about how you are supposed to approach each criterion.

A recent template I received says under Investigator that one is to describe how the

..investigator’s experience and qualifications make him particularly well-suited for his roles in the project?

Grrr.

12 responses so far

The Germain nonsense on fixing the NIH

Jun 19 2015 Published by under Fixing the NIH

I know you guys want to talk about this ridiculous commentary because the blog ephone has been ringing off the hook. Unfortunately I really don't have the time for a proper post.

Discuss 

UPDATE: One thing I noticed about the proposal that merits a little more....specific discussion.

I believe the NIH should transition to a system that links getting a first job (faculty appointment) with sufficient funding to support a reasonably sized laboratory (three to five people, including the PI) in terms of staff salaries and supplies

Obviously there is a big range in terms of types of staff and the amounts that they are paid. However, I think we can start with the salaries of a 0 experience postdoc on NRSA scale ($42,840) and a 4 year postdoc (50,112). I am going to use $100,000 as the PI salary.

Benefits can range from 25% to 50% (again, as a rough approximation based on my limited experience with such numbers) which brings us to $241,190 or $289,428 per year for a three person laboratory. That is salary cost only. Obviously types of research vary tremendously but I have heard numbers in the range of 60% to 80% of research grant costs going to support staff salaries. Before we get into that, let's raise the estimates to Germain's upper bound of a lab of 5 individuals, the PI as above and two of each experience level postdocs ($357,380 and $428,856, depending on benefit rate).

With this estimate, if the staff cost is 80%, this brings us to the $357,380-$428,856 per year range. If staff cost is 60% of the research grant expenditure, then $595,633 - $714,760 range.

I invite you to compare these numbers, which Germain is recommending for 5-7 years starting presumably from Day 1, with the funding trajectories of yourself and your peers. At the upper bound, three modular R01s worth of funding for the entire duration of the pre-tenure interval.

This call is for a LOT fewer noob Assistant Professors being allowed to get in the game, by my calculation. Either that, a huge Congressional increase in the NIH budget or a massive retirement of those who are already in the game.

Note that I too would love to see that be possible. It would be fantastic if everyone could get three grants worth of funding to do whatever the heck they wanted, right from the start.

But in the real actual non-fantasy world, that would come with some serious constraints on who can be a scientist.

And I do not like people like Germain's ideas on who those people should be.

59 responses so far

Older posts »