Archive for the 'NIH Careerism' category

What do you know, the NIH has not solved the revision-queing, traffic holding pattern problem with grant review.

Nov 14 2014 Published by under Fixing the NIH, NIH, NIH Careerism

Way back in 2008 I expressed my dissatisfaction with the revision-cycle holding pattern that delayed the funding of NIH grants.

Poking through my pile of assignments I find that I have three R01 applications at the A2 stage (the second and "final" amendment of a brand new proposal). Looking over the list of application numbers for the entire panel this round, I see that we have about 15% of our applications on the A2 revision.

Oi. What a waste of everyone's time. I anticipate many reviewers will be incorporating the usual smackdown-of-Program language. "This more than adequately revised application...."

I am not a fan of the NIH grant revision process, as readers will have noticed. Naturally my distaste is tied to the current era of tight budgets and expanding numbers of applications but I think the principles generalize. My main problem is that review panels use the revision process as a way of triaging the review process. This has nothing to do with selecting the most meritorious applications for award and everything to do with making a difficult process easier.

ReviewBiasGraph1The bias for revised applications is supported by funding data, round-after-round outcome in my section as well as supporting anecdotes from my colleagues who review. ... What you will quickly notice is that only about 10% of applications reviewed in normal CSR sections get funded without being revised. ... If you care to step back Fiscal Year by Fiscal Year in the CRISP [RePORTER replaced this- DM] search, you will notice the relative proportions of grants being funded at the unrevised (-01), A1 and A2 stages have trended for more revising in concert with the budget flattening. I provide an example for a single study section here ... you will notice if you review a series of closely related study sections is that the relative "preference" for giving high scores to -01, A1 and A2 applications varies somewhat between sections. This is analysis is perhaps unsurprising but we should be very clear that this does not reflect some change in the merit or value of revising applications; this is putting good applications in a holding pattern.

In the mean time, we've seen the NIH first limit revisions to 1 (the A1 version) for a few years to try to get grants funded sooner, counting from the date of first submission. In other words, to try to get more grants funded un-Amended, colloquially at the -A0 stage. After an initial trumpeting of their "success" the NIH went to silent running on this topic during a sustained drumbeat of complaints from applicants who, apparently, were math challenged and imagined that bringing back the A2 would somehow improve their chances. Then last year the NIH backed down and permitted applicants to keep submitting the same research proposal over and over, although after A1 the clock had to be reset to define the proposal as a "new" or A0 status proposal.

I have asserted all along that this is a shell game. When we were only permitted to submit one amended version, allegedly the same topic could not come back for review in "new" guise. But guess what? It took almost zero imagination to re-configure the Aims and the proposal such that the same approximate research project could be re-submitted for consideration. That's sure as hell what I did, and never ever got one turned back for similarity to a prior A1 application. The return to endless re-submission just allowed the unimaginative in on the game is all.

Type1-2000-2013 graph-2
This brings me around to a recent post over at Datahound. He's updated the NIH-wide stats for A0, A1 and (historically) A2 grants expressed as the proportion of all funded grants across recent years. As you can see, the single study section I collected the data for before both exaggerated and preceded the NIH-wide trends. It was as section that was (apparently) particularly bad about not funding proposals on the first submission. This may have given me a very severe bias..as you may recall, this particular study section was one that I submitted to most frequently in my formative years as a new PI.

It was clearly, however, the proverbial canary in the coalmine.

The new Datahound analysis shows another key thing which is that the traffic-holding, wait-your-turn behavior re-emerged in the wake of the A2 ban, as I had assumed it would. The triumphant data depictions from the NIH up through the 2010 Fiscal Year didn't last and of course those data were generated when substantial numbers of A2s were still in the system. The graph also shows taht there was a very peculiar worsening from 2012-2013 whereby the A0 apps were further disadvantaged, once again, relative to A1 apps which returns us right back to the trends of 2003-2007. Obviously the 2012-2013 interval was precisely when the final A2s had cleared the system. It will be interesting to see if this trend continues even in the face of the endless resubmission of A2asA0 era.

So it looks very much as though even major changes in permissible applicant behavior with respect to revising grants does very little. The tendency of study sections to put grants into a holding pattern and insist on revisions to what are very excellent original proposals has not been broken.

I return to my 2008 proposal for a way to address this problem:


So this brings me back to my usual proposal of which I am increasingly fond. The ICs should set a "desired" funding target consistent with their historical performance, say 24% of applications, for each Council round. When they do not have enough budget to cover this many applications in a given round, they should roll the applications that missed the cut into the next round. Then starting the next Council round they should apportion some fraction of their grant pickups to the applications from the prior rounds that were sufficiently meritorious from a historical perspective. Perhaps half roll-over and half from the current round of submissions. That way, there would still be some room for really outstanding -01 apps to shoulder their way into funding
The great part is that essentially nothing would change. The A2 app that is funded is not going to result in scientific conduct that differs in any substantial way from the science that would have resulted from the A1 / 15%ile app being funded. New apps will not be any more disadvantaged by sharing the funding pie with prior rounds than they currently are facing revision-status-bias at the point of study section review....a great deal of time and effort would be saved.

11 responses so far

Top down or bottom up? NIH RFAs are a two-way discussion between Program and Investigators

One of the erroneous claims made by Steven McKnight in his latest screed at the ASBMB President's space has to do with the generation of NIH funding priorities. Time will tell whether this is supposed to be a pivot away from his inflammatory comments about the "riff raff" that populate the current peer review study sections or whether this is an expansion of his "it's all rubbish" theme. Here he sets up a top-down / bottom-up scenario that is not entirely consistent with reality.

When science funding used to be driven in a bottom-up direction, one had tremendous confidence that a superior grant application would be funded. Regrettably, this is no longer the case. We instead find ourselves perversely led by our noses via top-down research directives coming from the NIH in the form of requests for proposals and all kinds of other programs that instruct us what to work on instead of asking us what is best.

I find it hard to believe that someone who has been involved with the NIH system as long as McKnight is so clueless about the generation of funding priorities within the NIH.

Or, I suppose, it is not impossible that my understanding is wrong and jumps to conclusions that are unwarranted.

Nevertheless.

Having watched the RFAs that get issued over the years in areas that are close to my own interests, having read the wording very carefully, thought hard about who does the most closely-related work and seeing afterwards who is awarded funding... it is my belief that in many, many cases there is a dialog between researchers and Program that goes into the issuance of a specific funding announcement.

Since I have been involved directly in beating a funding priority drum (actually several instruments have been played) with the Program staff of a particular IC in the past few years and they finally issued a specific Funding Opportunity Announcements (FOA) which has text that looks suspiciously similar to stuff that I have written, well, I am even further confident of my opinion.

The issuance of many NIH RFAs, PAs and likely RFPs is not merely "top-down". It is not only a bunch of faceless POs sitting in their offices in Bethesda making up funding priorities out of whole cloth.

They are generating these ideas in a dialog with extramural scientists.

That "dialog" has many facets to it. It consists of the published papers and review articles, conference presentations, grant applications submitted (including the ones that don't get funded), progress reports submitted, conversations on the phone or in the halls at scientific meetings. These are all channels by which we, the extramural scientists, are convincing the Program staff of what we think is most important in our respective scientific domains. If our arguments are good enough, or we are joined by enough of our peers and the Program Staff agree there is a need to stimulate applications (PAs) or secure a dedicated pool of funding (RFAs, PASs) then they issue one of their FOA.

Undoubtedly there are other inputs that stimulate FOAs from the NIH ICs. Congressional interest expressed in public or behind the scenes. Agenda from various players within the NIH ICs. Interest groups. Companies. Etc.

No doubt. And some of this may result in FOAs that are really much more consistent with McKnight's charge of "...programs that instruct us what to work".

But to suggest that all of the NIH FOAs are only "top-down" without recognizing the two-way dialog with extramural scientists is flat out wrong.

15 responses so far

Rockey looking to leave the NIH

Nov 03 2014 Published by under NIH, NIH Careerism, NIH funding

It looks like Sally Rockey, Deputy Director in charge of NIH's Office of Extramural Research since 2005, is looking to depart the NIH.

and

show that she is on the short-list to become the next President of the University (system) of Nebraska. Other shortlist candidates are a state level commissioner of higher education, a State University President and a State University (system) chancellor.

In the US some State University systems (i.e., multiple campuses which act as semi-autonomous Universities) call their campus heads President and the System-wide leader the Chancellor whereas other systems reverse these titles. This job appears to be the system-wide leadership position. This explains why there are two system-level leaders in the hunt.

It also may influence your opinion on the appropriateness of someone who has been a research administrator her whole career being in the running for such a position. Obviously she is being looked at as some sort of Federal grant rainmaker/expert to upgrade the amount of money that enters the University of Nebraska system from the Federal government and possibly other sources. I cannot imagine why else such a person, with no related experience heading a University or University system would be on the shortlist otherwise.

The main point of this news can be summed up in this handy figure from Jimmy Margulies, New Jersey Record, who was commenting on a different topic. The point remains, however.
The NIH is a sinking ship. I suspect that the folks at NIH realize this and the ones who have opportunity to cash in on their authoritah! by finding a nice top level administrative gig at one of the supplicant Universities will do so. The have-not Universities which find themselves in the most difficulty obtaining NIH funding will be desperate to land a rain-maker and even the "have" Universities may see this as a good investment. Especially if you have an IC Deputy Director or better, you can argue that they have significant administrative experience within an organization not entirely unrelated to academics. It should be an easy sell for a search committee to make the argument for NIH insiders to be considered for University President positions, Deans of Research and the like.

Is it a smart move? Well yes, if you think that their will be some benefit to their insider status. If you think that the replacement figures and holdovers will take the calls of these NIH emigres and listen to the concerns of their new University.

UPDATE: This news account explains that an attempt to close a Nebraska open-records law was made when the previous President of the UN system resigned.

As the law stands now, candidates may be kept private until the search for a president is narrowed to a pool of at least four applicants, all of whom must be disclosed. The bill would have allowed search committees to keep confidential presidential, vice presidential and chancellor candidates until they’ve narrowed the pool to one finalist.

Proponents of the bill say a closed search would allow for a better pool of applicants, including those who may otherwise be hesitant to apply and jeopardize their current position by publicly seeking another one. Opponents say the current law allows for students, faculty, the general public and the media to meet, investigate and learn about the candidates.

Hadley introduced the bill on behalf of the University of Nebraska’s Board of Regents after President James Milliken announced last month that he would be leaving Nebraska to become chancellor of the City University of New York.

51 responses so far

Tenure expectations and PI dropout from NIH funding

Oct 31 2014 Published by under Careerism, NIH, NIH Careerism, NIH funding

Rockey's post on PI survival also had a graph on the persistence of PIs in submitting applications.

IMG_1135.JPG

She noted that the 1989 first-R01-equivalent folks dropped off in their grant submitting persistence around years 3-5 more than the younger cohorts.

A comment by qaz on the prior post of mine triggered a thought.

What about people who otherwise didn't really want or need a NIH R01 grant but it was a requirement for (or strongly supportive of) a tenure case?

If department expectations/preferences (for tenure or in who they hired in the first place) have changed since the late 80s, this could explain the difference in early drop-out, one-and-done rates across cohort.

19 responses so far

One-and-done NIH Grants: A bug or a feature?

Oct 29 2014 Published by under Fixing the NIH, NIH, NIH Careerism

There is an interesting new survival analysis for PIs that have been awarded at least one R01 posted at RockTalk.

Briefly, what you are looking at is as follows:


We chose three cohorts of first-time R01-equivalent awardees — those who received their first R01-equivalent award in 1989, 1997, or 2003....We used data on these three cohorts for a Kaplan-Meier analysis to look at rates of retention. ...So we used this to analyzing the number of years between the first year of R01-equivalent funding, and the last time an individual receives any additional research project grant (RPG) funding – whether it be from the non-competing continuation of their 1st R01 or another RPG award.

The bulk of Sally Rockey's analytical comments focus on the sharp dropout associated with the first (presumed) interval of support, i.e., 3-5 years after the very first R01-equivalent award. And her conclusion seems to be that this is a problem that needs fixing.

These data seem to support the concept that if there is an intervention needed in retaining scientists in research, it would need to come at the renewal stage of the first award, or as some call it the “second” award. Indeed, we are giving increased focus to this stage through some of our new award mechanisms, such as the National Cancer Institute’s Outstanding Investigator award, and will continue to seek ways of keeping our talent from leaking out of the pipeline.

I am not sure that I agree with this general conclusion from the data presented.

Remember, we are talking system-wide statistics here, not the fate of your five closest colleagues, your training mentor or yourself.

The extreme case would be that once anyone manages to land an R01-equivalent award as a PI, that the NIH should move heaven and earth to keep them funded for the duration of their career. That is an arguable position, but I think it is wrong.

It is wrong for two reasons, which are related to each other. The first reason is that if we re-adjust the system to keep everyone in once they have entered, this will sharply reduce the entrants. It will reduce the number of people who get a chance to prove themselves. This may seem fine and dandy once you have passed the first-R01 hurdle yourself, but this is mind bogglingly forgetful of the position one was in before this and mind bogglingly arrogant in assuming you would have been one of the lucky few.

From a system perspective, this cuts down on scientific diversity. It cuts down on the ability to try out a range of scientific ideas and approaches to see which ones stick. It substitutes the limitations of advance prediction for the virtues of empirical testing.

This would also increase stagnation and slow progress. It would. When you have sinecure funding, I'm sorry but the pressures are not as high to be creative, productive and to diversity your scientific thinking. Yes, from our current vantage point of the amount of time spent securing funds versus doing the science, this may look better than the usual. But take the longer view here. Our competitive system has its virtues in terms of clearing out the dead wood and encouraging better efforts from those who are actively participating.

I think the real question here is the appropriate balance. The desired survival rate.

And we should be very clear that we expect there to be some amount of PI dropout.

Personally, I think that having the major reduction in PIs after the first interval of funding makes a lot of sense. Better than at year 11 or 16, for example. A given PI has had a chance to try out her ideas. He has been given the opportunity to show what he can do. Again, on a system-wide basis, some of these individuals are going to fail so badly that they are never funded again. Do recognize that many will suffer intervals of no-R01 and then come back. Datahound addressed that in a post earlier this year. But many will disappear. Wouldn't it make more sense to have it happen before the NIH has wasted another 5 year interval on them?

Now before I get too far down the path, I will recognize that this is only the start of the data analysis. We want to know a few more things about who is shelled out of the system, never to return, who is able to fight back in after a gap and who is able to sail along with continual funding. This will allow us to see what may be undesired effects or categories of PI/applications that we wish to specifically protect. Is this part of the perfect storm that hits women particularly hard? Human subjects research? Physician-scientist PIs? Ecology or sociology?

Nevertheless, when I look at these initial survival curves, I just don't see the problem that seems so obvious to Dr. Rockey. I don't see how this is the next problem that requires special fixes from the highest offices at NIH.

74 responses so far

Your Grant in Review: Follow the Reviewers' Style Guide

Oct 27 2014 Published by under Grant Review, Grantsmanship, NIH Careerism, NIH funding

The NIH grant application has a tremendous amount of room for stylistic choice. No, I'm not talking about Georgia font again, nor your points-leaving choice to cite your references with numbers instead of author-date.

Within the dictated structure of Aims, Significance, Innovation, etc, there is a lot of freedom.

Where do I put the Preliminary Data now that there is no defined section? What comes first in the Approach- Aim 1? The Timeline? A bunch of additional rationale/background? Do you start every Aim with a brief Rationale and then list a bunch of Experiments? Which methods are "general" enough to put them at the end of Aim 3?

Do I include Future Directions?

What about discussion of Possible Pitfalls and Alternate Considerations and all that jazz?

Is the "Interpretation" for each Aim supposed to be an extensive tretise on results that you don't even have yet?

In all of this there is one certainty.

Ideally you are submitting multiple applications to a single study section over time. If not that, then you are likely submitting a revised version of an application that was not funded to the same study section that reviewed it in the first place. Study sections tend to have an evolved and transmissible culture that changes only slowly. There is a tendency for review to focus (overfocus, but there you have it) on certain structural expectations, in part as a way to be fair* to all the applications. There is a tendency for the study section to be the most comfortable with certain of these optional, stylistic features of a grant application included in juuuust the way that they expect.

So, and here is the certainty, if a summary statement suggests your application is deficient in one of these stylistic manners just suck it up and change your applications to that particular study section accordingly.

Is a Timeline silly when you've laid out a very simple and time-estimated set of experiments in a linear organization throughout the Aims? Perhaps. Is it idiotic to talk about alternatives when you conduct rapid, vertically ascending eleventy science and everything you propose right now is obsolete by the time Year 2 funds? Likely. Why do you need to lead the reviewers by the hand when your Rationale and experimental descriptions make it clear how the hypothesis will be tested and what it would mean? Because.

So when your summary statement suggests a stylistic variant that you wouldn't otherwise prefer...just do it.
__
Additional Your Grant in Review posts.

*If the section has beaten up several apps because they did not appropriately discuss the Possible Pitfalls, or include Future Directions, well, they have to do it for all the apps. So the tendency goes anyway.

59 responses so far

Should we continue long-funded NIH grant programs under younger PIs?

Oct 13 2014 Published by under Careerism, NIH, NIH Careerism, NIH funding

In the course of discussing the infamous graph showing the longitudinal increase in the median age of first-R01 award, and the other infamous slide deck showing the aging of the distribution of all NIH-funded PIs there is something that eventually comes up.

To wit, how do we ease the older investigators out of the system, at least to the extent of cutting down how many grants they submit and are awarded?
Continue Reading »

13 responses so far

Supporting postdoctoral training activities on research grants

Oct 10 2014 Published by under NIH, NIH Careerism, Postdoctoral Training

Jeepers.

This has to be the longest winded and most indirect way of putting it.

My interpretation (I could be wrong) is that yes, you can use R-mech research grant funds to send your postdocs to meetings, give seminars and do other postdoctoral training activities that are not directly related to the goals of the research grant.
Continue Reading »

15 responses so far

A pants leg can only accommodate so many Jack Russells

Oct 07 2014 Published by under #FWDAOTI, Careerism, NIH, NIH Careerism

Some DAOTI asked a silly question

got the simple answer of "no", demanded data and was summarily mocked. For this he got all fronty.

because of course he already knew the answer he wanted to hear in response to his question.

This all arose in the wake of an article in the Boston Globe about the postdoc glut that contained this hilariosity.

“They really are the canary in the coal mine,” said Marc Kirschner, a professor of systems biology at Harvard Medical School whose lab of 17 scientists includes 12 postdocs. “They decided they’d go ahead and try to understand why a cancer cell is different from a normal cell, and here they are a few years out. They knew it was a competitive situation, and they were going to work very hard, but they didn’t see the whole system was going to sour so quickly.”

I bolded for the slower reader. My initial reaction was:

Right?

On to the point.

The world of "R1, TT" positions in science is incredibly diverse, yes, even within "biomedical" or just plain "biology". I repeatedly urge postdocs who feel helpless about the glut of postdocs to start by doing some research. Find out ALL of the people who have recently been hired all across the US in jobs that are somewhat remotely related to your skillset. Note, not your "interests". Your SKILLSET!

I follow this up with a call to do the same on RePORTER to find out about the vast diversity of grants that are funded by the NIH. Diversity in topic and diversity in geographical region and diversity in University or Department stature.

This is even before I tell people to get their "R1" noses out of the air and look seriously at Universities that are supposedly beneath their notice.

So what makes for a successful "competitor" for all of the jobs that are open? Is it one thing? Such as "vertically ascending eleventy systems buzzword biology science" training? That is published in Nature and derives from a 12+ postdoc lab with everyone busily trying to hump the same pantsleg?

"Everyone" here is, guess what? Your competition. And yes, if you choose to only seek out "R1, TT" jobs that are in a University that boatloads of people want to work in, applying techniques to topic domains that a dozen fellow postdocs are also doing right beside you, chasing CNS "gets" that a few scores of labs worldwide are also chasing...well, yes, you are going to be at a disadvantage if you are not training in one of those labs.

But this doesn't also mean that making all of those choices is not also putting you at a disadvantage for a "R1, TT" job if that is your goal. Because it is putting you at a disadvantage.

Vince Lombardi's famous dictum applies to academic careers.

Run to Daylight.

Seek out ways to decrease the competition, not to increase it, if you want to have an easier career path in academic science. Take your considerable skills to a place where they are not just expected value, but represent near miraculous advance. This can be in topic, in geography, in institution type or in any other dimension. Work in an area where there are fewer of you.

Given this principle, no, a big lab does not automatically confer an advantage to obtaining a tenure-track position at an R1 university. According to Wikipedia the US has 108 Carnegie-approved "Very high research activity" Universities. Another 99 are in the next bin of "high research activity" and this includes places that would be quite reasonable for someone who wants to be an actual teaching + research old school professor. I know many scientists at these institutions and they seem to be productive enough and, I assume*, happy to be actual Professors.

Would coming from a big lab be a help? Maybe. But often enough search committees at R1s (and the next bin) are looking for signs of independent thought and a unique research program. That is hard to establish in a big lab...far easier to demonstrate from a lab with one or two concurrent postdocs. Other times, the "big labs" in a field (say, Drug Abuse) are simply not structured like they are in cannon-fodder, bench-monkey, GlamourHumping, MolecularEleventy labs. Maybe this is because the overall "group" organized around the subject has Assistant Professors where those "big labs" have Nth year "postdocs". Maybe it is because this just isn't the culture of a subfield. If that is the case, then when an R1 is hiring in your domain, they aren't expecting to see a CV that competes with three other ones just like it from people sharing your lab. They are expecting to see a unique flower with easily discernible individual contribution to the last three years of work from that small lab. That type of candidate has an advantage for this particular job search.

So yeah. It is a stupid question to ask if [single unique training environment] confers an "advantage" for some thing as general as a tenure-track job at an R1 University.

I'll close with a tweet from yesterday:

and a followup

This all reminds me of a famous Twitter "independent scientist" jackhole who applied to a few elite Universities, couldn't get an offer and summarily declared all of science to be broken, corrupt, crowded with "diversity" riff raff and all sorts of other externalizing excuses. Make sure you don't fall into this trap if you are serious about succeeding in an academic career.
__
*actually, they say so.

21 responses so far

CSR says that applications are up by 10%

Oct 02 2014 Published by under Grant Review, NIH, NIH Careerism

Again, according to the Peer Review Notes for September 2014, CSR of the NIH says that applications are up by 10%.

“Total numbers of applications going to CSR study sections have surged about 14 percent," said CSR Director Dr. Richard Nakamura. “The NIH Office of Extramural Research reports about a 10 percent increase in research project grant applications across NIH.”

The difference in the two number is because "CSR is reviewing a slightly larger portion of NIH applications (79%) now than before.", the balance are reviewed in study sections managed by ICs themselves.

Why the bump in applications?

The CSR appears to be blaming this slight increase on the revision of the policy regarding resubmitting previously unfunded applications. As you know, if your revised version (A1) of a proposal is not funded, you may now resubmit it as a "new" application, making no mention whatever of the fact it was previously reviewed.

“It’s clear a large part of this increase is due to NIH removing limits on resubmitting the same research idea,” he said. “The new policy was designed to keep alive worthy ideas that would have been funded had the NIH budget kept up with inflation.”

Obviously, success rates will go down since I see very little chance the budget is going to increase any time soon. The only possible bright spot would be if the recent award of BRAIN Initiative largesse frees up the regular funds within the general framework of neuroscience that would otherwise be won by these folks. I am not holding my breath on that one.

This bump is hitting around the time of the beginning of the fiscal year when there is no appropriation from Congress, as usual. So grants submitted in the summer (first possible after the policy change) will be reviewed this fall and sent to Advisory Councils in January. My assumption is that we will still be under a Continuing Resolution and many ICs will be conservative, per their usual practice.

So anyone who has proposals in for the upcoming Council round has a bit of extra stress ahead. Tougher competition at review and uncertainty of funding all the way through Council. Probably start hearing news about scores on the bubble in March if we're lucky.

The really interesting question is whether this is a sustained trend (and does it really have anything to do with the A2asA0 policy shift).

I bet it will be short lived, IF it has anything to do with that change. Maybe just that one round or at best two rounds and we'll have cleared out that initial exuberance, is my prediction.

20 responses so far

Older posts »