The Broken Pipeline 2: The Funding is Not the Entire Issue

I previously noted a new website (brokenpipeline.org) and glossy report on the career "pipeline" problem currently experienced by biomedical research science in the US. This report catalyzed more discussion in the blogosphere on the issue (SciGuy, Jonathan Gitlin, Greg Laden, Dr. Free-Ride; Update 3/13/08: Orac, Harvard Science on a related Congressional hearing, The Neurocritic, Chris Seper). My readers from the past year will recall that I've discussed these issues at some length.
My readers will also recall that I have some pointed views on what I see as cultural biases against young investigators at the stage of grant review that have little (specifically) to do with the amount of money available. That is today's topic for discussion.


The theme of the broken pipeline analysis starts with the data-backed observations that larger and larger fractions of the NIH grant money are going to older and older investigators. One example, albeit a bit dated, is the following which shows the percentage of NIH research awards by age cohort from 1980-2003.



This PDF is available from the National Academies
http://www.nap.edu/catalog/11249.html
Bridges to Independence: Fostering the Independence of New Investigators in Biomedical
Research Committee on Bridges to Independence: Identifying Opportunities for and Challenges to Fostering Independence of Young Investigators in the Life Sciences, National Research Council
ISBN: 0-309-54964-7, 138 pages, 6 x 9, (2005)

Furthermore the average age of various career transition measures such as initial independent appointment, major NIH grant award, etc are increasing, see next figure. Thus, the hard data suggest that our research force is getting older.



ibid

One of the things these figures also point out to me is that these longstanding trends are not directly tied to the "recent" NIH funding woes, nor to my eye to the approximately 10-yr cycle of NIH boom / bust that has been going on since (at least ) the early 70s. Does any of this sound familiar? (Women in science bonus here)
Where we start getting into punditry is what the graying of science means for the future state of our scientific research force, particularly the independent researcher (aka, Professor, Principal Investigator, Lab Head, Big Cheez, etc) that is really the prime driving force, for want of a better term. (No, not the backbone, that would be the techs. Nor the muscle for the really heavy lifting, that would be the postdocs and grad students.) The committed professional cadre moving forward with the expectation that this will be their primary career for their working life.
There are a number of primary concerns as I see them. First, that the great ideas and innovations of the best and the brightest scientists of the next immediate generation (i.e., exiting the pipeline) will be lost. Second, that the dismal career prospects are actively discouraging the best and the brightest in the next and succeeding generations from even trying to pursue the career. Third the twin bubbles represented by the expansion of academia in the 60s and 70s and by the Boomers (generically) will exit science in rapid order. Even if there is a thin cadre of the best and brightest of the next generation(s) left, these individuals will not have received the training necessary for true "replacement" of the geezers. This latter is a subtle point, perhaps. It reflects my view of science careers as gradually expanding the size and scope of the research a scientist oversees and influences. To be concrete, we can think about the experience necessary for heading large-scale collaborative projects such as Centers and Program Projects.
Without belaboring points that I've made before, hopefully you are starting to see that at least one major "pipeline" problem is more of the distribution of grants, jobs and responsibilities within any given pool of research funding rather than the overall level of funding. Whether the NIH is flush or not, the reasons for wanting to keep a relatively smooth distribution of funding across the age distributions are the same.
This brings me back around to the question of cultural practices.
NIH Director Elias Zerhouni was quoted in The Cancer Letter (V34No9; h/t Abel) as saying: "in 2005, NIH funded 1,683 new investigators but the next year the number dropped to 1,353. That's why last year we agreed to maintain a floor of about 1,523 new investigators per year." (This is consistent with my memory of a ppt slide in one NIH presentation or another which I've lost track of...anyone?)
The point is this. The "NIH", meaning the Program staff of NIH, accomplished this by means of overturning the results of initial peer review! They had to reach down and pull up New Investigator grants out of the review order. Other NIH ICs are making their policies explicit by setting their funding lines for New Investigator grants at a higher percentile (worse score). Why are they doing this? Well, implicitly, they are saying that initial peer review is getting the evaluation of applications wrong. That there are biases creeping into the process that are preventing otherwise meritorious proposals from younger scientists from being treated fairly. This, in my view, is what the personal stories on the BrokenPipeline glossy report are supposed to be telling us as well.
I cannot emphasize enough the degree to which study section behavior is biased for the established investigator. I've been over the arguments in the past and will re-state that this is not always overtly pernicious behavior. It results from cultural evolution, mores and expectations. Natural group dynamics. Like affiliating with like. Granfalloonery and bunny hopping.
To head off common rebuttal, yes, on average the New Investigator application is not as good as the well-established investigator application. On many grounds from the grantspersonship to the actual science. Of course. I am not talking about this. I object to what I see as the few highly meritorious applications from New Investigators not receiving that extra boost (necessary for actual funding) that is credited to run-of-the-mill or even poorly presented proposals from investigators with an established track record. To supposedly objective criteria like "productivity" being applied unfairly (I think one should divide the number of papers by the number of grants or total support dollars, personally). Etc. It is this tendency, IMO, that results in the continuing graying of the PI pool. Trends that much NIH hot air and establishment of new funding mechanisms have done nothing to halt over the past three or four decades. I would argue that this is because they refuse to address the inevitable bias inherent in the way study sections are constituted.
My solution is, as my usual readers are aware, the traditional remedy for bias. To allow competing biases equal voice in the hopes that this reduces the influence of any given bias. The Center for Scientific Review strives very hard for gender, geographic, institutional-type and subfield representation in review.
Guess which grouping factor they explicitly bias against representing? And guess what consistent line-item recommendation is demanded by "stake holder" input to the draft report on the Peer Review Self-Study?
This is why my proposal is that study sections should strive to be representative of the age of the applicant pool just as much as they strive to be representative of other factors such as gender, geographic region and the like.

27 responses so far

  • PhysioProf says:

    I think one should divide the number of papers by the number of grants or total support dollars, personally

    No. Rather than publications per dollar, it should be citations per dollar.

  • DrugMonkey says:

    No. Rather than publications per dollar, it should be citations per dollar.

    That would be "impact" rather than "productivity"!
    But citations would indeed be a fun number to see in the progress report section of a competing continuation.

  • Becca says:

    "(I think one should divide the number of papers by the number of grants or total support dollars, personally). "
    This alone would be have facinating implications. One would assume that the fact that some types of research are more expensive than other types would matter less if it everyone in a particular study group uses mostly the same techniques. Even if that didn't even out, maybe electrophysiology really isn't so critical (*whahaha*)
    And it could get very complicated- should somebody in N. Dakota benefit because they pay their lab personale less than people in Boston? Of course, if you're a nice enough boss you can attract people good enough to come to podunk and do comparable science for less money, maybe you deserve the funding more...
    ohh, the evil.
    I don't think it would fix the game, but it would shake it up a lot.
    "No. Rather than publications per dollar, it should be citations per dollar."
    This doesn't help address the bias toward well-known investigators coasting on their reputations as much. And citations generated from the previous grant cycle alone is very problematic as that will not necessarily indicate what will be the most useful in the long run.
    And because I've gotta ask- what about the age of attaining the PhD or MD degree over the same time period?
    And do we have any data from other fields, in academia and elsewhere?
    I need some reference points!

  • DrugMonkey says:

    One would assume that the fact that some types of research are more expensive than other types would matter less if it everyone in a particular study group uses mostly the same techniques...should somebody in N. Dakota benefit because they pay their lab personale less than people in Boston?
    While this would be entertaining I should point out that conversations about "productivity" don't necessarily get this fine-grained. My point about dividing by the funding is because one of the phrases that I hear commonly is essentially "well, sure the proposal is a bit confused and there are some areas where the methods are not clear...but given the amazing productivity of Dr. Graybeard over the past 5 years I'm certain great stuff will result". It should not be news that the younger investigators often get criticized about their "modest productivity". If one looks into the matter closely and examines overall output, the multiple grants acknowledged in each paper, parses reviews from primary research reports, etc, well all of a sudden the "amazing productivity" of Dr. Graybeard may not look too different from Dr. Newbie on a per-project basis. The aforementioned Graybeard just has more projects going.
    And because I've gotta ask- what about the age of attaining the PhD or MD degree over the same time period?
    From memory it has also increased but not at the same rate. The National Academies report linked above and the one found here
    http://www.nap.edu/catalog.php?record_id=11275
    are great places to start. If you are willing to navigate page by page it is free. In particular look for the graphs on time spent in postdoctoral training for a decent handle on your question.

  • Kay says:

    Age of average first-time R01 recipient: 43
    Age of this college senior: 21.5 -> 43/2
    The point isn't lost on my generation. (I'll be starting my RealJob(TM) in three months, thanks.)

  • bsci says:

    Other NIH ICs are making their policies explicit by setting their funding lines for New Investigator grants at a higher percentile (worse score). Why are they doing this? Well, implicitly, they are saying that initial peer review is getting the evaluation of applications wrong.
    I don't think they're saying their evaluations are wrong. I think what few want to admit is that the top 20% (actual % is a guess) of all grant proposals are all worthy of funding. Since the funding rate is below 20% they need some metric to say which good grants to accept or reject. Not only is there bias in this process, but, especially when the acceptance rate is as low as it is now, there's a large random component. It's best to remove the sources of bias as you suggest, but I still don't see the score of a first time submitted beating someone who has been breathing R01s in their sleep for a few decades. If beyond scores, there is already a focus on diversity of topics within each study section, why not also a diversity of seniority too?
    Anyway, postdocing at the NIH mothership will hopefully keep me out of this pipeline mess for at least a few years until it's more sorted out.

  • acmegirl says:

    This is very worrying. I had a different career for a while before I went to graduate school, so I'm the same age as the average new assistant professor, but I'm still working on my thesis. I didn't used to worry about this - I have lots of energy, and if anything, am more focused now than I was in my twenties, so I've always thought that I would have no trouble keeping up. But if I put myself in the age bracket that corresponds to my training stage, I'll have to keep keeping up for a lot longer if these trends continue.

  • S. Rivlin says:

    I can only speak from my own experience, especially now that I am retired. I do, in general, agree with Drugmonkey's list of reasons for the graying of our funded scientists. I am proud to say that I had never received NIH funding (as a PI) to carry out my research (I was a Co-I on several grants that only peripherally dealt with my main scientific focus). My lack of success in securing NIH funding as a PI was not the result of poor grantmanship or being too young (I submitted my first proposal at the age of 47, since I secured my first and only job at the age of 40). My applications were denied because they proposed ideas that were contradictory to the prevailing notions.
    Even after establishing my scientific ideas as relevant, by securing funding from other, smaller and sometime obscure, funding sources, and even after the NIH had recognized my contributions by selecting me to become a member of a study section, my NIH applications were never funded. After 26 years of doing research in the field of brain energy metabolism in health and disease, the several scientific breakthoughs that my lab is responsible for are regulary being cited in review articles and textbooks.
    The point is, the NIH funding system tends to fund the familiar, the less risky ideas and is stacked against funding ideas that go against the grain, those very ideas that usually lead to breakthroughs.

  • DrugMonkey says:

    This is very worrying. I had a different career for a while before I went to graduate school, so I'm the same age as the average new assistant professor, but I'm still working on my thesis. I didn't used to worry about this - I have lots of energy, and if anything, am more focused now than I was in my twenties, so I've always thought that I would have no trouble keeping up.
    acmegirl, we talk trends and averages but almost always fail to talk about variance around the mean. it is worth pointing out that one should not take the trends and averages as dictating what any individual can or cannot accomplish. I, for one, talk about the trends in the hopes that the reader can use this information to beat the odds, so to speak.
    I would not be surprised in the least if the life experiences and maturity you bring to the table help you to cut a good deal of time off of the "usual" timeline.
    FWIW, despite the fact that I operate in subfields of science that are not teh Hawwt, just about every one of the first dozen or two young guns that I can think of got their first R01 well prior to 42 years of age. NIH-wide stats are not always applicable to every subcategory of job type...

  • Charley Falb says:

    There are some things that could be done:
    1) Start forcing retirement of senior faculty at 70 or 72 years of age - and make them ineligible for NIH grant funding at that time as well. Very few faculty of that age remain extremely productive.
    2) Clean out the deadwood in the NIH intramural program - when I was a postdoc, there were at least two Sections in my advisor's Laboratory who were at best minimally productive. Fifteen years later, they are still there...extrapolate that across NIH, and you could probably fund another 100 deserving young investigators per year.
    3) This one will be unpopular - but why are we training more people than we can support? Cut back Ph.D. admissions, particularly in the second- and third-tier universities. Sorry folks, but someone trained at Harvard and MIT is ALWAYS going to have a better shot at funding than someone trained at Eastern Iowa State and the University of Eastern Montana. That's just the way it is.
    4) Stop expecting that you are going to be handed money for research just because YOU think it's important. Most of the world doesn't give a rat's patootie about biomedical research - until they get sick. Even then, they only care about research directly applicable to treating disease. Basic research, even though serendipity happens, is never going to be as well supported or understood - and in some cases, just doesn't need to be done.
    I do agree that there should be a distribution of age groups in peer review. The distribution of grants by age is very disturbing.

  • acmegirl says:

    Charley Falb,
    re: your third suggestion
    My, isn't that elitist of you. In the interest of addressing a skewed age distribution we'll just reduce the access to scientific training of whole subset of the population that, for various reasons is not going to a top-tier university.
    re: your fourth suggestion
    Disease related research is no better understood by the general public than basic research, judging from the garbled junk that passes for mainstream media coverage. Besides, some of the most useless, "I just thought it would be cool" research is done at the top tier universities you seem to like so much.
    We need to decide if we want the scientists funded by taxpayer money to be representative of the population that is footing the bill. Those two suggestions fail at moving us in that direction. What direction did you have in mind?

  • bayman says:

    A related problem I've observed seems to be the ballooning of established labs. At least here in Canada eh (dunno about your R00210848282a), there seems to be a trend to piling more and more money onto existing labs instead of diverting money to funding new labs. So not only is this keeping young investigators from working on their ideas independently, they also end up working in mega-labs for a PI who has too many trainees to keep track of or interact with (and all the other problems this comes with).
    I say more small-medium labs is better for science than a smaller number of mega-metropolis factory labs.

  • Eric says:

    While metrics on productivity being changed from simply reviewers impressions of journal impact factors and numbers of publications, these aren't going to help a new investigator and certainly aren't going to be comparable between labs, fields or even paper types. Cell culture/in vitro papers ($) compared to mice studies(\(), compared to studies on primates(\)$) are all going to give wildly different results in terms of cost per citation or publication. There is also likely a citation bias in favor of certain labs in various fields based on who has the most papers. In many publications the citation doesn't accurately describe the original work, and I've never heard of a reviewer checking citations (well, except maybe their own).
    Also, new laboratories are going to take awhile for publications to begin appearing during the funding process. For the laboratory I currently work in, the papers published today were started anywhere from 5 years ago (only one very drawn out paper) to six months ago.
    While I emphatically agree that younger investigators need to be funded and that my generation needs to be able to "break into" a scientific career and run our own labs.
    Side note: @#10: The basic scientific research of 20 years ago is the applied research of today. The problem is that you can't tell what basic research will become important.

  • bsci says:

    Charley Falb,
    per your point 2 on NIH intramural deadwood, I don't have the numbers in front of me and I can't speak for the entire NIH (or speak for anyone officially since I am a lowly postdoc), but the number of tenured or tenure-track intramural faculty in NIMH has dropped from around 75 to around 50 in the last decade. Of the faculty in NIMH around half of them weren't faculty at NIMH 10 years ago. While the NIMH total budget increased during the doubling only a small fraction of that increase went to the intramural program meaning the % of the total NIMH budget that goes to the intramural program is MUCH lower than it was a decade ago. Anything else you'd like them to do besides firing the two people you consider deadwood?

  • Charley Falb:
    Regarding your suggestion 3: I think this is brilliant. In fact we should take this a step further; why just limit PhDs? Limit all college admissions, esp to lower tier colleges! We have so many jobs that Americans won't do, are overqualified to do etc, and this only leads to an immigration problem, unemployment amongst educated Americans and all that crap. So if you limit access to college, you solve several problems with one genius stroke! Also, logically, we should eliminate second and third-tier universities themselves---heck their grads WILL ALWAYS be inferior to ivy league grads, so why waste everyone's time and money? Besides, how many non-ivy-league grads or PhDs have ever contributed anything valuable to science, or to any other field for that matter?
    I could go on, with examples of whole classes of people we should eliminate from the granting process based on IQ studies etc, but I'm preaching to the choir here, I'm sure.
    Regarding your suggestion #4: I can't formulate a worthy response till I totally understand the brilliant complexities contained therein. But right off the bat, I will agree with you that we should have just eliminated, way back, work on such useless things as T4 phage, lambda phage, assorted bacteria, fungi, fruit flies, worms, frogs, fish etc etc, as they had no obvious connection to human disease research. What a colossal waste of time and money! Everyone knows that pharma company researchers (they are the people who really cure human diseases, for those who aren't familiar) don't waste their time reading any ivory-tower basic research publication crap---it is all about the applied research baby! Also, isnt it obvious that any discovery worth a crap has come from applied research? Damn, if only we hadn't wasted decades and billions in basic research, can you imagine where medical/disease research would be today? Makes my head hurt just to contemplate it!

  • BiophysicsMonkey says:

    Although DM alluded to it, I think that not nearly enough attention has been paid to the role of universities in creating the current situation. During the doubling period, universities all over the country massively expanded their research programs. Often this took the form of constructing a shiny (and expensive) new research building and planning to pay for it by filling it with 50 PIs with 2-3 RO1s each.
    In short order this doubled the number of investigators competing for grants (not to mention exacerbating the overproduction of Ph.D.s, since these new labs needed graduate students).
    If the NIH announced a new budget doubling tomorrow, universities would be planning new expansions & construction by next week, and within a few years we'd be exactly where we are now.
    Focusing on just the NIH and the individual PIs misses a major source of the problem. Unless universities change their policies, no amount of additional money will be enough.

  • PhysioProf says:

    What would you have the universities do when the NIH funding flows excessively like that? Not compete for it? That is delusional thinking.
    It's about as useful as abstinence only sex education. People are gonna fuck; it's their intrinsic nature. Universities are gonna compete fiercely for research funding; it's their intrinsic nature.
    BTW, love the handle! Maybe I gotta become PhysioMonkey!!

  • BiophysicsMonkey says:

    PhysioProf,
    I don't really have an answer. I suspect change can only come in the form of some kind of top down policymaking from the NIH, although I can't think of what such a policy would realistically be. Maybe just imposing stricter limits on how much total salary can be paid from grants would help a bit. My own institution just announced that it wants us to pay 75% of our own salary (up from 50%), and I hear that some schools are up near 100%. I that encourages an overly casual attitude towards expanding research programs. Why not hire dozens of new investigators if you assume that you won't have to pay them?
    Glad you like the handle.

  • whimple says:

    Universities are gonna compete fiercely for research funding; it's their intrinsic nature.
    Wrong. Universities compete fiercely for indirect costs, not research funding. How many new research buildings would have been built during the NIH doubling if only the money to be spent on actual research doubled? (and why didn't they actually implement the doubling this way?)

  • juniorprof says:

    I think NIH should standardize % pay and indirect costs. Right now its like the wild west, everyone negotiates their indirect cost % individually and the % pay they ask for their PIs is different. My personal opinion is that NIH should make an across the board rule... % indirect cost is set at 50% for everyone and tenure track PIs must be paid 50% by their institution. I'm probably being naive and this would cause some serious pain at some serious places at first but I think something should be done. I have no solution for soft money positions.

  • I_like_cows says:

    Seems like there is potential for Figure 1-2 to be over-interpreted. Shouldn't it be expressed as percent successful of total applications from the age group if you want to conclude there is age bias? That is, lets say that all of the applications from the under-35 group were funded but they only represented 2% of the total applications. In this scenario, there is no bias but you get the same graph. The implicit assumption seems to be that there has been no change over time in the percent of applications generated by each age group, which may not be true. The advancing age at which a degree is obtained and a lab set up to generate a grant might explain it rather than bias.

  • DrugMonkey says:

    I think that not nearly enough attention has been paid to the role of universities in creating the current situation.
    BiophysicsMonkey (nice 'nym!), Orac over at Respectful Insolence likes to bash the local uni's on this topic. His latest is here and I think it has links to his prior musings.
    BiophysicsMonkey: My own institution just announced that it wants us to pay 75% of our own salary (up from 50%), and I hear that some schools are up near 100%.
    juniorprof: My personal opinion is that NIH should make an across the board rule... % indirect cost is set at 50% for everyone and tenure track PIs must be paid 50% by their institution.
    100% = "soft money job", btw. There are plenty of these situations, from all-soft-money institutions to selected categories at traditional universities and med schools.
    It is interesting that institutions are upping the "soft" percentage. Decreasing the percent of salary on the NIH dime has been a whisper theme of the NIH self-study navel inspecting over the past year and made it into the Draft Report. There is a LOT of weasel language because they likely do not wish to kill their golden goose. Those of you (us?) berating the soft-money system need to keep a simple fact firmly in mind. The institutions do not have some magical pot of leprechaun gold waiting to start paying investigator salaries!.
    So suppose juniorprof's suggestion (which is similar to language that made it into the Draft Report I'll note) was put into place. First, a whole buncha soft-money investigators who are very highly accomplished would be departing their now-defunct research institutions and looking for university jobs. There would next be at least a 3-5 year period in which essentially zero new people where brought into the system. A whole bunch of people already in hardmoney jobs would find tenure prospects disappearing as these mid- to senior hires descended upon universities.
    And this would exacerbate the next little fun situation which would be caused by any decrease in overhead rates. Namely that institutions fully know that $$ is fungible and would upgrade their cost recovery practices. "Cost recovery" being the subterranean way they suck even more money out of the NIH grant. Absurd increases in support service fees and policies to ensure that you are forced to use those internal support services. Ever rant about your animal per diems? Wonder why your graphics department charges 3X the commercial rate? Why sequencing and other Core type services are higher than the open market?
    This, my friends, is a little "extra" overhead charged to NIH grants. Being forced to support more investigator salaries would simply motivate universities to increase these practices.

  • #18, #19, #20, Bingo, Bingo, Bingo as to the potential moves on salaries and indirects.
    Of course, universities aren't going to like anything in that direction. And if they have to move in that direction, I also have a hunch that a lot of the faculty are not going to like it as the univs are probably going to require them to spend more time in teaching, univ administration matters etc i.e. want bang proportional to their buck.

  • DrugMonkey says:

    That is, lets say that all of the applications from the under-35 group were funded but they only represented 2% of the total applications. In this scenario, there is no bias but you get the same graph.
    Actually your scenario is indeed correct and part of the point. Especially when considering the age of employment / funding numbers of the second graph.
    What I am saying with this post is that one factor that contributes to this overall picture is the extra hurdle placed in front of the New Investigator grant application.
    The willingness of departments to hire younger / less experienced people is also contributing to the problem and results from some of the same in-group tendencies. But this differs in many particulars from the application review issue I discuss here.

  • I actually think that limiting overhead could do a lot for the funding problem. Snooty U charges over 50%, while under 20% is more typical of state schools, from my (limited) research. That's several grants worth of money going to subsidize the building program and expensive dinners for the trustees.

  • (But of course the amount of funding is irrelevant to me as compared to the funding rate of continued vs. new applications. I'm just saying it sucks up a large amount of budget. And our building STILL leaks.)

  • S. Rivlin says:

    Physioprof,
    Universities fuck, too, whether they like it or not 😉
    In my university, during the boom of the NIH funding (1998-2004), they built two huge research buildings and hired a new dean for the medical school. The dean's salary was $450,000 + bonus of 3% of any overhead amount charged on extramural grants funded of up to a total of $9,000,000 ($300,000 bonus). The dean received that bonus every year until he was forced out (2005) for not working well with the president. PIs receive no bonus.

Leave a Reply