Predicting the future

Jun 24 2015 Published by under NIH, NIH Careerism

One of the biggest whoppers told by Ronald Germain in his manifesto on fixing the NIH is this:

it is widely accepted that past performance, not a detailed research plan, is the best predictor of future success. So why stay with the fiction that R01 grant proposals are the best method for determining support of the individual scientist,

As I often say there is nothing so a-scientific and illogical as a scientist on the business of science.

The way he states this plays the usual sleight of hand with the all-important, unmentioned variable.

Namely, the means to do the research. Grant funding.

There is no bigger predictor for the success of a given research plan submitted to the NIH than whether or not the PI receives the funding to do the work. Funnily enough, Germain actually recognizes this and totally undercuts his argument in one of his caveats:

with 5–7 years of support per round and 1–2 years of bridge funding available, I think it is unlikely that a highly competent investigator will fail to produce enough during 6–9 years of research to warrant a “passing grade” without further extensions, except in extenuating circumstances.

Right? He sees right here that all that matters is funding. Most competent investigators will succeed if they but have the funding! Which makes his idea that this will cut down on competition and the "stochastic" nature of getting the grant funding look as silly as it is.

It's just another way to say "We'll pick our favored winners in advance of any independent accomplishment based on who they trained with (i.e., us!) They will keep right on winning because they will be the only ones with the means to accomplish anything. All others can stay the heck away from our effortless stream of moola, no matter how good their ideas might be".

This is important and it is why basing funding on accomplishment, rather than great ideas and the capacity to fulfill them is recipe for a death spiral of the Extramural NIH productivity as a whole.

This plan will self-reinforce and harden a silo around a limited set of brains, doing science in the way they see fit. Good ideas from outside this silo will not be given a chance to compete....unless they happen to occur to someone inside the silo. And on the whole, that person will not represent a diversity of ideas, approaches and interests. This will, across the enterprise of NIH-funded science, reduce the rate of discovery.

Those who manage to accomplish will continue to have a stranglehold on the means to accomplish. Means leads to accomplishment leads to more means in the Germain scheme.

So what gets accomplished will be narrowed, iteratively, with each 5-7 year review. Only to be refreshed, minimally, with each squeezed down cohort of new hires who manage to make it into his starter, block-grant scenario. Those, of course, will be selected by Universities on the basis of seeming like the people who are already most successful since the review will be anticipated to be on the basis of the person. Naturally, the trainees of the insider club will be most highly sought after. (Take a look at the way HHMIers, espcially the Early Career ones have been trained folks. ...talk about the past predicting the future and all, right?)

So when you hear someone talking about "the best predictor of future performance is past performance", make sure to ask whether that is with or without the funding and how they know this.

The second truthy whopper Germain tells follows soon after.

true creativity is often cause for lower scores?

Personally I have yet to see a well-prepared truly creative grant get killed just for being creative and new. Maybe wackaloon geniuses who have great ideas but simply refuse to write an actual grant proposal struggle in some sections. I guess. But here's a secret for Germain. (A "secret" known to just about anyone who has served on 2-3 traditional standing study sections.) People that he is talking about, those who have demonstrated a high level of accomplishment in the past 5-7 years, get away with utterly crappy proposals and still get their funding based on their record of accomplishment.

That's right. We ALREADY have a system in place that HUGELY benefits and prioritizes the funding of people with a track record of accomplishment. The "creativity" in their proposal does not prevent them from getting funded. Nor, btw, does diverging substantially from the plan they got the money for hurt them in the next round of evaluation.

Given this, there is no conceivable way that switching to Germain's plan changes the ability to be creative.

Now, for those outsiders or people with a brand new idea absent a track record....yeah, they may take it on the chin under the current NIH system. But they would ALSO fail to gain support under Germain's. It isn't like we're inventing up some new peers to do the reviewing here. No matter if it were McKnight's panel of NAS members or Germain's ideas of the deserving elite or traditional NIH-style panels judging the "track record"....there is no way that we can assume that genius PI behind every PCR or gene knockout technology or whatever Nobel-worthy breakthrough will be immediately recognized as awesome and funded.

29 responses so far

  • boehninglab says:

    My favorite part of his manifesto is when he declares the brilliance of his idea:

    "I have discussed this specific plan with various HHMI investigators, tenured and non-tenured faculty at diverse institutions, postdoctoral fellows, and students around the U.S. The responses range from substantial interest to enormous enthusiasm, most at the latter end of the scale."

  • drugmonkey says:

    boehning lab- he repeated something like this in the video as well. I don't know if he is lying, Fox News style, or is really this out of touch. I suspect, however, since his son is in touch with the Twitter and blog sentiments familar to the both of us that he's lying. The son, at least, knows full well that there is a substantial and substantive opposition to this plan.

    I also wonder how honest his discussions have been. IMO, it is only a truthful discussion if he looks each of them straight in the eye and says "Assume you won't be / wouldn't have been one of the selected ones. Now, here's my plan".

  • I wonder to what extent Germain's rose-colored glasses result from no one being willing to tell him "Are you out of your f-cking mind?"–except for those in positions that would make them inclined to agree with him.

    Is someone on the tenure-track or a post-doc really going to insult his precious snowflake of an idea, or are they going to feign "substantial interest? ("That's very interesting*")

    *[...you dumbass].

  • drugmonkey says:

    Presumably his son, of all people, would have at least said that there is a substantial number of people on social media that think this idea is bullshit. I know for a certain fact he, the son, heard it from at least one person long prior to this manifesto being published.

    I suspect it is more down to him, the dad, refusing to hear than it is down to him not being told.

  • A Salty Scientist says:

    For us youngins, were scientists clamoring for "people, not projects" when success rates were above 30%?

  • Grumble says:

    " We ALREADY have a system in place that HUGELY benefits and prioritizes the funding of people with a track record of accomplishment."

    Yes. And those people who benefit in this way have to waste an enormous amount of time writing grants, because success has become, as Germain correctly points out, stochastic. So why not do away with the charade entirely and just fund some subset of these people (who are going to get funded anyway) based on their records, rather than forcing them to waste so much of their time on grants?

    "People that he is talking about, those who have demonstrated a high level of accomplishment in the past 5-7 years, get away with utterly crappy proposals and still get their funding based on their record of accomplishment."

    That hasn't been the case in my experience. I see the knives come out for grants from everyone. The younger generation, which has grown up knowing nothing but tight funding, has no patience with crap grants from BSDs. Where established investigators have an edge is in reviewers' attitudes that "So-and-so has published extensively on it so I trust her to do it right even though I have some minor concerns." But, as Germain again points out, this doesn't contribute to creativity and boundary-crossing, but to repetitive, "safe" experiments.

  • drugmonkey says:

    The reason, Grumble, is that there is a chance. Not the same chance, but a fighting chance. A chance for someone who is not in the insider club, heck maybe not even with any record of accomplishment to speak of, to win funding based on a panel of peers thinking that gee, this is a good thing to do. And sure, maybe someone else would be better but this guy is proposing it and there is a smidge of evidence he can pull it off so what the hell.

    So-and-so has published extensively on it so I trust her to do it right even though I have some minor concerns.

    yeah and many other people on the panel are saying to themselves if not to the panel "minor? more like moderate to major if this had been the last noob we reviewed".

  • Pinko Punko says:

    If pay lines stay this way, they will simply get the system they propose. There will be no one left. The rich will survive on the basis of more resources and greater funding. The in betweens will wither on the vine or slowly be strangled (getting along on 1 R01 that keeps being cut, or goes into gaps), and then of course not appearing to compete at the same level, and then getting dinged for it- though never ever funded at the same level as those already grandfathered in to the productivity train. Subsequently, this cohort will be replaced by the next round of ESIs perhaps baby MIRAs etc. The mid career investigators will be gone.

  • CD0 says:

    It is ironic that somebody who does not compete in the extramural system (and probably never did so; and probably never served in a study section) declares himself an expert on how to fix it.
    Firstly, I challenge the notion that the best predictor of future performance is past performance. I have met too many individuals who made great contributions in the past and completely out of touch with the latest advances. In my experience, this is actually the rule.
    Secondly, the system in many places in Europe and elsewhere prioritize people over ideas. That has not been the philosophy of the NIH since its inception, and we generated the most productive system in History. The goal for decades has been to keep active, productive scientists funded, and not old wood, as I have seen too many times in Europe.
    In fact I think that the investigator category sometimes receives too much weight in peer-review. The old view that “you are as good as your next grant”, which kept us successful and creative, does not apply at the NIH, where this individual has developed his career. And I believe that most scientists would agree in that, on a per dollar basis 9and likely with any criteria), the best science is produced outside the intramural program.

  • Grumble says:

    "A chance for someone who is not in the insider club, heck maybe not even with any record of accomplishment to speak of, to win funding based on a panel of peers thinking that gee, this is a good thing to do."

    As CPP would say, you are "fucken deluded." No record? No accomplishment? Tough shit. The chances of getting funded to do something you have no experience with is so vanishingly small that it is laughable. I find it hard to believe that you are serious with your attempts to defend the current system with it.

    "yeah and many other people on the panel are saying to themselves if not to the panel "minor? more like moderate to major if this had been the last noob we reviewed"."

    Right. But if the PI has already published 17 versions of the experiment successfully, is it really wrong to give her the benefit of the doubt?

    "Secondly, the system in many places in Europe and elsewhere prioritize people over ideas. That has not been the philosophy of the NIH since its inception, and we generated the most productive system in History."

    The NIH's success relative to the European system's is partly due to its philosophy of "funding ideas, not people" -- but only partly. The other part is that the NIH spends enormously more money than the Europeans do.

    I don't think the European model is a good one for the NIH to follow. But I also think that the current NIH model is limiting productivity in times when funding is scarce. People are simply wasting too much time writing grants. One way to solve that problem is to set up a hybrid system, whereby established PIs get some low but consistent level of funding based only on recent past accomplishments, and newcomers are able to compete amongst themselves (and/or amongst everyone) with "idea-based" proposals.

  • physioprof says:

    Self-interested nepotistic shittebagges like Germain make these claims because they are delusional about how scientific progress works, and think it depends on their individual magical wagical snowflake "genius", as opposed to funders throwing money at particular areas of inquiry and ensuring that reasonable numbers of reasonably competent scientists work on the shitte. If "geniuses" like Watson & Crick didn't figure out the double helix when they did, does anyone really doubt that some other schlubs would have within a short period of time anyway? Crispr/Cas would still be unknown if Doudna had no funds to study bacterial immune systems, but some other schlubs did?

  • Alfred Wallace says:

    BTW Europe =/= Europe

    There are actually many countries with many different systems of how to fund science.
    Some good, some probably worse. A hybrid system (base funding incl. salary by university, additional money can be obtained by competitive funding from different bodies) is for example in place in Switzerland, which is in its research output per capita at least up there with the US.

  • Pinko Punko says:

    Grumble, the current "system" is not the historical system. There is not enough money or there are too many PIs. Either way. If a BSD occasionally gets hammered for proposing a turd, that is legitimate.

  • DJMH says:

    The intramural system works more or less in this fashion that he's proposing. And, as noted above, with very few exceptions the intramural system has not produced rock stars or startling, exciting science. This is despite the fact that intramural absorbs a full 11% of the NIH budget.

    Is there really more that needs to be said?

  • drugmonkey says:

    Intramural labs that work in areas I follow certainly do not do research that couldn't be done extramurally. And I can think of a few programs that would never have lasted so long in the extramural environment.

  • zb says:

    I agree that the system Germain describes is similar to the intramural system.

    "Intramural labs that work in areas I follow certainly do not do research that couldn't be done extramurally."

    That's not an indictment of the intramural system; if the same research *could* be done intramurally, then why shouldn't it? Especially true if the feds are funding the full cost of the research (that is, not leveraging state funds, private funds, other externalities, like an individual contributing 50% of their time to research, but also teaching, allowing, in theory at least, to have 2x as many minds working on a problem, if for a lower amount of time).

    "I can think of a few programs that would never have lasted so long in the extramural environment."

    This is a necessary component of a more stable system -- that programs that are not at the cutting edge are going to last longer than they would in one one where each program lives or dies on a 5 year grant cycle.

    "And, as noted above, with very few exceptions the intramural system has not produced rock stars or startling, exciting science."

    This is an indictment of the intramural system, but I'd like to hear on what grounds "rock star" science is being judged. The intramural labs in my field have been major players, and were so during the start-up/expansion phase of the work (and, potentially, uniquely able to play that role because of the ability of PIs to take higher risks with their funds).

  • qaz says:

    Are rock stars the goal? The vast majority of important science is small, pedestrian steps, one step in front of the other. Wouldn't it be a good thing if there were a place for good, basic, step-by-step science to be done?

    Besides, which I don't think that it's true that there aren't superstars in intramural NIH. Certainly in the fields that I play in, there are historical NIH intramural people who were powerhouses in their day (some of whom are retiring and some of whom moved extramural). And it is certainly true that in the fields that I play in, there are some amazing up-and-coming superstars (call them top-of-the-line-mid-career people).

  • DJMH says:

    I'm sure it depends on your field. But look at it this way: the intramural folks
    (A) don't have to do (much) grant-writing,
    (B) don't have to teach, and
    (C) don't have a ton of service (thesis committees, curriculum committees, etc.)

    Shouldn't these folks be simply amazing? Shouldn't the bar be higher there? I'm not defining rock-star as constant C/N/S papers, but these folks have a huge leg up on those of us for whom A and/or B are massive parts of our days, and I think it should show in their productivity. Do you think it does?

  • zb says:

    It might be true that folks in the intramural program have a lower work burden but that only means they should produce more if they cost more (in a broadly defined sense of cost) or produce less. Otherwise, the program is just an alternative but equal model that might have some benefits (like career stability or better control of long term decision making).

    There are drawbacks to the intramural program, but arbitrarily demanding more from them is a false standard.

  • kalevala says:

    The intramural system is there so that the NIH has whitecoats in sciency environs to show to any pols or taxpayers who wish to see where all that money is going. It's like a Tesla showroom. The car is real, but also just a sample.

  • drugmonkey says:

    Are there a lot of machines that go "Ping!"?

  • CD0 says:

    The point is, do we want to replace a competitive system that forces scientists to remain productive and sharp for their entire career with an even more nepotistic version of the NIH intramural system?
    Do we want dead wood to accommodate at the top and block the access of younger and more promising individuals because 20 years earlier they made a good contribution to the field? Or because they trained in a good lab?
    Of course there are awesome scientists at the NIH but, at least in my field, not the best.
    Start level for junior faculties at the NIH is the equivalent of 2 RO1s. Much more from there on.
    Just compare their advances with those in labs that have that level of funding.
    I think that we can do better.

  • poke says:

    DJMH said:

    (A) don't have to do (much) grant-writing,
    (B) don't have to teach, and
    (C) don't have a ton of service (thesis committees, curriculum committees, etc.)

    Shouldn't these folks be simply amazing? Shouldn't the bar be higher there?

    Another possibility is that these factors you list, much bandied about in the extramural for decreasing productivity, actually don't have much effect. Maybe all the whinging "Oh, if only I taught one less class then I'd show them my true genius rockstar potential," is just a bunch of bullshit.

    Also, how certain are we that intramural is just a candyland of easy money, no teaching, no thesis committees, etc.? I could easily imagine that whatever time intra investigators might save not writing grants or teaching is more than outweighed by the administrative and bureaucratic nonsense endemic to the gvmnt.

  • […] has been praised, apparently by people I don’t know, and roundly criticized by my friends and colleagues on line. In my med school applications, I wrote that having been an East Asian Studies major, I had […]

  • DJMH says:

    Ok, but again, the question isn't "Are there competent scientists at the NIH?" but "Should we all move to effectively an intramural form of funding?" Based on what we can see of intramural output, especially given the lower external demands on intramural folks, my vote is no.

  • drugmonkey says:

    No record? No accomplishment? Tough shit. The chances of getting funded to do something you have no experience with is so vanishingly small that it is laughable. I find it hard to believe that you are serious with your attempts to defend the current system with it.

    My comments arise from experience. Experience that shows that it is possible. This is a feature that should be encouraged and celebrated, of course, because it is a key strength of the NIH extramural, peer-review selected system.

  • The Other Dave says:

    Glad to see that someone is tearing away at this incredibly dopey essay. My jaw dropped when I read it. The reasoning isn't even internally logical.

    But the worst thing about it is that this dork will list the Cell publication on his CV.

  • […] opposes Ronald Germain’s suggestions to fix the NIH (which The Conversation piece is based on), arguing […]

  • […] is laughably illogical when it comes to NIH grant awards. What really predicts future performance and scientific productivity is who manages to land the grant award. The money itself facilitates the productivity. And no, they […]

Leave a Reply