The NIH Grant system already funds "people, not projects"

Sep 30 2011 Published by under Grant Review, NIH, NIH Careerism

First PhysioProf baited me with this ridiculous commentary in Nature from John P. A. Ioannidis. Then Genomic Repairman took a cast and then PiT threw a big box of chumsicle into the water.

What could I do, DearReader? What could I possible do but chime in?

Before I get into this, a minor caveat to my remarks. I do like the fact that Ioannidis proposed a number of different fixes and suggested a pilot-study type of approach. I've made similar proposals for trying new ideas on this very blog. However. Proposals for change should be based on a clear and honest recognition of the present reality.

Ioannidis:

Leading thinkers and experimenters worthy of unconditional support could be identified through peer assessment of their work and credentials. Appraisals of project-based proposals already take a scientist's merit into account, but they typically give less weight to it than to the project plan. Peer assessment does not work well for early-career scientists, who have a short track record. But for those more established in their field, a career trajectory offers a wealth of information. By contrast, an isolated project is only a snapshot.

This is a not uncommon suggestion. Our old friend Noonan was proposing this just a little while back.

It is based in large part upon this false belief that the NIH system actually works as formally designed. It does not. Or at least, not so far as it is supposed to be a strictly project-based (i.e., proposal-based) system of grant funding. Ioannidis deploys "typically" in a context that makes it clear that he equates this with "nearly exclusively". This is total nonsense.

Admittedly, I had an experience in the very first months of my independent career that emphasized the person-based nature of the NIH funding system. In spades, with emphasis and I will admit quite shockingly. I mean, I had grasped the substantial table-slanting toward the established investigator already as a late-postdoc. But this was a whole 'nother bit of whammo.

So in my case I have been aware of this for some time. I didn't understand how it really worked, insidiously and no doubt unconsciously for the most part, until my first study section meeting. That's all it took.

I have related my anecdotes regarding this before. The first time, I think my jaw was literally hanging open in disbelief. It happened in my presence many times after and I have heard similar anecdotes from friends on other study sections. I have no reason to think that "our" end of the NIH world is so startlingly unique. It is encapsulated by the review of a grant, typically a competing continuation but sometimes a new proposal, from a very well respected and established scientist. The reviewer(s) get to the end of a rambling critique in which it is made emphatically clear that the proposal is jam-packed with stuff, confusing, devoid of logical design, consideration of alternatives, hypothesis testing and generally full of StockCritique Bait common to that particular section. In short, a crappy proposal that would be kicked unceremoniously to the curb, were it anyone less legendary. Then the reviewer finishes by saying "But I know Professor Grey Fox's lab is going to knock our socks off with great stuff because she has such a fantastic track record of unbelievable contributions. Post-discussion score, 1".

The perception that the NIH system is in part a Person- or Program-based funding system is reinforced by experiences with the Programmatic pickup behavior. When times started getting grim I had personal conversations with POs in which they stated nakedly that keeping "their established investigators" in grant funding was a high priority...they were deaf to my observations that new investigators who were not able to launch were also a high priority and a better long term investment.

When you are are considering a competing review, on study section, you get to see the Summary Statement and scores for the original proposal. Sometimes, that original Summary Statement makes it clear that the proposal sucked, was lucky to get a marginal score in review and Program picked up that dog turd anyway! Then, to add insult to injury, sometimes the current competing app is just as bad as the original one, the critiques you make are similar to many criticisms made by that earlier group of reviewers...and you later find out that Program picked up this dog turd too!!!

GAaaahhhh!

Unless it is ME, of course. Dude, I totally have NIH proposals under review these days and I am no longer a NoobProf. I mean, what am I saying here? Nevermind that stuff. If you are a reviewer out there looking over my proposals and notice any StockCritique Bait, feel free to ignore any deficiencies in my proposal writing. In favor of the fact that you know me and I've managed to publish a paper or two. And for the most part been "productive" on my prior awards. C'mon now. I need some of that People-based funding love! (Oh, and for any of my POs that are reading, I was KIDDING about that dog turd stuff- friends?)

Returning the NoobProfs who are now screaming in dismay back to earth, let me note that despite this fact, the Project-based part of the NIH system does work too. Good proposals from less known investigators get funded all the time. Just go to RePORTER and look up some Noobs in your subfield. Some of them get funded. From my study section experiences it became clear that in many of these cases the Noob got funded by proposing something excellent, not just because she happened to have postdoc'ed with ol' Horace Grizzler or Grey Fox. Great proposals from young (and not so young) investigators who are not tied cosily into the system get funded.

So when you hear a guy like Ioannidis implying that the system needs novel introduction of "person-based" grant funding, realize that we already have a balanced system. It is not exclusively person-based, nor is it exclusively proposal-based. There is a mixture.

Naturally, people's assessment of the current "balance" is greatly influenced by their perceptions (and misperceptions*) of their own status within the system and their predictions about how to make it easier for them to get the grant money that they deserve. I have a pretty short fuse for such nakedly self-serving myopia.

__
*How do they know they would be the one judged to have a track record that merits some sort of BSD-based funding?

34 responses so far

  • Its perfectly natural for people to push for the advantage that is theirs but to try to thinly veil it as the perfect system, now that's just stupid.

    And not to go all Seinfeld but, "Hello #Noon4n! "

  • becca says:

    It seems to me the easiest critique of project based systems is "there's a large amount of waste in writing grants"
    And the easiest critique of person based systems is "there's the risk of wasted grant dollars in the form of stupid projects".

    When paylines are very low, people write more grants to make up for it. And when paylines are low and universities use soft money funding schemes, good people get kicked out of the system.

    Therefore, as paylines get lower and unis rely more on soft money to fund people, you would expect person-based funding to seem more appealing. Since the waste under project based system goes up, and it's harder to speculate that the waste under person based systems would be as bad.

  • drugmonkey says:

    Actually, becca, I think the worst risk of person-based systems is the blocking of people with great ideas but no track record or the wrong academic pedigree. Preventing them from ever getting into the system in the first place. Or making them languish under the "credit" of the BigCheez's umbrella (see France, Germany).

    This is *much* worse than the notion of deadwood wastage. (Personally I'm not worried if the grant goes to a good person and they risk it on something that doesn't work out. That is a thing to support in the type of criticism levied by Ioannidis. )

  • Bashir says:

    I'm a newb, so my only real experience is through F and K awards. Those are explicitly about funding people as well as projects. In the sense that the environment, mentors and CV of applicant clearly count. A significant portion of my critiques were not about the project, but the person (lack of CV length, mentor critiques).

  • postdoc says:

    "Having it both ways"--trying to fund both people and projects, whichever seems more deserving of merit--means researchers with pretty good ideas but humble pedigrees will lose to researchers with weaker ideas and better pedigrees. You can substitute "glamour-mag pub" for "pedigree," if you want. It's the rich-get-richer effect, and it's not meritocratic or conducive to good science or sanity. The basic problem is that our assessments of almost anyone else's productivity and expertise are exceedingly superficial unless we're in the same subfield.

    I say this as someone who has probably benefited considerably from glamorous institutions/mentors and personal fellowships.

  • drugmonkey says:

    researchers with pretty good ideas but humble pedigrees will lose to researchers with weaker ideas and better pedigrees. You can substitute "glamour-mag pub" for "pedigree," if you want. It's the rich-get-richer effect, and it's not meritocratic or conducive to good science or sanity.

    Yes but am I making my point that this is not absolutes? All we are debating is the relative proportions and shading within a system that is flexible. Some grants that are good ideas from relative unknowns get funded. Some grant proposals (a lot, actually) from pretty well established folks in respected institutions go unfunded.

    The whinging from the old guard is IMNSHO just because now, finally, after a good long run of easy funding they are having to slug it out with revisions and triaged applications just like the rest of the mortals have had to do for a longer interval. Now that they have to struggle, all of a sudden the sky is frigging falling.

    Where the hell were they 10 years ago when n00b faculty were taking it in the chops, relatively speaking, even when the doubling was going on?

  • Grumble says:

    "Actually, becca, I think the worst risk of person-based systems is the blocking of people with great ideas but no track record or the wrong academic pedigree. Preventing them from ever getting into the system in the first place."

    But it doesn't have to be this way. As I've suggested before, there could be two parallel systems, one for Old Farts and one for Young Farts. As you point out, the system already works that way informally. What's the disadvantage to formalizing it? That way, NIH would presumably have more influence over how each kind of review is conducted. They can set more exact criteria for each kind of grant. And they could allocate the total amount of money for each category in some (presumably) rational way: for instance, if they think there need to be more newer investigators, they could allocate more to that program at the expense of the Old Fart program.

    Would an official two-track system be such a bad thing?

  • DrugMonkey says:

    What do you think the ESI policies do, Grumble?

  • whimple says:

    The problem is the current system doesn't explicitly spell out how many grants are pre-destined to be funded because of who submitted them. So, if the payline is 10%, but if 8% of that has been pre-allocated, that's information worth knowing.

  • drugmonkey says:

    You know it is more fluid than that, whimple

  • One source of such information is the relative success rates for competing renewals and new applications. The former is vastly higher than the latter. And anyone who has served on study section more than a few times (or looked at the relevant data that Jeremy Berg has posted on the NIGMS Feedback Loop) knows this is not solely based on program pickups. Competing renewals get much better scores in study section. (Some of that may be due to the fact that some PIs who were horribly productive in the prior project period probably decline to submit one.)

  • drugmonkey says:

    I suspect your parenthetical selection bias contributes a great deal, though. PP.

  • Grumble says:

    DM, yes, the ESI policies help, but the issue is this: why should more senior scientists with lots of name recognition and very productive labs have to (constantly) submit grants? If they are going to get auto-funded anyway no matter what kind of crapola they submit, why continue with the charade or requiring them to submit? Why not just have them submit a biosketch and a 1 page summary of what they've accomplished and another page on what they plan to do? (Again, this would be only one way to get money from NIH - the other would be the traditional system, in which younger folk would compete more or less among themselves rather than with big shots, who would get funded the other way.)

    This wouldn't directly help me right now - but I would love to be able to look forward to a future when I don't have to be writing grants 90% of the time.

  • drugmonkey says:

    If I were one of the perma-funded ones, Grumble, that would be a great system!!!

    (have you really thought about how you'd like it if you were not one of the few selected?)

  • Joat-mon says:

    Grumble, that sounds like a terrible idea. For those who didn't get picked are going to be very bitter. Maybe you should consider getting a job at the NIH so that you don't have to compete for extramural money, no grant-writing.

  • CD0 says:

    Certain PIs and certain institutions already get way too much credit in study sections. And much more in peer-review in certain journals, which also helps to get grants later.

    If you are truly outstanding, write an outstanding grant. Period.

    If you are empty or tired and you are not able to elaborate a good plan for the next years of your career, you do not deserve further support. Make room for somebody else and do not bother.

    (at this point I was tempted to add an insult)

  • qaz says:

    One of the important results of person-based funding that gets lost is that it would be very hard to justify person-based funding at the many-R01 level. That is, person-based funding is likely to be "everyone at this level gets $X." One of the major (dis)advantages of project-based funding is that super-big-lab proposes super-cool project and the project can get funded, even if that lab already has lots of other funding. Person-based funding probably means the end of big empire labs. I'm not sure this is a good thing. The question is how much do 5x-R01-equivalent empire labs outcompete small 1x-R01 operations?

    The key, I think, is to have two tracks. (This is what I proposed before in an earlier discussion.) You need one track for person-based funding that is small (say one full R01) but steady. You would need a full grant proposal to get into the system, but once you get into the system, renewal would be just a list of funded papers and a short progress report. (*) If you ever miss your renewal, you fall back out of the system and have to do a full proposal to get into the system. The other track would be project based. The second track could even be similar to the system we've got. This would allow project and person tracks to co-exist.

    * I know this is how NIH renewals are supposed to work, but that has not been how it has been working on the study sections I've been on for the last 5 years. Maybe it used to be that a good person could just put cr*p on the page and get renewed, but on the study sections I've been on recently, renewals do get a bonus, but they still have to propose a good project or they simply don't make the cut.

    I think the REAL problem with the whole system is that NIH is pretending that their goal is to "just fund the best science", instead of treating science as an investment portfolio. NIH needs to be more explicit about their investment portfolio. What percentage of the funds go to senior people? What percentage of the funds go to junior? What percentage of the funds go to first R01s? What percentage to second and higher? NIH puts in these little flags (like ESI) but explicitly tasks study section with judging science. Study section is never allowed to say "funding". (As in "this grant should not be funded" or "this grant should be funded" or "this grant is on the edge" -- yes, I know that this is often how study section thinks in practice, but we're not supposed to consider funding. Budget is not considered until after scoring is done. So, if I see a super-expensive grant from a well-funded researcher, I'm not supposed to take that into account when I score the grant.)

    The worst thing one can do when playing the stock market or investing is to chase the next big thing. A diverse portfolio is the key to good investment. NIH needs to be explicit about their investment portfolio.

    PS. "Where the hell were they 10 years ago when n00b faculty were taking it in the chops, relatively speaking, even when the doubling was going on?" AMEN, BROTHER! ... Or the last 10 years when NIH instituted all this help for the young'uns and the mid-range faculty (first renewal/second R01) got no help?

  • drugmonkey says:

    qaz-
    *no, formally speaking, renewals are not supposed to be *only* a list of papers

  • drugmonkey says:

    A diverse portfolio is the key to good investment. NIH needs to be explicit about their investment portfolio

    Agree with the investment portfolio approach...and I thing NIH as a whole does as well. But to be "explicit" about percentages? That would be as stupid as always chasing the next big hit like a Vegas loser. What the *could* be better about is *tracking* their investment percentages ao that it is not a surprise to find out they have become light on some critical sector. Like the alleged ignorance they had about Early Stage Investigators.

  • whimple says:

    There's nothing wrong with being explicit about percentages when you're carrying a sufficiently large portfolio. Mutual funds do this all the time (%large cap, %bond, %international etc).

  • qaz says:

    "no, formally speaking, renewals are not supposed to be *only* a list of papers" I know. And I follow the rules and don't score them that way. But I think they *should* be (or at least it should be for one grant per PI).

    Also, at least in my (limited) experience at study sections over the last 5 years or so, I've never seen a renewal given a free pass. At best, being a renewal with good progress gets you a few percentage point bonus (about -0.5 on the old system, the new system is too quantized to see effects without a lot of averaging, but the discussion hasn't changed much). (Which is really no more than having had good training, a good pedigree, being a new PI, or any of a number of little "bonuses" that seem to help.)

    Given the current rules - that it's supposed to be about the project, this seems right to me. But this is certainly not the stories I was told by my more senior colleagues, about "the good old days" where bunny hopping study sections let bunny hopping renewals through, and certainly not evidence that renewals are easy. I would like to see a track for one renewal per PI to be easy. What I'd like to see is a mechanism by which a PI can trust a base scale to ensure tech and critical staff continuity.

    And in terms of the portfolio, any major investment fund definitely reports percentages. If you look at your mutual funds (your retirement portfolio, for example), each of those funds has an explicit target percentage that they hit so that you can decide how risky you want to be. Given the scale of NIH, it would make sense to have an explicit percentage of each class of PI that they have. They already do this by institute, why wouldn't they do it by risk?

  • drugmonkey says:

    Because they are not beholden to you PIs any more than mutual funds are beholden to the companies in which they invest.

  • qaz says:

    If NIH is an investment portfolio, the customers/stock-holders are the people of the United States and the companies they invest in are the PIs. So just as mutual funds use explicit percentages across different riskinesses of companies so as to get the best return on their investment for their stock-holders, the NIH should use explicit percentages across different riskinesses of scientists so as to get the best scientific production for the US citizenry.

  • drugmonkey says:

    In money management, the score card is obvious and singular. Moar $$ = better. this can be calculated on a quarterly basis quite easily. Would you care to try to define "success" for the NIH investors?

  • drugmonkey says:

    it would make sense to have an explicit percentage of each class of PI that they have.

    This is interesting because in many industries, just not hiring new folks and allowing for attrition of the old folks is considered the less painful approach to downsizing. You appear to be suggesting making the pain of a down cycle hit all worker classes equivalently, yes? And not only that but perhaps across productivity levels as well? After all, the small town grocer segment is just as important to a diverse portfolio as is the slave driver or mad genius..

  • At best, being a renewal with good progress gets you a few percentage point bonus (about -0.5 on the old system, the new system is too quantized to see effects without a lot of averaging, but the discussion hasn't changed much). (Which is really no more than having had good training, a good pedigree, being a new PI, or any of a number of little "bonuses" that seem to help.)

    This is not my experience at all serving on three different chartered study sections since the change to the new scoring system, where competing renewals seem to be getting a full impact score bonus for productivity over the last competing project period. Obviously, this makes a *massive* difference in percentile and likelihood of funding (as can be seen from the data that Jeremy Berg has posted at the NIGMS Feedback Loop): 20 = near certain funding, 30 = very good chance of funding, 40 = no fucken way.

  • drugmonkey says:

    So does that mean the discussion centers on whether a sufficient amount of scientific progress has been made, PP?

  • whimple says:

    Would you care to try to define "success" for the NIH investors?

    Not having done this adequately is a key failing of the NIH today. When Collins tries to argue his case for additional NIH budget support, his argument is hurt very badly by not having clear metrics on what the NIH has delivered to the American taxpayer and instead having to rely on "feel-good" philosophy.

    I'd be happy to try to define "success" for the NIH investors if I were adequately compensated for my efforts along these lines.

  • Grumble says:

    Whimple, you raise an excellent point. A while back I submitted an opinion piece to my local newspaper advocating for more money for NIH. I wanted to point out some of the great health care advances that have been a direct result of NIH funding. I figured I'd be able to go to the NIH website for a quick list of its best accomplishments.

    To my surprise, I couldn't find a single page on the NIH that simply lists the most impressive advances NIH has funded in the last 50 years or so. Maybe I missed it and someone can point it out?? It certainly can't be found on the home page: all I come up with is some babble about "mission" that says nothing about actual accomplishments. NSF, by the way, has the same problem.

    This absence of self-promotion not only reflects a likely absence of success metrics: it also reflects an absence of urgency that the public understand why it is paying for NIH and NSF. If even people who want to advocate for science can't find a concise list of NIH's and NSF's accomplishments, it becomes that much easier for the right wing cut-government whackos to convince people that public funding of science is a waste.

  • So does that mean the discussion centers on whether a sufficient amount of scientific progress has been made, PP?

    It "means" exactly what I wrote. Are you asking whether in my experience of panel review of competing renewals there is discussion of scientific progress in the prior project period, DM?

  • drugmonkey says:

    No, I was wondering if in your experience this was the main part of the discussion for competing continuations.

  • It is a significant part of the discussion. The main part of the discussion is the proposed new research. But this discussion is less hypercritical than that of new applications, and receives more benefit of the doubt.

  • Eli Rabett says:

    NSF accepts accomplishment based renewals
    ----------------------------
    PIs are advised that the ABR is a special type of renewal proposal appropriate only for an investigator who has made significant contributions, over a number of years, in the area of research addressed by the proposal. Investigators are strongly urged to contact the cognizant Program Officer prior to developing a proposal using this format.

  • [...] this isn't strictly accurate. I have heard people try to reign in a comment about "wonderful productivity under this award" with a close analysis of the listed publications. To point out where a publication appearing in [...]

Leave a Reply