NIH Grant Lottery

Apr 18 2016 Published by under Fixing the NIH, NIH

Fang and Casadevall have a new piece up that advocates turning NIH grant selection into a modified lottery. There is a lot of the usual dissection of the flaws of the NIH grant funding system here, dressed up as "the Case for" but really they don't actually make any specific or compelling argument, beyond "it's broken, here's our RealSolution".

we instead suggest a two-stage system in which (i) meritorious applications are identified by peer review and (ii) funding decisions are made on the basis of a computer-generated lottery . The size of the meritorious pool could be adjusted according to the payline. For example, if the payline is 10%, then the size of the meritorious pool might be expected to include the top 20 to 30% of applications identified by peer review.

They envision eliminating the face to face discussion to arrive at the qualified pool of applications:

Critiques would be issued only for grants that are considered nonmeritorious, eliminating the need for face-to-face study section meetings to argue over rankings,

Whoa back up. Under current NIH review, critiques are not a result of the face-to-face meeting. This is not the "need" for meeting to discuss the applications. They are misguided in a very severe and fundamental way about this. Discussion serves, ideally, to calibrate individual review, to catch errors, to harmonize disparate opinions, to refine the scoring....but in the majority of cases the written critiques are not changed a whole lot by the process and the resume of the discussion is a minor outcome.

Still, this is a minor point of my concern with their argument.

Let us turn to the juxtaposition of

New investigators could compete in a separate lottery with a higher payline to ensure that a specific portion of funding is dedicated to this group or could be given increased representation in the regular lottery to improve their chances of funding.

with

we emphasize that the primary advantage of a modified lottery would be to make the system fairer by eliminating sources of bias. The proposed system should improve research workforce diversity, as any female or underrepresented minority applicant who submits a meritorious application will have an equal chance of being awarded funding.

Huh? If this lottery is going to magically eliminate bias against female or URM applicants, why is it going to fail to eliminate bias against new investigators? I smell a disingenuous appeal to fairness for the traditionally disadvantaged as a cynical ploy to get people on board with their lottery plan. The comment about new investigators shows that they know full well it will not actually address review bias.

Their plan uses a cutoff. 20%, 30%...something. No matter what that cutoff line is, reviewers will know something about where it lies. And they will review/score grants accordingly. Just Zerhouni noted that when news of special ESI paylines got around, study sections immediately started giving ESI applications even worse scores. If there is a bias today that pushes new investigator, woman or URM PI's applications outside of the funding, there will be a bias tomorrow that keeps them disproportionately outside of the Fang/Casadevall lottery pool.

There is a part of their plan that I am really unclear on and it is critical to the intended outcome.

Applications that are not chosen would become eligible for the next drawing in 4 months, but individual researchers would be permitted to enter only one application per drawing, which would reduce the need to revise currently meritorious applications that are not funded and free scientists to do more research instead of rewriting grant applications.

This sounds suspiciously similar to a plan that I advanced some time ago. This post from 2008 was mostly responding to the revision-queuing behavior of study sections.

So this brings me back to my usual proposal of which I am increasingly fond. The ICs should set a "desired" funding target consistent with their historical performance, say 24% of applications, for each Council round. When they do not have enough budget to cover this many applications in a given round, they should roll the applications that missed the cut into the next round. Then starting the next Council round they should apportion some fraction of their grant pickups to the applications from the prior rounds that were sufficiently meritorious from a historical perspective. Perhaps half roll-over and half from the current round of submissions. That way, there would still be some room for really outstanding -01 apps to shoulder their way into funding.

The great part is that essentially nothing would change. The A2 app that is funded is not going to result in scientific conduct that differs in any substantial way from the science that would have resulted from the A1/15%ile app being funded. New apps will not be any more disadvantaged by sharing the funding pie with prior rounds than they currently are facing revision-status-bias at the point of study section review.

What I am unclear on in the Fang/Casadevall proposal is the limit to one application "per drawing". Is this per council round per IC? Per study section per Council round per IC? NIH-wide? Would the PI be able to stack up potentially-meritorious apps that go unfunded so that the get considered in series across many successive rounds of lotteries?

These questions address their underlying assumption that a lottery is "fair". It boils down to the question of whether everyone is equally able to buy the same number of lottery tickets.

The authors also have to let in quite reasonable exceptions:

Furthermore, we note that program officers could still use selective pay mechanisms to fund individuals who consistently make the lottery but fail to receive funding or in the unlikely instance that important fields become underfunded due to the vagaries of luck.

So how is this any different from what we have now? Program Officers are already trusted to right the wrongs of the tyranny of peer review. Arguing for this lottery system implies that you think that PO flexibility on exception funding is either insufficient or part of the problem. So why let it back into the scheme?

Next, the authors stumble with a naked assertion

The proposed system would treat new and competing renewal applications in the same manner. Historically, competing applications have enjoyed higher success rates than new applications, for reasons including that these applications are from established investigators with a track record of productivity. However, we find no compelling reason to justify supporting established programs over new programs.

that is highly personal. I find many compelling reasons to justify supporting established programs. And many compelling reasons not to do so preferentially. And many compelling reasons to demand a higher standard, or to ban them entirely. I suspect many of the participants in the NIH system also favor one or the other of the different viewpoints on this issue. What I find to be unconvincing is nakedly asserting this "we find no compelling reason" as if there is not any reasonable discussion space on the issue. There most assuredly is.

Finally, the authors appeal to a historical example with is laughably bad for their argument:

we note that lotteries are already used by society to make difficult decisions. Historically, a lottery was used in the draft for service in the armed forces...If lotteries could be used to select those who served in Vietnam, they can certainly be used to choose proposals for funding.

As anyone who pays even the slightest attention realizes, the Vietnam era selective service lottery in the US was hugely biased and subverted by the better-off and more-powerful to keep their offspring safe. A higher burden was borne by the children of the lower classes, the unconnected and, as if we need to say it, ethnic minorities. Referring to this example may not be the best argument for your case, guys.

45 responses so far

  • physioprof says:

    Dude, looks like you've fucked up some of your blockquotes.

  • drugmonkey says:

    fixed.

  • DJMH says:

    DM, is it true that distribution of Vietnam soldiers was *more* skewed to lower classes / minorities than the current distributions of soldiers? You are of course correct that the Ws of the world found ways to avoid going over there, but I would think the skew might be even worse now, because the draft for all its weaknesses cast a much wider net than today's all-volunteer.

    But yeah the proposal seems silly. The US for all its funding faults consistently produces better science than virtually any other place in the world. So it's hard to see why radical change in evaluation would help.

  • drugmonkey says:

    DM, is it true that distribution of Vietnam soldiers was *more* skewed to lower classes / minorities than the current distributions of soldiers?

    what in the hell does the answer to this question have to do with whether or not a lottery will fix current bias in NIH grant review?

  • baltogirl says:

    I would certainly take a lottery in which chance favors me at least half of the time (ie top 20% of grants winnowed to 10%) over resubmitting the same (&()!(* grant three and four times.
    At one chance out of three, it is less appealing, but in fact, looking back over the many submissions of the last few years, I would likely now have more funding- unless, of course, I were unlucky.
    But I would rather be unlucky than be told that the same work that was really terrific last round is now considered garbage.

  • Philapodia says:

    I advocate for a Thunderdome funding model. Put all of the applicants in a big metal cage, covered with barbed wire, and grants are written on provided computers in one sitting. Hook a bomb to each wrist that is only activated if the words "novel" or "game-changing" are used. Last person standing leaves with a 5 year R01 + psychotherapy for life.

  • Ferric Fang says:

    Dear DM,

    Thank you for blogging about our article, even though many of your comments are critical. We welcome the discussion but would like to clarify a few points of disagreement:

    1. While it is true that critiques are written prior to the face-to-face meeting, virtually every study section on which we have participated has spent much of the face-to-face meeting attempting to prioritize the applications that are competitive for funding. If no ranking were required, and applications were simply given a binary rating of ‘meritorious’ or ‘non-meritorious’, we suspect that there would be little need for a meeting. Most discrepant evaluations could probably be resolved in a brief teleconference.

    2. It is true that a modified lottery cannot prevent reviewers from placing a larger proportion of grants originating from specific subsets of applicants into the ‘non-meritorious’ group. However a modified lottery can at least ensure that bias does not determine the funding decisions made among meritorious applications, and would thus represent a step forward in addressing bias. If underrepresentation of certain groups is found to persist despite the institution of a modified lottery, then further attention could be directed to examine the reasons for the excessive classification of applications from these groups as ’non-meritorious’. The alternative of placing all applications into a lottery would be unacceptable to most people because it would divert funding to a substantial proportion of applications with serious flaws and remove incentives for submitting quality applications.

    3. There is no need to assign a specific cut-off for grants to be entered in the lottery, simply a designation as ‘meritorious’ or ‘non-meritorious’, which we estimate would separate out 20-30% of applications from the remainder but the percentage might be higher.

    4. The problem that new investigators have in obtaining funding is not necessarily a result of bias but rather that it is more challenging for new investigators to write applications that are competitive with those of established investigators because as newcomers, they have less data and fewer accomplishments to cite. The benefits of ensuring that new investigators are brought into the system warrants special consideration for this group. However we are not wedded to any particular solution, as long as it ensures that a reasonable proportion of funding is distributed to new investigators. It is possible that the modified lottery by itself would obviate the need to create any special rules for new investigators.

    5. Individual investigators cannot be permitted to have more than one application per drawing, otherwise there would be a perverse incentive to flood the system with applications in order to increase one’s chances of success. The restriction would have to be NIH-wide to prevent applicants from circumventing the restriction by submitting applications to multiple Institutes or Centers.

    6. We agree that arguments can be made in favor of both established programs and new ones. Hence we conclude there is no compelling overriding reason to favor one over the other. We suggest that new programs should be allowed to compete with established programs on an equal footing.

    7. The selective service lottery was not unfair because it was biased. The selective service lottery arose in an effort to increase the fairness of the system because the wealthy or well-connected were often able to obtain deferments which exempted them from the draft. The system that we are proposing does not create special exemptions for well-connected individuals but rather is designed to prevent such favoritism.

    Best,

    Ferric Fang and Arturo Casadevall

  • Grumble says:

    How about this: instead of peer review followed by random selection, do it the other way round. So, first randomly pick 25%-35% of the grants and chuck the rest. The selected grants get reviewed for feasibility, and just the ones that are laughable are thrown out. Success rates within the randomly selected grants would be much higher than overall success rates, and for applicants, there would be much less stress over grants that get randomly chucked (vs the current system, where every chucked grant is chucked for cause) because there's neither malice nor poor judgment behind random selection. And colleges would have more skin in the game because they couldn't, in could conscience, dump faculty who can't get grants because of a string of bad random selection outcomes - they'd have to commit to supporting faculty through such periods.

  • qaz says:

    How about this instead? We fund science at the levels we need to! Then 25-30% of the grants would get funded and we wouldn't have this problem.

    We could just steer away from the stupid iceberg instead of fighting over who gets into the lifeboats. (And if the lifeboats don't find land soon, do we turn cannibal in the lifeboat?)

  • Ola says:

    A lottery is a tax on the poor/stupid.

    I don't see this as any different. Knowing that PO's are "persuadable" to bend the pay curve in ones' direction, why would any BSD worth their salt throw their lot in with the poor folks?

    I do like the (your) idea of rolling over previous well-liked proposals that didn't get funded, to the next council round. After all, if they're going to decide percentile scores based on the average of three study section cycles, the least they could do is let the proposal come up at council for three cycles. Of course, the reason I like this idea is because it gives me three chances to talk to the PO and persuade them, instead of the usual one.

  • Eli Rabett says:

    How much does a ticket to the lottery cost?

    This is obviously an invitation to gaming the system.

  • Eli Rabett says:

    Oh yeah, if this is for renewals to it has two effects.

    1. It throws the BSDs into the deep end of the pool with everyone else
    2. It makes it impossible to have a continuing research program beyond the five year horizon

    YMMV

  • Joe says:

    First off, I'm with qaz. We need more money for science.

    Secondly, are you kidding me? You want to make the system more random in terms of who gets funding? No. I think there have to be ways that working hard makes it more likely that you will get funded. Otherwise, why should I do anything but hide away and write grants and write papers?

  • AcademicLurker says:

    You want to make the system more random in terms of who gets funding?

    I'm not necessarily in favor of the proposal, but I believe the rational is to make the system openly acknowledge the degree of randomness that already exists.

  • Dave says:

    This is pathetic. The fact that these guys have dedicated any time to this 'plan' is beyond belief.

  • drugmonkey says:

    If no ranking were required, and applications were simply given a binary rating of ‘meritorious’ or ‘non-meritorious’, we suspect that there would be little need for a meeting. Most discrepant evaluations could probably be resolved in a brief teleconference.

    Ah, well perhaps it was my failure to grasp the full insanity of your plan. I will note first that we have two versions of the "binary" rating that contribute to scoring at present. We can draw upon this to evaluate your predictions. Reviewers are reasonably well aware of where the triage line falls and where the likely-fundable payline falls. They score accordingly. Current evidence from grant review shows that disagreement across these binary fault lines is considerable. This means that your apparent assumption that mostly people will agree on the binary line at 20%, 30% or wherever is flawed. I also disagree very strongly that discrepant scores could be resolved in a brief teleconference. In my experiences on study section it is the disparate scores near the perceived funding line that get the hottest and lengthiest discussions. It is in fact a huge demerit to the CSR that they have worked so hard to bring about Scarpa's dream of eliminating the face to face meeting. Structural impediments to score resolution, harmonization and movement may have been put in place (discussion in order of preliminary score, videoconference, telephone review and that horrible web forum ridiculousness) but these do not prove that meetings are not valuable. They have simply done an end run around that prior value.

    It is true that a modified lottery cannot prevent reviewers from placing a larger proportion of grants originating from specific subsets of applicants into the ‘non-meritorious’ group. However a modified lottery can at least ensure that bias does not determine the funding decisions made among meritorious applications, and would thus represent a step forward in addressing bias.

    The first is a huge problem. It is what drove the ESI bias in the first place. It can be observed in the Ginther report if you actually read the thing. It can be detected in study section if you are paying attention. Triage and perceived funding lines are approximately known and reviewer behavior is collectively organized around it. Your new drawing of the lottery line will simply change the point around which reviewer behavior organizes, making the in/out bias even more important.

    Given this, how do you know there is remaining bias that determines which grants get under the payline and which ones do not? After all, you are making the argument that it is already a crap shoot so we should formalize it into a lottery, are you not? That they are all equally meritorious under about 20%ile, yes? I think you are trying to have this argument both ways.

    we estimate would separate out 20-30% of applications from the remainder but the percentage might be higher.

    My point is independent of whatever you select as your cutoff for the lottery entry. When I first started reviewing, under the old scoring scheme, my SRO would send scoring distributions to the panel. You could watch the scores cluster around perceived funding lines with almost round by round fidelity to what the POs in the respective ICs were telling applicants about likely funding chances (these were ICs without published paylines so this is why I always talk about perceived rather than actual paylines) and past score/percentile relationships. It was positively spooky to see how a panel clustered scores around a moving payline target. The point at which you choose to place your cutoff doesn't matter.

    The problem that new investigators have in obtaining funding is not necessarily a result of bias but rather that it is more challenging for new investigators to write applications that are competitive with those of established investigators because as newcomers, they have less data and fewer accomplishments to cite.

    It will come to you in a minute dudes. Really. The NIH grant reviewing instructions are pretty clear on this. Reviewers may (and are encouraged in so many ways) ignore the "less data and fewer accomplishments" when it comes to ESI, NI and/or younger applicants. It is explicit. They obviously choose not to do so, which is a bias.

    Apply your same faux-legit reasons to all other categories of investigators that have, or do, suffer from lower grant successes. Zip code, women, URM, topic, models, approaches, etc, etc. Your logic says that none of this is a bias. Again, you can't have it both ways and you can't claim that some biases will be present in your meritorious/triage binary scheme and others will not. You are falling into the same hole that the NIH does when it comes to the ESI/Ginther responses. It proves that you don't really care one bit about one type of bias in review but you are careful to worry about and redress the other. Weak.

    The restriction would have to be NIH-wide

    Yes, it would. I am glad you recognize this. I disagree entirely with your one-grant-for-all approach. Sounds very NIGMS and very wrong. Not to mention there is no way in hell this will come to pass and you know it. The powerful will always protect their "right" to scoop up as many grants as they can.

    However, let's say you manage to accomplish this. It will still be an issue with respect to maintaining at least one proposal in the lottery pool at all times. It will still be an issue with respect to younger and not-yet-funded investigators having a faster-ticking clock. Within the ESI/young investigator population, you are biasing against the hard-charging and highly energetic ESI with such a restriction...almost Harrison Bergeron style. I am not okay with such a selection pressure...we WANT the hard chargers to do well. I want to see effort and imagination and ideas rewarded. Saying any PI can only have one proposal under consideration across the whole NIH at one time means the low-energy PI and the high-energy PI have more equivalent chances of getting funded than they do at present. I reject this notion that "all comers deserve one small grant" as the best way forward for the NIH.

    We agree that arguments can be made in favor of both established programs and new ones. Hence we conclude there is no compelling overriding reason to favor one over the other. We suggest that new programs should be allowed to compete with established programs on an equal footing.

    You are either missing the point or are being disingenuous. Your "equal footing" is siding against the argument in favor of established programs and in favor of new ones. I happen to agree with you that dismantling the advantage that competing renewals enjoy would be a good way to address certain forms of distasteful bias. But it comes at the price of a burden placed on other aspects of the structure of NIH extramural research. And that has proven already to be a huge cost to the system and the people who operate within it. Huge. Ignoring this, and pretending that the NIH extramural funding system could ever be truly project-based to the exclusion of any program-style support elements is wrong headed and doomed to fail.

    The selective service lottery was not unfair because it was biased. The selective service lottery arose in an effort to increase the fairness of the system because the wealthy or well-connected were often able to obtain deferments which exempted them from the draft. The system that we are proposing does not create special exemptions for well-connected individuals but rather is designed to prevent such favoritism.

    The rich and powerful still managed to keep their kids out of the lottery. The outcome was biased. And the cover it provided let people like you pretend that fairness was enhanced when the system was still biased. I view your proposals no differently. You skip blithely over the big driver (your binary merit decision) and quibble over the smaller input (the lottery to save the 20%ile-payline score population). I am unimpressed. Your Vietnam selective service analogy is certainly apt in all of its glorious fail, but I don't think it makes the argument you think it does in favor of your scheme.

  • drugmonkey says:

    The fact that these guys have dedicated any time to this 'plan' is beyond belief.

    YEAH! who would spend any time thinking, writing and talking about how to make the NIH system work better? Screw those guys!!!!

    .....wait.

  • Dave says:

    You know what I mean. I realize we are in a place where ridiculous solutions are given space, but this is just stupid.

  • Jonathan Badger says:

    If it is really a ridiculous suggestion, would you be willing to bet that a sample of grants given randomly compared to a set of those given competitively would yield significantly worse results (by whatever metric you choose)?

  • Ferric Fang says:

    I understand the skepticism about a modified funding lottery. I was initially skeptical too. However, after thinking more about it and poring over the available data, I have concluded that such a system might provide substantial benefits for the research community. The process would become more transparent, applicants would have fewer applications to write, the burden of reviewing would be reduced, and the influence of bias in ranking meritorious applications would be neutralized.

    1. Of course I agree that an increase in research funding would be better than any scheme to reallocate available funds. Nevertheless, a lottery is not without its potential benefits in this regard. A modified lottery would allow research advocates to state that 'last year scientists submitted xxxx grant applications that were judged to be meritorious after peer review, but only xx% of these were awarded as a result of insufficient funds.' This would be a useful piece of data in negotiations over research funding.

    2. Our and others' recent analyses of funded applications have shown that study sections have essentially no ability to predict the future citation and publication productivity of research projects (Fang, eLife 5:e13323, 2016; Danthi, Circ Res 114:600, 2014; Berg, ASBMB Today 12:3, 2013). Therefore the current system is already arbitrary, as well as being prone to bias. A lottery of meritorious applications would still be arbitrary but would at least reduce the potential for bias.

    3. As DM points out, the initial triage stage would still be susceptible to bias. I see this as a necessary evil, because the disadvantages of a completely random system would be greater. Nevertheless, there are measures that could be undertaken to improve the precision of triage decisions (Kaplan, PLoS One, 3:e2761, 2008), such as involving larger numbers of reviewers so that a single negative review doesn't sink an application. Authors of triaged applications would receive detailed reviews in the hope that they could subsequently submit a qualifying application. However, for the 'meritorious' group, subsequent selection would be random, providing an advantage over the present system in which the influence of reviewer bias appears to be pervasive (Johnson, PNAS 105:11076, 2008).

    4. I do not agree that racial bias is solely reflected in the initial triage decision. In table S1 of the Ginther report, it is evident that the distribution of scores of applications from White and Black investigators are not identical-- the scores of White applicants are skewed leftward, with a median around the payline, whereas the scores of Black applicants are closer to a normal distribution. Thus, a lottery that included all scored applications would have improved the proportion of funded applications from underrepresented investigators.

    5. I would also point out that the triaging of applications as meritorious or non-meritorious would not rely on a pre-determined cut-off, hence there would be no clustering of scores.

    6. Any change in a system produces winners and losers. We agree that labs with large numbers of grants under the current system would be likely to experience a reduction in funding. However science as a whole would be likely to benefit from a larger number of labs receiving funding, and perhaps from greater efficiency as well, as there is a fall-off in productivity as funding levels for individual labs exceed $750K per year (Wadman, Nature 468:356, 2010).

    7. The unfairness of the Selective Service Lottery was not due to the lottery itself but rather a result of the many ways in which privileged individuals could obtain deferments. There is no reason that analogous exceptions would have to be part of a funding lottery. The analogy of the Selective Service Lottery was only raised to illustrate that lotteries have been used in life-or-death situations, so we are not trivializing the importance of research funding by suggesting that a lottery system might be used to award grants.

    Science is inherently uncertain, and the biggest discoveries are typically the least likely to be predicted. Hence it is intuitively appealing to stop pretending that we can predict the future success of grant applications and to adopt a more impartial funding process. Thanks for your consideration.

  • jmz4 says:

    @Ferric Fang
    One, I assumed Ferric Fang was a pseudonym. Congrats on having such a cool name.

    I don't think you're getting pushback just because of the lottery system per se. A lot of people have acknowledged that deciding amongst the top 20% is essentially throwing a dart at a wall. Maybe more details are in the references you cite, but for the sake of discussion, could you clarify what your answer to Eli's question would be.
    Namely, how would you handle renewals and the like? Obviously having a competing renewal go into the lottery every 5 years is unworkable, since that would make the funding system significantly *less* stable, and many worthwhile questions take more than 5 years to answer and follow up on.

    Your system seems geared to support low budget science. It would decimate labs that rely on more 2 concurrent R01s and have high research costs since funding could not be sustained long term by anyone, statistically.

    " ' The reason markets work is because they allow people to be lucky, thanks to aggressive trial and error, not by giving rewards or incentives for skill.' "
    -I think you'll find you get a lot of pushback from vertically ascending scientists on this sentiment.

    For those of you on study sections. What percentage of applications do you think are "meritorious" anyway?

    Is it worth doing all this essentially just to assuage the psyches of the PIs routinely just missing funding lines?

  • drugmonkey says:

    applicants would have fewer applications to write

    The lottery aspect doesn't directly affect this outcome. This potential benefit is related to your proposal to roll unfunded apps forward for continued chances at funding. There are other ways to accomplish this goal.

    Therefore the current system is already arbitrary, as well as being prone to bias.

    Your productivity analysis, btw, is flawed by the truncation of the distribution as well as convenient definitions of productivity. but the more directly relevant issue for today is that you have not proved that there is consistent bias in placing grants between the payline and whatever binary cutoff you select. What is your evidence for this and what is your evidence this bias would not be the same bias that keeps some applications outside of your lottery population?

    the initial triage stage would still be susceptible to bias. I see this as a necessary evil

    It astonishes me that you cannot grasp that this initial triage stage has a larger impact than any rearranging of the deck chairs under the cutoff might effect. Consequently you cannot just blow it off as a "necessary" evil while continuing to argue that your method has an impact on "bias".

    I do not agree that racial bias is solely reflected in the initial triage decision. In table S1 of the Ginther report, it is evident that the distribution of scores of applications from White and Black investigators are not identical-- the scores of White applicants are skewed leftward, with a median around the payline, whereas the scores of Black applicants are closer to a normal distribution. Thus, a lottery that included all scored applications would have improved the proportion of funded applications from underrepresented investigators.

    I want you to think about this a little harder and my points about reviewers being sensitive to the perception of where the payline lies. I didn't say racial bias is solely reflected by anything. as a matter of fact Ginther makes an excellent case for the death of a thousand cuts. But you show right here that the nature of privilege is the skew toward the good side of the perceived place where fund/don't fund decisions get made. You might call this the benefit of the doubt.

    I would also point out that the triaging of applications as meritorious or non-meritorious would not rely on a pre-determined cut-off, hence there would be no clustering of scores.

    As I mentioned, my experience is that scores cluster around the perceived place where the payline was going to be, even for ICs that don't publish paylines. As we see in Berg and Writedit's book, there are effectively paylines even when unpublished. Reviewers have a sense of this because they are interacting with peers and POs and figuring out what funds and what doesn't all the time. Trust me, there will be fairly accurate ideas of where your merit cutoff will be and reviewers will cluster around it.

    However science as a whole would be likely to benefit from a larger number of labs receiving funding, and perhaps from greater efficiency as well, as there is a fall-off in productivity as funding levels for individual labs exceed $750K per year

    That is a completely bogus thing to continue to bleat about until and unless you grapple with the effects of glamour humping labs versus the rest of us. Also, when it comes to policy, it is ridiculous to ignore the effects of funding gaps versus continued support. Finally $750 is three R01s at full mod. Which never happens so we know that is really 4 grants. The vast, vast majority of NIH PIs don't have this kind of sustained funding. So where will your lottery plan really work? Will it keep the 4+ grant labs down to 3? Or is it really about keeping the 2-3 grant labs at 1? Let's be forthright if we are going to refer to 3 grant-is-most-productive data to support our plans to reduce everyone to one budget-reduced grant.

    . There is no reason that analogous exceptions would have to be part of a funding lottery.

    You yourself in your proposal have already called for exceptions for ESI and for the serially unlucky. And asserted a role for Program to put a finger in the pot. So you have already let the camel's nose under the tent.

    Hence it is intuitively appealing to stop pretending that we can predict the future success of grant applications and to adopt a more impartial funding process.

    What is your evidence that it is not impartial in ways that your plan could fix? Or that a lottery is needed to fix it instead of merely rolling grants forward for further consideration? I think you are asserting a plan from a position that everyone who applies deserves to get a grant and you imagine that the system somehow can satisfy this. I am not sure you've been following the math over recent years. We have too many people vying for a fixed pool of funding. The typical grant remains fixed in current dollars, or shrinks, while costs inflate. Purchasing power of "one grant" now is not anything close to what it was when they started the modular system.

    Your efforts to save the NIH extramural system would be much more profitably devoted to figuring out how to align the size of the research mission with the available funding. Instead of figuring out how to distribute the funds so thinly that we have lots of labs "funded" that aren't actually able to accomplish any science. The R01 is not a participation medal.

  • drugmonkey says:

    What percentage of applications do you think are "meritorious" anyway?

    This is an unreasonable question. It is a matter of priorities.

    For example, I'd rather fund every damn grant (okay, say 98%) ever submitted to the NIH if we could subtract the necessary money from the Department of Defense budget.

    What grants should be funded in preference to the ones I want funded in my lab? zero.

    In between that space we have a variety of ways to construe "merit". And it is never just one thing.

    In my more serious moments I'd argue that it would be good taxpayer investment to fund any of the top 50% of applications that I've seen come through a study section. Maybe 66%. Because this is about the percentage that have at least one reviewer who is reasonably excited about it. If you have one person saying this grant would be good enough to meet the percentages that they used to enjoy in the 80s.....well that's good enough evidence for me that the difference between 1, 2 or 3 reviewers agreeing on that judgment is chance.

  • 3. There is no need to assign a specific cut-off for grants to be entered in the lottery, simply a designation as ‘meritorious’ or ‘non-meritorious’, which we estimate would separate out 20-30% of applications from the remainder but the percentage might be higher.

    The percentage of grants that my study section reviews that are "meritorious"--defined as likely to lead to interesting useful published results--is probably about 75%. Regardless, my sense reading their gibberish is that neither of these fuckers has served on a study section in a long fuckcne time.

  • Grumble says:

    "But you show right here that the nature of privilege is the skew toward the good side of the perceived place where fund/don't fund decisions get made."

    Right. So let's suppose that study sections had a scoring range of 2: Fund or Not Fund. You'd then expect more applications from minorities to be scored Not Fund, due to bias. That's bad, but maybe a binary scoring system would actually be less bad than the current system. First, a binary system at least forces reviewers to no longer disguise their bias with a wishy-washy score. They have to come out and say "Not Fund". It might be harder to do that than to give, say, a 4 instead of a 3. Second, it might be easier for SROs to remind study sections that they are expected to produce similar success rates for minority and non-minority applications.

  • Dusanbe says:

    I wish my name was Ferric Fang.

  • Eli Rabett says:

    "I wish my name was Ferric Fang."

    Not if you are under 18.

  • Pinko Punko says:

    ^Eli is a treasure

    I'm not convinced reviewers wouldn't be able to handle such a situation. I certainly don't think it would increase bias. I expect it merely compiles many arbitrary factors into one. There are inequalities to current review that would be mitigated, for example reviewer presentation skill- a massive wild card. This is not in support of the proposal, just an identification of an arbitrary current feature of NIH process.

    More dollars in the system and I think review quality increases, and reviewers are not tasked with as difficult of a task. I think the process works reasonably well in absence of massive structural pressure.

  • Jonathan Badger says:

    @Dusanbe,

    I hear his brother Ferrous Fang feels rather reduced in comparison.

  • drugmonkey says:

    PiPunk- more dollars isn't likely. Is there another alternative that serves the same purpose? One that is in our own capability to accomplish?

  • Pinko Punko says:

    This is what I doubt. The cull will continue and I think we will see grant dollars redistribute/be increasingly concentrated at high resource/high density places. Places where resources allow longer survival and continued hiring.

  • drugmonkey says:

    The Cull is coming. It is within our collective power to manage it, if we have the will. Otherwise it will be imposed. Organically, as they say.

  • jmz4 says:

    @Qaz, more money isn't going to fix the problem unless we first address the ravenous feed-forward loop that is the scientific workforce pipeline. The NIH budget could triple and 10 years from now we'd end up right back where we started. Unless you think there is a ceiling where everyone who wants to and can be a PI has a lab (but then who would work for them?)

    In terms of what we can realistically accomplish within the current framework to address the tendency of the system to boom and bust, there's a couple things. Reducing the number of soft money positions is a possibility, and would help ease the boom phase. So would limiting the number postdocs that can be supported on NIH grants as well as reducing/eliminating grad student tuition payments off grants. For the former, I'd advocate a max of 5 years PD salary support per R01. The idea being to shunt postdocs towards fellowships (so 2 years on your boss, 3 on a fellowship), which will give the NIH more control over how many of them there are, and also select for those that have grant-writing ability to make it as PIs. Concomitant, you need an expansion of senior technical support staff. Really, nothing that hasn't already been discussed on this blog, but the NIH just has to realize that, beyond the overblown doomsayers' dire predictions, these reforms will actually benefit research productivity, as well as make them infinitely more credible when they go to Congress, hat in hand.

  • qaz says:

    @jmz4 - the question is how many people are there in the US really want to do science? I think (a) That there is a limit to how many people really want to be PIs [we are nowhere close to that limit, but it is a limit - being a scientist is not for everyone, really, not], (b) If we got away from the "grad students have to become PIs to be successful", we could train lots of people to go to other careers and it would be fine. (c) Moreover, if funding were stable and strong and easy to get, then there would be plenty of stability to support long-term non-PI scientists. (Which was very common in the Golden Age, where a "lab manager" or "researcher" or "technician" could make an entire career in someone else's lab.) And then we wouldn't need to live off of graduate students and postdocs. A large part of the reason we live off graduate students and postdocs is because they can get fellowships, while technicians can't.

    If we funded science like we did in the Golden Age (when people could live on one R01 because funding percentiles were at 30%), the entire science budget, including NIH, NSF, DOD, DOE, NASA, USDA, etc would quadruple (funding then was 12% GDP, it's now 3%). Moreover, back in that day, we spent additional money making college nearly free, so UG and Grad students were a lot cheaper because they/we didn't have to pay tuition.

    To give a sense of scale, the Apollo Mission to the moon would be $160 billion in today's dollars. The Brain "Apollo Mission to Neuroscience" and the Cancer "Apollo Mission" that Obama proposed in the last state of the union was $100 million each. We're off in terms of funding by a factor of 1000. Do you really think there are 1000x more potential PIs? (By the way, that's not 1000 more, but 1000 times the number of current funded PIs.)

    Of course, all of that's a pipe dream and it is never going to happen and we're trying to find ways to climb on each other's backs before the whole boat sinks. The Cull is already here and happening and has been for the last decade or more. We can argue how to manage it, but it still comes down to trying to survive in the thunderdome.

  • Jonathan Badger says:

    @jmz4
    "Reducing the number of soft money positions is a possibility, and would help ease the boom phase"

    For a lot of people that *would* be the cull that we supposedly are trying to avoid, though.

  • AcademicLurker says:

    The Cull is already here and happening and has been for the last decade or more.

    Do we know how many casualties The Cull has already claimed? The most relevant information I can recall is a post of Data Hound's showing some numbers on the decline in investigators holding R01 equivalents, but doesn't really tell the whole story. Are those non-NIH funded investigators still submitting proposals? If the answer is "yes", then I wouldn't say that they've been culled yet.

  • drugmonkey says:

    Nope, we have no clear information on the Cull other than the sort of thing DataHound posted.

  • drugmonkey says:

    @jmz4
    "Reducing the number of soft money positions is a possibility, and would help ease the boom phase"

    For a lot of people that *would* be the cull that we supposedly are trying to avoid, though.

    There is no conceivable way to avoid the Cull, until and unless 1) NIH budgets improve and/or 2) we figure out nonCull ways* to match the research force with the spending power of the NIH allocation.

    Culling soft-money PIs preferentially is one clear item for discussion that is on the table.

    And then we wouldn't need to live off of graduate students and postdocs. A large part of the reason we live off graduate students and postdocs is because they can get fellowships, while technicians can't.

    Individually this may be the case but do not lose track of the fact that the NIH extramural system depends on grad students and postdocs because they are cheaper. Weaning ourselves from this system of labor exploitation* is the proper moral and ethical course but it will put further pressure on the available grants. Therefore it will amplify the Cull. We need to face this reality with fortitude and honesty.

    *Getting serious about shrinking graduate student enrollment serves both of these goals, btw.

  • […] was just discussing the continuing nature of the ESI bias in a comment exchange with Ferric Fang on another thread. He […]

  • effie says:

    Shakespeare wrote "Good ideas must of force give way to better" but science shows that actually "Good ideas suppress the better ones". Known as the einstellung effect, this pattern is well documented in cognitive science. I believe it is a very significant factor in NIH review and has reduced the overall effectiveness of the U.S. research enterprise.

    An element of randomization would be most effective in reducing the impact of intellectual biases.

    Fang and coauthors correctly point to the analogy of a financial bubble; we appear poised to enter another bubble now with the rise of "precision medicine", particularly in cancer.

    I am a PI with three years of NIH scores consistently in the top 20% and no R01s funded from that mix. I am personally likely benefit from a lottery. But more broadly-- I expect that many innovative approaches are just missing the pay line due to the inescapable tendency of humans to adhere to einstellung thinking.

    The fact that some respondents on this blog reacted with such a knee-jerk negative reaction to the lottery proposal further emphasizes this impression.

  • jmz4 says:

    @Qaz
    Increasing research money to 100 or 1000x what it is right now is fiscally impossible. 10x would be about the max, and that's politically impossible, since it would require reducing the military budget by 80% (or some combination of military and social spending). But let's say that's what happens, we get a 400 billion NIH budget.

    Currently something like 70-80% of newly minted PhDs plan on doing a postdoc and running a lab. Only about 1/10th of them land a faculty job in their first 6 years. So the system could get a 10x increase and just barely accommodate all the current postdocs. And that's at the current thunderdome funding rates.
    So I don't think talking about fixing the pipeline issues is secondary to advocating for more funding.
    As DM points out, many of the fixes for the workforce require more money (like paying solid wages to staff scientists to make it an attractive career option). So I'd advocate any extra NIH bumps get allocated to addressing these issues first. Then we'll be in a stable place that can roll with the ebb and flow of research funding.

    DM, what's your reasoning behind advocating for reducing grad student numbers instead of just bottlenecking at the PD phase? I'd argue that grad students currently get a pretty good deal (free degree and reasonable stipend), and so are less exploited. Also, scientific training is useful in many other endeavors, and so the net benefit to society is to continue training grad students.

  • drugmonkey says:

    The fact that some respondents on this blog reacted with such a knee-jerk negative reaction to the lottery proposal further emphasizes this impression.

    I've written considerable more on why I think F&C's argument for a lottery is flawed than you have. How is this more "knee-jerk" than your opinion based on what appears to be the logic of "I haven't been successful yet under the current system so something else must be better"?

    I'll ask you the thing I was asking Fang. How do you know that your scores between the payline and some cutoff for this future lottery reflect a bias that needs to be redressed?

    This is the argument they did not make in their Editorial and seems to me to be the essential justification for the lottery system they are proposing.

    Let me put it another way. Even in F&Cs lottery, some meritorious applications will go unfunded. Just as right now, if we accept most of our opinions that current study section based ranking is random once you get down within the 20-30%ile range, the difference between funded and unfunded is chance not merit.

    F&C are asserting some sort of bias that their lottery would address but they haven't made the case for that bias. The real nugget for improvement in their proposal is really the fact that a given reasonably-meritorious application would re-enter the lottery unchanged. Without a PI having to lift a finger. That is the only clear advantage.

  • Grumble says:

    "F&C are asserting some sort of bias that their lottery would address but they haven't made the case for that bias"

    Effie made the case for it by bringing up the einstellung effect. Most people accept that study sections tend to be too conservative. Poor scores are therefore likely to be due to a conservative bias. A dose of true randomness would serve as a partial corrective to that bias.

Leave a Reply