NIA paylines and anti-ESI bias of review

Apr 20 2016 Published by under NIH, NIH funding

MillerLab noted on the twitters that the NIA has released it's new paylines for FY2016. If your grant proposal scores within the 9%ile zone, congrats! Unless you happen to be an Early Stage Investigator in which case you only have to score within the top 19% of applications, woot!

I was just discussing the continuing nature of the ESI bias in a comment exchange with Ferric Fang on another thread. He thinks

The problem that new investigators have in obtaining funding is not necessarily a result of bias but rather that it is more challenging for new investigators to write applications that are competitive with those of established investigators because as newcomers, they have less data and fewer accomplishments to cite.

and I disagree, viewing this as assuredly a bias in review. The push to equalize success rates of ESI applicants with those of established investigators (generational screw-job that it is) started back in 2007 with prior NIH Director Elias Zerhouni. The mechanism to accomplish this goal was, and continues to be, naked quota based affirmative action. NIH will fund ESI applications out of the order of review until they reach approximately the same success percentages as is enjoyed by the established investigator applications. Some ICs are able to game this out predictively by using different paylines- the percentile ranks within which almost all grants will be funded.

NIA-fundingpolicyAs mentioned, NIA has to use a 19%ile cutoff for ESI applications to equal a 9%ile cutoff for established investigator applications. This got me thinking about the origin of the ESI policies in 2007 and the ensuing trends. Luckily, the NIA publishes its funding policy on the website here. The formal ESI policy at NIA apparently didn't kick in until 2009, from what I can tell. What I am graphing here are the paylines used by NIA by fiscal year to select Exp(erienced), ESI and New Investigator (NI) applications for funding.

It's pretty obvious that the review bias against ESI applications continues essentially unabated*. All the talk about "eating our seed corn", the hand wringing about a lost generation, the clear signal that NIH wanted to fund the noobs at equivalent rates as the older folks....all fell on deaf ears as far as the reviewers are concerned. The quotas for the ESI affirmative action are still needed to accomplish the goal of equalizing success rates.

I find this interesting.

__
*Zerhouni noted right away [PDF] that study sections were fighting back against the affirmative action policy for ESI applications.

Told about the quotas, study sections began “punishing the young investigators with bad scores,” says Zerhouni.

Note: It is probably only a coincidence that CSR reduced the number of first time reviewers in FY2014, FY2015 relative to the three prior FYs.

17 responses so far

  • Pinko Punko says:

    What I see is that newbs write reasonable grants that are always modular. They then write an amount of science that makes sense for those budgets. They will compete in pool with larger grants from established PIs with other means of support, etc. Therefore the significance of the grants will usually un some way appear to be different, so the smaller grant almost always will be scored lower. The lower score is basically built into the system, and pressure is only increasing given the push from panels for "impact impact impact"- I also think established PI grants that are in the churn start to get better and better. It is possible that they have gone 2-3 rounds. These will look better than a newb grant. The are many reasons scores go against ESIs. Some of it likely bias, some of it systemic.

  • drugmonkey says:

    The review instructions are very clear that one may account for career tenure. The affirmative action policy itself, and all the associated shouting, make it VERY clear that one should take it into account. Therefore this is all reviewer bias. There are no valid objective and justifiable reasons for the disparity in scores.

  • potnia theron says:

    Just because the instructions say you *can* or *should* take something into account, that the reviewers do or don't. Differences in paylines to achieve equal % funding could be bias or more poorly written grants. I do not perceive that you have proved that this is the case.

    Have you not gotten better at writing over time? I believe that I have. Is not grantsmanship (grantspersonship?) something that be learned?

    However, I am not sure how one could set up a test, with available (or potentially available) data to separate out whether it is bias, or in fact the inherent and (whatever) weaknesses that a junior person would have in putting together a proposal.

  • damit says:

    It's because the ESI/New Investigators haven't learned to use DM's favorite reference format yet.

    Once they're clued into that, no more problems.

  • Pinko Punko says:

    I was just raising point about the drift, which I would link to churning in some cases. I also would link it to when paylines are in better shape, I think reviewers likely to feel more comfortable looking past minor grantspersonship or easy StockCritiques for ESIs. Each of those are easier to employ against ESIs as they are more likely to be prevalent. I don't support this happening, but if there was any aspect of ESIs getting a "break" due to being new on those issues in the past, now there is no question about getting that type of "break"- no way.

    I also think that there may be assumptions that ESIs have a cushion of startup, while mid careers are in the grant or street mode. Think all of these things are at work, but only based on my personal anecdata experience. I feel that my funded ESI app now, if it were currently in the mix, might only get discussed because it were an ESI app, and possibly funded, but in my study section I would see it getting a much lower score and likely definitely lower percentile than it did when it was funded.

    Can also say that a few years ago I had feeling that ESI proposals I had read were of higher quality, yet it is possible that the churn increases the number of senior investigator proposals in the pile that would normally have been funded under better paylines. This also does bring up the average quality of those proposals that might be under a 9% payline. I think based on dynamics of how things are churning, relative to rate of ESI submission, there could be some situation where the gap in quality between the two sets could increase.

    Established PI apps are going multi-investigator, are having increasing amounts of prelim data, and are being really tenderized by the system.

  • Dave says:

    Established PI apps are going multi-investigator, are having increasing amounts of prelim data, and are being really tenderized by the system

    Definitely. I'm a (small part) collaborator on a multi-BSD PI R01 app and after reading the review comments it's quite demoralizing as an ESI trying to compete in the same/similar SS. How can I compete? I thought the grant was badly composed, unimaginative and "me too" science with barely any preliminary data. But it did very well in the first round and that clearly had a lot to do with the PIs.

    What about completely separate SS for EI/ESI?

  • Jim Woodgett says:

    Hey, at least the application and reviewer guidelines don't actively discriminate against ESIs like the inaugural CIHR Foundation scheme which scored for qualities such as demonstrated mentorship and leadership. They set a minimum of 15% success rate to partly offset the bias but the awardees were then slapped with one third of the budget on non-ESI awardees.

    The second competition (underway) adjusted this to separately review the ESIs but this meant that (as Canada has a small research community), the breadth of expertise of the reviewers was stretched (as each must review multiple grants).

    Seems to me that the current NIH system that adjusts for reviewer bias by equalizing proportional success to applications is probably the best way to achieve equality. Reviewers be biased.

  • DrugMonkey says:

    I do not perceive that you have proved that this is the case.

    The scores prove that it is the case. The burden is on you to advance evidence for whatever reasons you think make this deserved rather than undeserved bias.

  • dr24 says:

    How often does getting an ESI award come back to bite someone too?

    "He only got his first R01 because of the affirmative action, not because of the quality of the grant. No soup for him!"

  • drugmonkey says:

    Your first (major) NIH grant is by definition either ESI (if you meet the time since Phd criterion) or NI. So I'm not sure this really applies.

  • drugmonkey says:

    Historically I guess you could ask about those who got a R29 FIRST award versus a regular old R01 at the same time. But my recollection was that FIRST was viewed fairly positively (it tended to cripple your launch relative to a regular R01 but that's a topic for another day).

  • Anonymouse says:

    [Amonymouse wakes up, perks up his ears and squeaks: "double-blind review!"]

  • odyssey says:

    The R29 FIRST awards were crippled by low direct costs ($75k/yr?)...

    Anyone doing Alzheimer's-related work should be very, very happy with NIA's new pay lines.

  • Pinko Punko says:

    I've been thinking about this more. The flip side to DMs argument is how do you disprove that ESIs were previously getting a break, but now there is likely perception that those breaks not needed because of the normalization policy? I would say that this could lead to bias because whatever reviewers unconsciously thinking about "breaks" would be operating for other grants, whereas there will be incorrect perception of "score freedom" on ESIs. I don't know. I do think there is increasing gap between ESI grant average quality and senior PIs. Part of this could be new PIs are always new. Senior PIs are increasingly the survivors of selection. Losing a lot of mid career people.

  • drugmonkey says:

    I agree with you that the perception of a new payline for ESI is probably more accurate than Zerhouni's "punish" descriptor. But I think that interpretation just points even harder at the bias. Reviewers are still saying the apps should not fund equally with the established PI apps.

  • Pinko Punko says:

    Or that the most commonly employed criteria are stacked against ESIs, which is true. I think the prelim data one is most pernicious. Even for established researchers, if their is significant record of accomplishment what does it mean to ask for clearly generatable data? Paid for by whom? This is part of the wink wink- those projects getting off the ground are mostly paid by other grants once startups run out. If an idea seems really good and could be important, and experiments are obviously feasible, what does it mean to reflexively demand prelim data? It's an unfunded mandate. When ESIs get this, it is definitely a "wait in line". When mid career get this, it could be "go die". But these days, there will be so many grants that are stuffed with data because they've been around for 2-3 cycles. The more that happens the thinner the ESI grants will appear (and the harder road back for anyone with a gap). So here is a question: is it even possible for someone to come back from a gap? At what point is it impossible?

  • Ass(ociate) Prof says:

    I'll share my anecdata about study section bias. When I was an ESI circa 2o13 and targeting applications to one institute, I noticed that another institute had broken out their clinical and basic R01s into two separate RFAs. Work I was developing fit very well in their basic science descriptions. I called the PO to talk about submitting and get a little more feedback on fit. The response as follows:

    PO: "You're junior faculty? Don't bother applying. I have a lot of senior people going in and you won't be able to compete."

    We didn't even get to the specifics of the project. I think that acknowledges that some aspects of merit review are indeed a charade.

Leave a Reply