MENU
Close Menu

More on CSR’s attempt to reduce “reputational bias” in grant review

Diversity in Science / NIH / NIH Careerism / NIH funding

A Science and Policy item by Jocelyn Kaiser covers the recent proposals on changing grant review criteria from CSR. In her framing “Researchers familiar with the National Institutes of Health’s (NIH’s) peer-review process can cite many cases where the reputation of a scientist or their workplace seemed to count for more than the strength of their ideas.“. Distressingly anecdotal, but sure, many scientists do assert this now and again. I have certainly had experiences in grant review panels that would be consistent with the notion that a Big Deal PI can often get a great score with a proposal that one predicts would do much more poorly with a lesser-known PI at the helm. Kaiser quotes Noni Byrnes, Director of the CSR, as follows:

“Anyone who actually attends a study section sees [reputational bias] happening,” says Noni Byrnes, director of NIH’s Center for Scientific Review (CSR), referring to NIH peer-review panels. Now, in a bid to reduce that bias, her center is proposing the first overhaul of NIH’s scoring system in 14 years. But the idea is getting a mixed reception.

I had previously discussed my skepticism that merely instructing reviewers not to focus on the applicant institution and PI reputations was going to produce any functional change. Some wag on Twitter (I’ve since lost track of the tweet) observed that it is odd that someone who has been employed at CSR for over two decades is only now getting around to doing anything about this reputational bias problem. She started as a SRO so presumably she “actually attend(ed)” many a meeting.

As per my usual, this has me returning to the structure of NIH grant review panels and the inherent grinding circular conservatism of recruiting reviewers from the ranks of the already-successful. That is a pretty entrenched rule. A related concern is the idea that SRO’s select reviewers as a limited subset of those who are submitting grants. It is much harder to assess, but this is another place where reputational biases related to the person’s institution or the person’s connections in the field may accelerate the inherent conservatism. Today I am off in the little corner of the Conflict of Interest rules that may be relevant. Not the institutional employment ones, but the ones having to do with a significant mentor/mentee relationship. For NIH grant review, this is codified as having co-published within the past three years or has had other professional relationships (“served as mentor” is the only example given) in the past three years.

Training pedigrees play into both the kind of bias for topics that was identified in Hoppe et alia (2019) as well as the reputational bias of present concern to Director Byrnes. There is just no denying that a given research active scientist get direct career benefit from the successes of their academic progeny. Amazing scientists who publish great papers, get grants, train additional scientists…these are of specific credit to their mentors. They are the evidence used in mentoring awards. They are writing papers that cite the mentor’s work. They are doing the science in the same general area or using the same general techniques, thereby persuading everyone that this is hott stuffe. Mentees generate that sense of importance for the mentor’s research that then influences said mentor’s grant review outcomes. These benefits of the success of a mentee do not magically end after three years. If anything, the benefits are weakest in the first three years since the mentee is just getting their operation up and running.

It is in the career-long interest of a mentor to give good scores to the proposals of their once-mentees.

And vice versa, once that mentee has established themselves. And it just keeps going on into the next generation. The success of academic grandchildren also benefit the career success of the academic grandparent.

Interestingly, Twitter is not seeing it quite this way today. Now I asked the question as generically about peer review as I could.

I was heartened to see that at least on the time limitation, there was some agreement that the COI basically never ends. But when I asked about extending this to subsequent generations…

…the audience balked.

I think because it is easy to think of bias as a very specific and individual thing. A matter of personal relationships. Merit being overmatched by a presumed close personal feeling of wanting that person to do well. It is much harder to think about these longer arcs of professional benefit. It seems entirely defensible for individual reviewers to advocate strongly for funding science in their own areas of interest or using techniques, models or approaches that they just so happen to also use. If that happens to be a trainee from >3 years past, well, that’s just the science talking, right?

And yet this is very intimately related to the problem of “reputational bias” that Byrnes is trying to diminish.

Does it help to translate this to academic hiring? A very specific and individual benefit to the prospective new Assistant Professor accrues if their mentor is on the search committee and can talk them up. Or, to maybe lesser extent, can talk the competitors down. Maybe your institution has COI rules in this case, and maybe the colleagues all know, because there’s the mentor, prominent on the CV. But what if it is a second generation? Maybe it isn’t so obvious that Candidate is the postdoc of Sonny who trained with your colleague on the search committee.

What about manuscript review? Well, here we probably have much lesser concern since the publication of papers is much less zero-sum than the awarding of NIH grants or Assistant Professorships. Yeah, yeah, impact factor, but ultimately you can publish just about anything…somewhere. But to the extent that citation homophily is a problem…it likely extends to academic family trees. And high impact journals do count. And peer review comments can either speed publication or mire it in the stickiest of muds. Those have real impact on the career as well….and ultimately on grant success.

And so the cycle of reinforced conservatism of science continues.

Read More...

Describing your “contribution to diversity” is a workaround

Academics / Diversity in Science

I don’t know who needs to hear this but….

(as they say)

The statement about how you “contribute to diversity” in the academy can be very, very irritating for people of color.

It is a workaround. Maybe that will make you feel better about it, maybe not. But it is true.

It is a workaround for the brief fight (ok, at least fifteen years through the 80s and 90s) which Affirmative Action lost.

In 1996 the voters of the allegedly liberal state of California voted on Proposition 209, a ballot initiative to ban affirmative action from any government institution, most pointedly the University of California and the California State University. It passed.

There are numerous other laws, policies, initiatives, etc. and countless news articles and political blather that contributed as well. Far too many to mention. Try Google if this is news to you.

Affirmative Action was declared null and void. Illegitimate. In cases like Prop 209, illegal.

It is no longer cool to address your concerns about your lack of diversity, equity and inclusion (which are, for now, still legal) by going out and hiring people because this very act will improve your institution.

Hiring someone because of their race is illegal. Hiring because they contribute to the diversification of your institution is totally fine.

The forces of goodness may really, in their heart of hearts, intend to hire someone because of their race. They may point to under representation of that race in their ranks as an example of the DEI deficits, totally legal. You may apply to that job or training program with the assumption that they indeed mean that your race is something that makes you extra attractive.

But nobody can say it. Nobody can hire you *because* you are Black or LatinX or other category intrinsic to your racial and ethnic identity. They can hire you because you contribute to diversity. So you have to work with that. The workaround. That we all have to live with.

Because Affirmative Action lost the war.

Read More...

Peer reviewers cannot be trusted with the details

Diversity in Science

The invention of the modular budgeting process for NIH grants was in large part, I have gathered, because reviewers were too often getting too deep into budget details. Wanting to complain about how listed rat per diems were clearly outrageous at Big U because their per diems were less. To try to account for those who thought that clearly the PI was estimated 20% more pipette tips than needed, and quoting full price when the reviewer always knew how to get em on the cheap. Being found ranting about paying a 15 year veteran tech an outrageous salary when surely a fresh out of B.S. one would suffice.

I am not absolutely certain about this since the modular budgeting was put in place before I wrote my first NIH grant.

More recently the NIH is proposing changes to the review criteria that are supposedly to “simplify” the process. One of those changes is an attempt to force reviewers to stay out of the details of the Investigator qualifications. The bullet point and criterion score approach of today is proposed to be replaced with a qualified/not qualified drop down menu. The report [PDF] of the peer review working group also found that “Reviews of approach too often focus on technical minutia, overly weighting problems that skilled scientific teams could overcome” which “drove Recommendation 9, to revise the review template to discourage reviewers from identifying methodological problems that skilled scientists are likely capable of overcoming.“. They even recommended ditching “Innovation” as a major criterion “to avoid having otherwise highly meritorious scientific applications be inappropriately downgraded by lack of innovation, while not diminishing the general value of scientific innovation.“

These can be viewed as the NIH feeling that peer reviewers inappropriately delve into details that the NIH wishes they would not consider when arriving at an overall merit rank. In some cases the solution for the NIH is simply not to provide the details in the first place, preventing this sort of quotidian penny-ante complaining over normal-course-of-science stuff.

Yet here they are, about to start right down that same road when it comes to any FOAs that rely upon their Statement of Interest in Diversity [NOT-OD-20-031]. I have been discussing this in the context of PAR-22-181, a funding opportunity for R01s that “Promote Workforce Diversity”, a concept which leans upon their above-referenced Statement of Interest.

When I wrote those, I was uncertain about whether the letter of eligibility from the Signing Official of the applicant institution would remain with the application through peer review. I have now been informed that it will be. So the reviewers will see the grounds on which the University or other applicant institution claims that an R01 awarded to this PI will promote workforce diversity. Which means that their review of the merit could, in larger or smaller ways, explicitly or implicitly, be influenced by their view of the details.

Taking on the “explicit” part, I have always understood that the review of proposals submitted for targeted funding opportunities should be at least in part about how well they meet the intent of the PA, RFA or, now, NOSI. In the case of Special Emphasis Panels, I have been in many experiences where the responsible Program Officers address the reviewers about the intent of the FOA in a meeting held in advance of the review meeting. This is the one case I have experienced where the firewall between Program and Review is lowered. I’ve seen responses to queries about application fit to the FOA as bland as “well if it makes it to review it has been decided it does indeed accord with the intent of the FOA”, to useful comments about the scientific preferences and intentions to rather direct instructions about what sort of prior experience of the PI is “necessary” (e.g., “Department Chair level”) for a good score.

Whether there has been explicit PO instruction or whether it is a regular panel where the reviewers just get the FOA to look at (if they choose, no obligation!), there is no obligation for any reviewer to have any constraint on their review. They might engage in jury nullification where they think the whole FOA is misguided and they refuse to give conventionally fundable scores to anything. They might criticize the grants on grounds that are explicitly requested in the FOA. They might settle on one narrow aspect of the FOA and make this an all-critical pivot. Or, more happily, the reviewer might use the terms in the FOA to try to bulldoze past more conventional Stock Critiques offered by other reviewers.

Implicitly? Well who knows for sure. However it is beyond all rational belief to think this is not a potential problem, even if the reviewer thinks they are behaving one way on supposedly objective and explicit grounds.

Evaluating the details of how the application matches the intent of the FOA is absolutely baked into the NIH review process. Yes, in a highly variable way, just like every other thing about NIH grant review. It is not a formal scoring criterion, it mostly gets smuggled in under Significance, Innovation or Approach. But it is a reality. If you give the reviewers details to work with, those details will affect review.

As you know, Dear Reader, I am much more concerned about the details of “diversity” with respect to this particular FOA. I have not really addressed the fact that it applies not just to Early Stage and New Investigators, but also to “At Risk” Investigators. I may eventually get into this since we know full well that opinions vary sharply about how much sympathy should be extended to various categories of PI who have never had funding and to various categories of PI who have once had funding but do not at present. To remind you, it was a member of the CSR advisory council that opined something like “well if a PI has never gotten a grant after 5 years of trying they don’t deserve to get one ever“. I’m paraphrasing, but this is pretty close to what she said. So there are those who think “NI” shouldn’t be a thing that gets an extra boost. The question of how we view a PI who was once successful but it in a dry spell is a very long discussion. This came up rather pointedly in the short-lived attempt to give ESI awardees another bit of help on their second grant/first renewal (I think this was called “Early Established Investigator” but it is hard to find details, now that this idea was abandoned). So I am limiting my comments to the diversity angle.

What is the “real” intent of the FOA and how should the axes of diversity be prioritized? Well, one take is that all apps that are accepted by the NIH under this FOA should be viewed as equivalent on the “contributes to diversity” angle and all evaluation should rest exclusively on the proposal merits that have nothing to do with PI qualification on the diversity end. I feel certain some reviewers, and even some panels or SROs may take this approach. This then throws any consideration of the axes of diversity to the POs. This has some appeal, after all Program is the place where an entire portfolio can be considered. Maybe they just funded a bunch of proposals with Black PIs by chance or under some other initiative and so therefore anything that comes to them under the PAR-22-181 with a Black PI is a relatively lower contributor to the diversity of their portfolio compared to the ones from PAR-22-181 with disadvantaged background white PIs. Maybe they have had precisely one application from a PI with a disability ever in their Branch or Division, and that shoots it to the top of the diversification list.

The Ginther Gap will be (should be) the elephant in the room. There are clear data from multiple reports about this. The particular qualification that is at the short end of that gap, a Black PI, is echoed in the clear demographic data of 3-5% (PIs) versus 14% (US population). In contrast, the other aspects by which a PI might contribute to diversity under the NOT-20-31 are much more nebulously supported. Sure, there are less specific data, but we don’t have a Ginther / Hoppe type of deep dive into underprivileged upbringing or aspects of disability when it comes to existing grant award. Some reviewers might evaluate proposals under the FOA with an explicit or implicit stance that closing the Ginther Gap is the “real” goal. Others might evaluate with the implicit or explicit stance that under the current societal discussions of Black American after the killing of George Floyd, and the more recent “look busy, Jesus is coming” actions of the NIH regarding the Ginther Gap, that this FOA should be used for something else.

These considerations will play out in terms of scrutinizing the institutional letter attesting to the PI qualification. Or so I am predicting.

Read More...

On verifying eligibility under NIH’s Interest in Diversity

Diversity in Science / Fixing the NIH / NIH / NIH Careerism / NIH funding

The prior post was just supposed to be publicizing the fact that NIH ICs have been joining the PAR-22-181 (NIDA, NIMH and NINDS Research Opportunities for New and “At-Risk” Investigators to Promote Workforce Diversity) originally issued by NIDA, NIMH and NINDS. But of course I got off onto a tangent about the eligibility letter and the whole idea of PIs having to perform their trauma, potentially for their peers if the letter goes along with the application.

Commenter wally underlined my concerns, and I started writing a comment that got out of hand.

It’s important to note that at this point I do not know for sure what happens with the institutional letter. I do not know if it goes along with the application to review, if it is stripped off entirely at the point of receipt and referral, or if it goes only to the Program Officers. [UPDATE 01/10/2023: I have received confirmation that the letter remains with the application as it goes to peer reviewers on study sections.] However, it is very clear to me that the details of each PI’s qualifications have to be included in the application as submitted.

an applicant institution must include a statement explaining how participation of the identified PD/PI(s) would further the goals of the New and At-Risk Investigators to Promote Workforce Diversity Research program, consistent with the Notice of NIH’s Interest in Diversity

At the very least, the PI has to report their qualifications under the Interest in Diversity to their contracts and grants office. Typically this would require a couple of administrators be made aware. Depending on a number of factors potentially a Chair or Dean as well. So already, one is required to put what may be personally sensitive details out there.

I had a larger concern. One of the things that could get sticky is if the University requires the PI to “prove” their status to the satisfaction of the contracts and grants folks. I think it uncontroversial to say there is at least some interest on the part of the applicant institution in not lying to the NIH.

You could see where there would potentially be some tail-covering interest in having the PI qualifications documented.

On criterion A, race and ethnicity, sure for the most part people self-identify, have no reason to fake anything and all would be well. However, we have all heard of cases like Rachel Dolezal. And the political foo fraw involving Elizabeth Warren and her family story that they were part Native American, which maybe turned out not to be true. Would PIs be desperate enough to get in on this PAR to fudge a bit? Would PIs report family histories that turned out to be untrue when put to the 23andme test? Would institutions be best off doing some due diligence? I bet some of them will think this so.

On criterion B, impairment under the ADA, it looks like it is going to need some evaluation of the “substantially limits one or more major life activities of such individual” definition. Is an institution going to take an assertion that one suffers from crippling depression or fibromyalgia at face value?

Then we get to criterion C. Which is an absolute minefield.
Being unhomed is defined clearly but who would have retained documentation of, say, a childhood experience of homelessness decades later. Remember, part of this PAR addresses PIs of any age/career status who lose funding. We are talking the entire span of ages that are active in research, yes people in their 70s and 80s are still applying for grants on the regular. Could someone prove it, if this was demanded by Contracts and Grants before signing off on the submission?

Foster care criteria seem to depend on being “in” the foster care “system”. Foster care in the US appears to date to 1935 in the Social Security Act, but the NOT points to a site specifying the Title IV-E amendment so it is unclear what “system” means. I am not readily seeing where there is a date on the Title IV-E but I saw an indication of a Title IV-D amendment in 1975 so presumably after that. Precise limits on when a previously fostered PI would be eligible are unclear. Even given this question is answered, one wonders if all PIs who were once in foster care have retained documentation of that.

Eligible for free or reduced lunch for two years during primary and secondary education? Who on earth would retain that documentation? Who has records on their parents’ income for the critical years?

First generation bachelor’s degree requires proving a negative. How is one going to document this? Affidavit from any living parents or close relatives?

The Pell grant was created in 1965 and one doubts whether and how anyone would go back and document their Pell grant. I think more recent folks can go into the FAFSA system and look themselves up but I am unclear how far back that would go. Furthermore, since you only had to be eligible for a Pell grant, and didn’t have to actually take one, how would you document that? How would you know what the standards were for “exceptional financial need” for each year in the past?

WIC support? It dates back to 1972. Again, not clear to me that one would have that documentation if one was the child.

What about the “grew up in” rural or low-income/health-shortage areas? Has this eligibility changed over decades or is it fixed? How many years equals “grew up in”? Is it necessarily the case that a PI will have any documentation of where they lived? Particularly if they are generally in this category and may have experienced a mobile and undocumented life featuring many address changes and little retention of records? I barely have anything testifying to the address I resided in for the vast majority of my pre-adult life.

In the prior post I mentioned my concern about grant review turning into some sort of Oppression Olympics where the reviewers were contaminated by their explicit or implicit assumptions about whether a PI is deserving under these criteria.

“Sure this PI is Black, and lived as a child in a rural ZIP code, but her parents are a physician and a lawyer. She was sent to the best private schools and her Uncle Bob who we know as good old Professor Jones had her interning summers in the lab through high school“.

I have been in more than one discussion similar to this in other diversity in academia scenarios. It is definitely a thing. Many of you are probably nodding along that that is a perfectly good line of thought on why NOT to give such a person any boost. Change the above example to a white man, who qualifies under Criterion C on ZIP code grounds and, oh I dunno, happened to be adopted out of foster care at the age of 6 months, and you can bet your boots I am going to have a huge bias that this is NOT a person that needs any sort of boost…even though they fully qualify to equal extent under the PAR and the NOT.

If this stuff were easy, it wouldn’t be worth talking about.

N.b. As in the olden dayes of blogging, my response to a comment got out of hand and now deserves it’s own post.

Read More...

ICs are trickling in to join the NIH PAR for workplace diversity

Fixing the NIH / Grantsmanship / NIH / NIH Careerism / NIH funding

In June of 2022 PAR-22-181 (NIDA, NIMH and NINDS Research Opportunities for New and “At-Risk” Investigators to Promote Workforce Diversity) informed us that NIDA, NIMH and NINDS would like to solicit R01 applications that proposed research which would “support New Investigators and At-Risk Investigators from diverse backgrounds, including those from groups underrepresented in the health-related sciences“. The diverse backgrounds are defined in NOT-OD-20-031 (Notice of NIH’s Interest in Diversity) and include A) underrepresented racial and ethnic groups, B) investigators with disabilities, C) investigators with socioeconomically disadvantaged backgrounds and D) women from the above backgrounds. (No, I am not clear on why D needs to be further emphasized, and the following sentences sound almost like they mean any women.)

The PAR came with one month earlier receipt dates (Sep 8 instead of the Oct 6 for new R01 and Oct 6 instead of Nov 6 for revisions, in 2022), an emphasis that proposals had to meet the usual research goals of the issuing ICs, and statements of intent to commit funds in the usual weaselese (e.g. “NIDA intends to commit up to $5 million per fiscal year, 12-15 awards, dependent on award amounts“). No guarantee of set aside funds. A bit of questioning around revealed that despite the PAR designation, these would be reviewed in standing study sections.

I am definitely down with something this specific directed at the Ginther gap. It provides cover for what I’ve been saying they should do for over a decade now- fix the gap immediately via exception pay decision making.

The PAR was joined by NINR (August 4), NCCIH (Sept 7), NIAAA (Sept 14) and NEI (Sept 14), in each case adding their abbreviation to the PAR title. NIBIB joined Jan 5, 2023, and at this point NIH apparently decided to stop using the IC indicators in the title. For unknown reasons, NIAID decided to issue their own version, PAR-22-241, Sept 21. NIDCD did theirs by RFA (RFA-DC-23-001 issued 4/5/22) but limited it to ESI and New Investigators only. I predict we are now running downhill and most if not all ICs will be participating in this initiative eventually. This is good news for those PIs who tend to seek funding from ICs other than the original three. A reminder for those of you who do not yet see your IC on the list to stay tuned.

You will recall at this point that NINDS tried something like this via NOSI in May of 2021. It was joined by NIDA and NIAAA and then no other ICs. Crickets. Increasingly suspicious crickets. In October 2021 the Office of the Director rescinded the NOSI. Reporting on the matter had the NIH claiming that it was duplicative with the above referenced statement on diversity and furthermore it was confusing. Some implication from NIH that it was illegal to award grants on the basis of skin tone or some such. Actually, Bernard and Lauer wrote “may have led to an impression that by linking demographic characteristics to grant proposals, applications supporting scientists from underrepresented groups would be automatically prioritized for funding.” I cannot for the life of me figure out what the difference is with this PAR that differentiates it on these issues from the prior NOSI.

The ONLY difference that is obvious is that Francis Collins retired as the Director of the NIH at the end of 2021.

There does not appear to be much that is unusual in the PAR about the preparation and review of these R01s. The one added item is a letter signed by the Authorized Organization Representative certifying that the PI meets the eligibility criteria and stating how the “participation of the identified PD/PI(s) would further the goals of the New and At-Risk Investigators to Promote Workforce Diversity Research program, consistent with the Notice of NIH’s Interest in Diversity“. This information is to be evaluated “at the time of application“, presumably at receipt and referral. It is unclear from this whether this letter goes along with the application through the review process, whether it is only retained by Program for their reference, or whether it is solely for receipt/acceptance purposes. One presumes at the very least it will go to Program. [UPDATE 01/10/2023: I have received confirmation that the letter remains with the application as it goes to peer reviewers on study sections.]

I admit to being slightly perturbed by the idea that the eligibility letter would be included with the application being reviewed by study section. It is bad enough that these are being reviewed in regular standing study sections, for structural reasons that are going to have to wait for another post. But putting some of these criteria in front of peers in the PI’s field seems…invasive. Not the racial, gender or ethnicity criteria, although I can imagine there are those who might prefer their peers do not know their race or ethnicity. I am most bothered by the disability and the disadvantaged background criteria. The latter include being homeless, in foster care, being eligible for free or reduced lunch, receiving WIC benefits, being eligible for Pell grants. For some reason you have to have two of the poverty criteria checked…I am entirely unclear on how that defines disadvantage- why not three? why not four? anyway.

It all seems sort of invasive to me.

Beyond the personal humiliation factor, we have a focal version of the problem that is inevitable merely from the FOA being listed on the application. It is also a version of the broader issue of dealing with representation, equity and bias in the academy, namely the argument by anecdote and proven, specific, individual harm.

I’ve covered this before on this blog but it boils down to those that feel that any affirmative action has to only apply to individuals who it can be proven have specifically and individually suffered the worst of the worst hinderances, all of which are more severe than the factors suffered by the most disadvantaged of the majoritarian population of reference. It holds that we only extend help to individuals who have not been at all successful. If an individual has experienced any measurable success, see, this proves that they are not affected* by any biases we are trying to fix.

This is the kind of headwind at peer review that this PAR is facing. The pre-potent tendency of academics to want to litigate bias and disparity on an individual basis and have it proved to them beyond all doubt that the person getting a leg up has faced more disadvantage than any person who is not receiving a leg up.

I would say that of all the NIH diversity categories, the one most likely to let people “pass” is the socioeconomic disparity one. However, another way to look at the phenomenon of passing is through the lens of the “well she clearly wasn’t held back in any way, look at how well she’s done and nobody has any idea she grew up poor” litigator. It will be counter productive to the attempt to fix the Ginther gap to have reviewers fighting this out, overtly or covertly, between or within reviewer.

I think it’s a mistake to do affirmative action as the Oppression Olympics. Both on this individual basis, to try to glean out who is “most” deserving of help, but also in the more general sense that a PI who checks, say, all 4 of the Diversity letters is more deserving than the PI who only checks one. I will admit that this is one possible counter to my preference that these should be reviewed in special emphasis panels. Putting them all in competition with each other could sharpen this kind of disparity comparison. OTOH it could also make sure that everyone is more or less on board with the idea that each PI identity is equal, once accepted by NIH as qualifying under the PAR.

*Despite the fact that Ginther and Hoppe showed persisting disparity of grant award, no matter whether a PI worked in the most fundable topics, had seniority, prior grants, prior study section experience, etc, etc.

Read More...