If I stroke out today it is all the fault of MorganPhD.
Jeffery Mervis continues with coverage of the NIH review situation as it pertains to the disparity for African-American PIs identified in 2011 (that’s five years and fifteen funding rounds ago, folks) by the Ginther report.
The main focus for this week is on the Early Career Reviewer program. As you will recall, this blog has advocated continually and consistently for the participation of more junior PIs on grant review panels.
The ECR program was created explicitly to deal with underrepresented groups. However, what happened is that there was immediate opposition which insisted that the ECR program had to be open to all junior faculty/applicants, regardless of representation in the NIH game.
One-quarter of researchers in ECR’s first cohort were from minority groups, he notes. “But as we’ve gone along, there are fewer underrepresented minorities coming into the pool.”
…
Minorities comprise only 13% of the roughly 5100 researchers accepted into the program (6% African-American and 7% Hispanic), a percentage that roughly matches their current representation on study sections.
Ok, but how have the ECR participants fared?
[Nakamura] said ECR alumni have been more than twice as successful as the typical new investigator in winning an R01 grant.
NIIIIIICE. Except they didn’t flog the data as hard as one might hope. This is against the entire NI (or ESI?) population.
The pool of successful ECR alumni includes those who revised their application, sometimes more than once, after getting feedback on a declined proposal. That extra step greatly improves the odds of winning a grant. In contrast, the researchers in the comparison group hadn’t gone through the resubmission process.
Not sure if this really means “hadn’t” or “hadn’t necessarily“. The latter makes more sense if they are just comparing to aggregate stats. CSR data miners would have had to work harder to get this isolated to those who hadn’t revised yet, and I suspect if they had gone to that effort, they could have presented the ESIs who had at least one revision under their belt. But what about the underrepresented group of PIs that are the focus of all this effort?
It’s also hard to interpret the fact that 18% of the successful ECRs were underrepresented minorities because NIH did not report the fraction of minorities among ECR alumni applicants. So it is not clear whether African-Americans participating in the program did any better than the cohort as a whole—suggesting that the program might begin to close the racial gap—or better than a comparable group of minority scientists who were not ECR alumni.
SERIOUSLY Richard Nakamura? You just didn’t happen to request your data miners do the most important analysis? How is this even possible?
How on earth can you not be keeping track of applicants to ECR, direct requests from SROs, response rate and subsequent grant and reviewing behavior? It is almost as if you want to look like you are doing something but have no interest in it being informative or in generating actionable intelligence.
Moving along, we get a further insight into Richard Nakamura and his position in this situation.
Nakamura worries that asking minority scientists to play a bigger role in NIH’s grantsmaking process could distract them from building up their lab, finding stable funding, and earning tenure. Serving on a study section, he says, means that “those individuals will have less time to write applications. So we need to strike the right balance.”
Paternalistic nonsense. The same thing that Scarpa tried to use to justify his purge of Assistant Professors from study sections. My answer is the same. Let them decide. For themselves. Assistant Professors and underrepresented PIs can decide for themselves if they are ready and able to take up a review opportunity when asked. Don’t decide, paternalistically, that you know best and will refrain from asking for their own good, Director Nakamura!
Fascinatingly, Mervis secured an opinion that echoes this. So Nakamura will surely be reading it:
Riggs, the only African-American in his department, thinks the program is too brief to help minority scientists truly become part of the mainstream, and may even exacerbate their sense of being marginalized.
“After I sat on the panel, I realized there was a real network that exists, and I wasn’t part of that network,” he says. “My comments as a reviewer weren’t taken as seriously. And the people who serve on these panels get really nervous about having people … that they don’t know, or who they think are not qualified, or who are not part of the establishment.”
If NIH “wants this to be real,” Riggs suggests having early-career researchers “serve as an ECR and then call them back in 2 years and have them serve a full cycle. I would have loved to do that.”
The person in the best position to decide what is good or bad for his or her career is the investigator themself.
This comment also speaks to my objection to the ECR as a baby-intro version of peer review. It isn’t necessary. I first participated on study section in my Asst Prof years as a regular ad hoc with a load of about six grants, iirc. Might have been 2 less than the experienced folks had but it was not a baby-trainee experience in the least. I was treated as a new reviewer, but that was about the extent of it. I thought I was taken seriously and did not feel patronized.
__
Related Reading:
Toni Scarpa to leave CSR
More on one Scientific Society’s Response to the Scarpa Solicitation
Your Grant In Review: Junior Reviewers Are Too Focused on Details
If they’re serious about addressing Ginther the single best thing the NIH can do is get more minorities on study section. Guess they’re not really serious.
I am happy that they are trying but I’d like to see more effort on smarter approaches and less excuse making and distracting make-work. This quoted individual makes the point. Bring these folks in (there’s what, a few hundred at most) over time and ASK THEM WHAT WOULD HELP people like them be successful.
Is the ECR pool selected from applicants or do they include anyone that asks? If they are selected, does this 2x success probability increase hold against matched cohorts? I guess that I’m asking who were the control groups; if they were the general NI population (as the paper says), I’m interested but not too impressed. If they are comparing ECR pool members that participated in study sections vs those that didn’t, that would be more impressive.
I wish they would actually ask me to review. Been approved on the ECR program for a while now. Crickets ever since. Bit insulting that they think they know what’s best for us without asking us. Paternalistic indeed.
I don’t even have a problem with being babied into it through the ECR program as much as DM. I just want to be involved somehow.
I’ve been registered on the program for 3 years. I have 100+ pubs at good journals (including Annals, PLOS Med, JAMA IM) and am an editor at a big five journal. I’ve bugged my PO, both in writing and informally at conferences, about putting me forward as a suggested reviewer. Crickets.
zOMG….do the people who are setting up these programs even think about the metrics of success? Put a panel together of the ESI and ESI minority participants and let them answer the questions. Oh, and if you are so worried about their time and labs and you value them so much, PAY THEM to be part of your experiment. Hey….here’s a 25K supplement to offset your time in participating in this class goes a long way to make folks feel valued.
A Tasso
Contact the SRA for the study section that has reviewed your grants and/or sends grants to your PO. That is the person (along with the IRG Chair) who actually contacts folks about serving on study section. If they can’t put you on next cycle, ask about filling in if someone drops out or ask them when they will be looking for Reviewers next. Then put it on your calendar.
It took me a long time to break through and I did random ad hocs for other grants (foundations, other countries, the DOD) and I think the ice breaker was having someone on a panel recommend me when they were rotating off.
Oh and to the main point: I had no idea ECR was created to address Ginther. Is this well publicized? I think maybe it is not.
What an obvious way to boost minority success! I just can’t understand why it isn’t being used more effectively and why reviewers aren’t asked to serve longer terms. What is the downside? Am I missing something? ECRs seem to be assigned R3 or R4 roles, so it’s not like the inexperience will impair review. On the contrary, in my experience they are some of the most concientious reviewers. This seems like such an obvious and easy win. What am I missing??? Seriously can someone explain this?
Noah, I first got wind of this around June 2011 and there was NIH murmuring by July 2011. http://drugmonkey.scientopia.org/2011/07/21/nih-to-invite-early-career-reviewers-to-study-section-panels/
Presumably the brass knew all about the Ginther findings long before they were published in Aug 2011.
My remarks about the intent come from conversations with SROs in 2011 who gave me the low down on the initial motivation and intent for ECR recruiting… and the push back within CSR that ended up opening up the ECR to overrepresented individuals as well. I’m sure there is no way to confirm that, however.
Nakamura’s comments in the recent Mervis piece seem to be the usual “we can’t find enough” excuse-making. Notice that he does not address whether black participants in ECR are being brought into the system as regular ad-hocs and as appointed members of study sections. The quote from Dr. Riggs would seem to indicate at least one willing person has not been asked. Leaking reviewer pipeline!!!!
I’ve bugged my PO, both in writing and informally at conferences, about putting me forward as a suggested reviewer.
Remember that while this can be a route to appointment (it was for YHN), there is a firewall between Program and Review that the SROs often guard like the Night’s Watch.
The firewall is related to the idea that POs should not be able to influence the outcome for their favorite (or disfavored) PIs by influencing panel composition or application assignment, in case anyone is wondering. This is a very good thing.
So there is sometimes friction between POs and SROs. I would be entirely unsurprised if in some cases a given SRO totally ignores any suggestions given by POs (generally) or even one specific PO that they think gets too pushy.
So as Noah suggested, your primary target is the SRO of your study sections of interest. And, as also suggested, try to get people on the panel to put in a good word for you. I have very little doubt that this also happened in my case, the SRO in question was the type of person that would have checked with panel members after receiving a CV / recommendation from a PO.
I sat as an ECR member last year (was contacted directly by an SRO, and they offered to let me do either a full load or set me up in the ECR). Given that I had been an Assist. Prof. for all of two months when they contacted me, I chose the ECR route. Good experience, helped with my grant writing no-end.
We had a survey a couple of months back from CSR about our thoughts on the value of the ECR program, and what would improve it. I suggested guaranteeing a couple of cycles (rather than saying two max, and then not hearing anything again after the first one – unless I was just really poor…) on the study section would be better than randomly being brought in. Or at least bringing people back to ad hoc for any resubmission of a grant you reviewed the A0, so that you could see the whole process to completion.
This ECR thing is stupid. They should just bring assistant professors back on the study sections. ‘Nuff said.
(My experience was much like DM’s, but then we are of the pre-ECR generation. I was treated as a new reviewer, I was taken seriously, worked hard, schooled when I said something wrong, and learned a lot, but I did not feel patronized.)
For the record, I reviewed my first NSF grants as a postdoc, for NASA as a second year professor and was on an NIH study section by my third year. There is nothing in any of the rules of either program that precludes any of this. It’s all internal decisions. They could bring the kids back with a stroke of the pen.
And to be clear, Asst Profs were only about 10% of all *reviewers* (not reviews given newbie / ad hoc loads) when Scarpa went on his jihad against them. CSR used to publish interesting info on grant review stats including the sequential fate of revisions. Haven’t seen anything like that data book report since 2010, maybe 2009.
https://drugmonkey.wordpress.com/2007/09/10/preferential-funding-for-first-submissions-of-nih-grants/ This post referenced the document but the link doesn’t work anymore.
@drugmonkey and @NoahStroyer, very helpful advice. Thank you!
DM,
I am not too sure that Nakamura’s position on the issues is his own one or he is just under pressure from Collins and/or external/internal groups and, somehow, forced to express their positions. When, in the past, you talked about Scarpa’s lies, who knows whether the lies were his or those of the higher-ups in power and/or connections.
In the end, how much is left of the big peer review overhaul initiated by Zerhouni and all that transparency, funding the best, taking care of young investigators as the future of the scientific enterprise ?…. It seems that some of the government organizations, namely NIH, are most about PR and narcissistic self-contemplation….. mah.
https://youtu.be/gWXVZlrLa6E
I understand that there are certain job requirements that may or may not reflect his personal views. Does it matter though? I’ve met the guy once. I have no investment in him as anything other than CSR Director.
One of the big differences between the ECR experience (as I’ve watched junior professors participate) and my experience as a new reviewer is that the ECRs only get one maybe two easy grants to review. They seem to be there more for the experience of being AT rather than ON study section.
When I was a new reviewer and given a full set of grants to review, I had the opportunity to see a range of grants, good and bad, well-written and not, and to see how my view on grants meshed with study section’s. The two most important lessons that I learned (which I still remember vividly) are (1) that getting the reviews right is hard [so make it easy for the reviewer] and (2) that the things I came in ready to trash or defend (expecting to have to disagree with others) were the same issues that the other reviewers found. Point 1 changed the way I wrote grants and point 2 changed the way I felt about the things study section complained about.
I don’t know that I would have seen either of these as an ECR.
I agree with qaz- was unbelievable privilege to be full reviewer vs junior- however, even if only assigned to fewer grants as ECR, should take opportunity to read as many as possible, and as many of reviews as possible. That mitigates somewhat the experience from being junior
should take opportunity to read as many as possible, and as many of reviews as possible.
Absolutely. Great advice. For anyone new to study section. I read everything in my first study section round out of stark fear of looking like an idiot who had no idea what was going on. It was really helpful for my comfort level during the meeting itself. Beyond that, I thought it was exceptionally good preparation for me to be able to see the dynamics and how things evolved from a preliminary set of independent reviews to a set of discussion points. You can see who is modulating their position and who is holding fast. Begin to appreciate what sorts of arguments result in agreeing nods around the table and which ones lead to confusion and probing.
I second what DM and PP are saying. In addition to the reviewing advice, DM gives (which I wholeheartedly agree with), I would point out that reading everything those first few times at study section was an amazing opportunity to see what tactics worked as a grant-writer. What things mattered? What didn’t? I treated it as an experiment – here was a large set of grant proposals and I was actually going to be able to see which ones led to good scores and which didn’t.
SERIOUSLY Richard Nakamura? You just didn’t happen to request your data miners do the most important analysis? How is this even possible?
How on earth can you not be keeping track of applicants to ECR, direct requests from SROs, response rate and subsequent grant and reviewing behavior? It is almost as if you want to look like you are doing something but have no interest in it being informative or in generating actionable intelligence.
Drugmonkey,
Hate to sound like a broken record, but they definitely want to look like they’re doing something without generating actionable intelligence. The audience for this (legislators, diversity advocates without research backgrounds, etc.) are going to take his “analysis” at face value.
Thank you for continuing to push on this and point out the obvious weaknesses. I think the only way we will see meaningful action on this issue, and related issues, is if NIH is forced to address pointed critiques like yours. Particularly, if those critiques contain logical suggestions for meaningful action.
NIH may want to obfuscate, but they do not want to appear incompetent.
I am perfectly willing to play broken record on this topic, and many others. Repetition is the only way to get any traction at all.