Archive for the 'Fixing the NIH' category

Collins announces NRSA salaries will meet Obama's overtime rule

In a piece on HuffPo, NIH Director Francis Collins announces the NIH response to Obama's new rules on overtime for salaried employees. Collins:

Under the new rule, which was informed by 270,000 public comments, the threshold will be increased to $47,476 effective December 1, 2016. ....In response to the proposed FLSA revisions, NIH will increase the awards for postdoctoral NRSA recipients to levels above the threshold.

"levels". Meaning, presumably the entire scale will start around $47.5K and move upward with years of postdoctoral experience, as the NRSA scale usually does.

What about the larger population of postdocs that are paid from non-NRSA funds, Dr. Collins?

..we recognize that research institutions that employ postdocs will need to readjust the salaries they pay to postdocs that are supported through other means, including other types of NIH research grants. While supporting the increased salaries will no doubt present financial challenges to NIH and the rest of the U.S. biomedical research enterprise, we plan to work closely with leaders in the postdoc and research communities to find creative solutions to ensure a smooth transition.

Imprecise and highly disappointing when it comes to the postdocs supported on "other types of NIH research grants". This would have been a great opportunity to state that the NIH expects any postdocs paid from RPGs to be on the NRSA scale, wouldn't it? Most postdocs are supported on NIH grants. This Rock Talk post shows in FY2009 something like 11,000 basic biomed postdocs on Federal research grants and only 1,000 on Federal fellowships and training grants (and ~7,800 on nonFederal support). So Francis Collins is talking the happy talk about 10% of the postdocs who work for him and throwing 90% into the storm.

The OER head, Michael Lauer, has a few more interesting points on the Open Mike blog.

Institutions that employ postdocs through non-NRSA support can choose how to follow the new rule. They may choose to carefully track their postdocs’ hours and pay overtime. Or, keeping with the fact that biomedical research – as in many professional and scientific careers – does not fit into neatly defined hourly shifts, institutions can choose to raise salaries to the new FLSA salary threshold or above it, if they do not yet pay postdocs at or above that level.

This would imply that Dr. Collins' supposed plan to "work closely with" and "ensure a smooth transition" is more realistically interpreted as "hey, good luck with the new Obama regs, dudes".

Before we get at it in the comments, a few lead off points from me:

The current NRSA scale pays 0 year postdocs $43,692 so in December the brand new postdoc will see a $4,000 raise, roughly. There is currently something on the order of $1,800 increases for each successive year of experience, this estimate is close enough for discussion purposes. If this yearly raise interval is maintained we can expect to see that same $4,000 pay rise applied to every salary level. Remember to apply your local benefits rate for the cost to a grant, if you are paying your postdocs at NRSA scale from RPG funds. Could turn this into a $5,000-$6,000 cost to the grant.

Postdocs getting paid more is great. Everyone in science should be paid more but there is something specific here. Postdocs frequently work more than 40 h per week for their salaried positions. This is right down the middle of the intent of Obama's change for the overtime rules. He is right on this. Period.

With that said, there is a very real disconnect here between the need to pay postdocs more and the business model which funds them. As mentioned above, 90% of Federally funded postdocs are supported by research grants, and 10% on fellowships or traineeships. (A population almost 8 times as large as the latter are supported by nonFederal funds- the percentage of these working on Federal research projects is likely to be substantial.) A grant may have one or two postdocs on it so adding another $5,000-$10,000 per year isn't trivial. Especially since the research grant budgets are constrained in a number of ways.

First, in time. We propose grants in a maximum of 5 year intervals but often the budget is designed one or two years prior to funding. These grant budgets are not supplemented in the middle of a competitively-awarded interval just because NRSA salary levels are increased. Given the way NRSA rises have been coming down randomly over the years, it is already the case that budgets are stretched. Despite what people seem to think (including at NIH), we PIs do not pad the heck out of our proposed research budgets. We can't. Our peers would recognize it on review and ding us accordingly.

Second, grants are constrained by the modular budgeting process which limits direct costs to $250,000 per year. This a soft and nebulous limit which depends on the culture of grant design, review and award. Formally speaking, one can choose a traditional budget process at any time if one needs to request funds in excess of $250,000 per year. Practically speaking, a lot of people choose to use the modular budget process. For reasons. The purchasing power has been declining for 15 years and there is no sign of a change in the expectations for per-grant scientific output.

Third, grant budgets are often limited by reductions to the requested budget that are imposed by the NIH. This can be levied upon original funding of the award or upon the award of each of the annual non-competing intervals of funding. These can often range to 10%, for argument's sake let's keep that $25,000 figure in mind when assessing the impact of such a reduction on paying a salary for a staff member, such as a postdoc. Point being, it's a big fraction of a salary. This new postdoc policy isn't going to result in fewer cuts or shallower cuts. Believe me.

I will be watching the way that local Universities choose to deal with the new policy with curiosity. I think we all see that trying to limit postdocs to 40 h a week of work so as to avoid raising the base salary is a ridiculous plan*. The other competitive motivations will continue to drive some postdocs to work more. This will put Universities (and PIs) in the extremely distasteful position of creating this elaborate fiction about working hours.

One potential upside for the good PI, who is already maintaining postdocs at NRSA levels even when funded from the RPG, is that it will force the bad PIs into line. This should narrow the competitive disadvantage that comes with trying to treat your postdocs well.

Final point. This will take away jobs. Fewer postdocs will be hired. Whether this is good or bad....well, opinions vary. But the math is unmistakable.

[UPDATE: The modular budget grant limit of $250,000 was established for R01s in FY2000 and the NRSA 0 year postdoc salary in FY2000 was $26,916. This is 10.8% of the direct costs of a full modular R01. In FY2017 when this new NRSA adjustment, the 0 year postdoc will be 19% of the direct costs of a full modular R01. In short the postdoc is now 76% more expensive than the postdoc was in FY2000.]
__
*It is, however, a failed opportunity to attempt to normalize academic science's working conditions. I see no reason we shouldn't take a stab at enforcing a 40 h work week in academic science, personally. Particularly for the grad student and post-doc labor force who are realistically not very different from the technicians who do, btw, enjoy most labor protections.

117 responses so far

Entitled to a Grant: What is fair?

May 02 2016 Published by under Fixing the NIH, NIH, NIH Careerism, NIH funding

I am genuinely curious as to how you people see this. Is there any particular difference between people arguing that that acquisition of the first major grant award should be protected versus multiple award and the people arguing that acquisition of the first and third concurrent awards should be on an equal footing?

If we agree that NIH (or NSF or CIHR or whatever) grants are competitively awarded, it follows that nobody is actually entitled to a grant. And as far as I am aware, all major funding agencies operate in a way that states and demonstrates the truth of this statement.

Specifically in the NIH system, it is possible for the NIH officials to choose not to fund a grant proposal that gets the best possible score and glowing reviews during peer review. Heck, this could happen repeatedly for approximately the same project and the NIH could still choose not to fund it.

Nobody is entitled to a grant from the NIH. Nobody.

It is also the case that the NIH works very hard to ensure a certain amount of equal representation in their awarded grants. By geography (State and Congressional district), by PI characteristics of sex and prior NIH PIness, by topic domain (see the 28 ICs) or subdomain (see Division, Branches of the ICs. also RFAs), etc.

Does a lean to prioritize the award of a grant to those with no other major NIH support (and we're not just talking the newcomers- plenty of well-experienced folks are getting special treatment because they have run out of other NIH grant support) have a justification?

Does the following graph, posted by Sally Rockey, the previous head of Extramural Research at the NIH make a difference?

This shows the percentage of all PIs in the NIH system for Fiscal Years 1986, 1998, 2004 (end of doubling) and 2009 who serve as PI on 1-8 Research Project Grants. In the latest data, 72.3% had only one R01 and 93% had 1 or 2 concurrent RPGs. There were 5.4% of the PIs that held 3 grants and 1.2% that held 4 grants. I just don't see where shifting the 7% of 3+ concurrent awards into the 1-2 grant population is going to budge the needle on the perceived grant chances of those without any major NIH award. Yes, obviously there will be some folks funded who would otherwise not have been. Obviously. But if this is put through in a systematic way*, the first thing the current 3+ grant holders are going to do is stop putting in modular grants and max out their allowable 2 at $499,999 direct costs. Maybe some will even get Program permission to breach the $500,000 DC / y threshold. So there won't be a direct shift of 7% of grants back into the 1-2 grant PI population.

There has been a small trend for PIs holding more grants concurrently from 1986 to the late naughties but this is undoubtedly down to the decreasing purchasing power of the modular-budget grant.

BRDPI.
I"ve taken their table of yearly adjustments and used those to calculate the increase necessary to keep pace with inflation (black bars) and the decrement in purchasing power (red bars). The starting point was the 2001 fiscal year (and the BRDPI spreadsheet is older so the 2011 BRDPI adjustment is predicted, rather than actual). As you can see, a full modular $250,000 year in 2011 has 69% of the purchasing power of that same award in 2001.

Without that factor, I'd say the relative proportions of PIs holding 1, 2, 3 etc grants would be even more similar across time than it already is.

So I come back to my original question. What is fair? What policies should the NIH or any broad governmental funding body adopt when it comes to distributing the grant wealth across laboratories? On what basis should they do this?

Fairness? Diversity of grant effort? PR/optics?

__
*and let us face it, it is hugely unlikely that the entire NIH will put through a 2-grant cap without any exceptions. Even with considerable force and authority behind it, any such initiative is likely to be only partially successful in preventing 3+ grant PIs.

DISCLAIMER: As always, I am an interested party in these discussions. My lab's grant fortunes are affected by broad sweeping policies that the NIH might choose to adopt or fail to adopt. You should always read my comments about the NIH grant game with this in mind.

106 responses so far

NIH Grant Lottery

Apr 18 2016 Published by under Fixing the NIH, NIH

Fang and Casadevall have a new piece up that advocates turning NIH grant selection into a modified lottery. There is a lot of the usual dissection of the flaws of the NIH grant funding system here, dressed up as "the Case for" but really they don't actually make any specific or compelling argument, beyond "it's broken, here's our RealSolution".

we instead suggest a two-stage system in which (i) meritorious applications are identified by peer review and (ii) funding decisions are made on the basis of a computer-generated lottery . The size of the meritorious pool could be adjusted according to the payline. For example, if the payline is 10%, then the size of the meritorious pool might be expected to include the top 20 to 30% of applications identified by peer review.

They envision eliminating the face to face discussion to arrive at the qualified pool of applications:

Critiques would be issued only for grants that are considered nonmeritorious, eliminating the need for face-to-face study section meetings to argue over rankings,

Whoa back up. Under current NIH review, critiques are not a result of the face-to-face meeting. This is not the "need" for meeting to discuss the applications. They are misguided in a very severe and fundamental way about this. Discussion serves, ideally, to calibrate individual review, to catch errors, to harmonize disparate opinions, to refine the scoring....but in the majority of cases the written critiques are not changed a whole lot by the process and the resume of the discussion is a minor outcome.

Still, this is a minor point of my concern with their argument.

Let us turn to the juxtaposition of

New investigators could compete in a separate lottery with a higher payline to ensure that a specific portion of funding is dedicated to this group or could be given increased representation in the regular lottery to improve their chances of funding.

with

we emphasize that the primary advantage of a modified lottery would be to make the system fairer by eliminating sources of bias. The proposed system should improve research workforce diversity, as any female or underrepresented minority applicant who submits a meritorious application will have an equal chance of being awarded funding.

Huh? If this lottery is going to magically eliminate bias against female or URM applicants, why is it going to fail to eliminate bias against new investigators? I smell a disingenuous appeal to fairness for the traditionally disadvantaged as a cynical ploy to get people on board with their lottery plan. The comment about new investigators shows that they know full well it will not actually address review bias.

Their plan uses a cutoff. 20%, 30%...something. No matter what that cutoff line is, reviewers will know something about where it lies. And they will review/score grants accordingly. Just Zerhouni noted that when news of special ESI paylines got around, study sections immediately started giving ESI applications even worse scores. If there is a bias today that pushes new investigator, woman or URM PI's applications outside of the funding, there will be a bias tomorrow that keeps them disproportionately outside of the Fang/Casadevall lottery pool.

There is a part of their plan that I am really unclear on and it is critical to the intended outcome.

Applications that are not chosen would become eligible for the next drawing in 4 months, but individual researchers would be permitted to enter only one application per drawing, which would reduce the need to revise currently meritorious applications that are not funded and free scientists to do more research instead of rewriting grant applications.

This sounds suspiciously similar to a plan that I advanced some time ago. This post from 2008 was mostly responding to the revision-queuing behavior of study sections.

So this brings me back to my usual proposal of which I am increasingly fond. The ICs should set a "desired" funding target consistent with their historical performance, say 24% of applications, for each Council round. When they do not have enough budget to cover this many applications in a given round, they should roll the applications that missed the cut into the next round. Then starting the next Council round they should apportion some fraction of their grant pickups to the applications from the prior rounds that were sufficiently meritorious from a historical perspective. Perhaps half roll-over and half from the current round of submissions. That way, there would still be some room for really outstanding -01 apps to shoulder their way into funding.

The great part is that essentially nothing would change. The A2 app that is funded is not going to result in scientific conduct that differs in any substantial way from the science that would have resulted from the A1/15%ile app being funded. New apps will not be any more disadvantaged by sharing the funding pie with prior rounds than they currently are facing revision-status-bias at the point of study section review.

What I am unclear on in the Fang/Casadevall proposal is the limit to one application "per drawing". Is this per council round per IC? Per study section per Council round per IC? NIH-wide? Would the PI be able to stack up potentially-meritorious apps that go unfunded so that the get considered in series across many successive rounds of lotteries?

These questions address their underlying assumption that a lottery is "fair". It boils down to the question of whether everyone is equally able to buy the same number of lottery tickets.

The authors also have to let in quite reasonable exceptions:

Furthermore, we note that program officers could still use selective pay mechanisms to fund individuals who consistently make the lottery but fail to receive funding or in the unlikely instance that important fields become underfunded due to the vagaries of luck.

So how is this any different from what we have now? Program Officers are already trusted to right the wrongs of the tyranny of peer review. Arguing for this lottery system implies that you think that PO flexibility on exception funding is either insufficient or part of the problem. So why let it back into the scheme?

Next, the authors stumble with a naked assertion

The proposed system would treat new and competing renewal applications in the same manner. Historically, competing applications have enjoyed higher success rates than new applications, for reasons including that these applications are from established investigators with a track record of productivity. However, we find no compelling reason to justify supporting established programs over new programs.

that is highly personal. I find many compelling reasons to justify supporting established programs. And many compelling reasons not to do so preferentially. And many compelling reasons to demand a higher standard, or to ban them entirely. I suspect many of the participants in the NIH system also favor one or the other of the different viewpoints on this issue. What I find to be unconvincing is nakedly asserting this "we find no compelling reason" as if there is not any reasonable discussion space on the issue. There most assuredly is.

Finally, the authors appeal to a historical example with is laughably bad for their argument:

we note that lotteries are already used by society to make difficult decisions. Historically, a lottery was used in the draft for service in the armed forces...If lotteries could be used to select those who served in Vietnam, they can certainly be used to choose proposals for funding.

As anyone who pays even the slightest attention realizes, the Vietnam era selective service lottery in the US was hugely biased and subverted by the better-off and more-powerful to keep their offspring safe. A higher burden was borne by the children of the lower classes, the unconnected and, as if we need to say it, ethnic minorities. Referring to this example may not be the best argument for your case, guys.

45 responses so far

Bias at work

A piece in Vox summarizes a study from Nextions showing that lawyers are more critical of a brief written by an African-American. 

I immediately thought of scientific manuscript review and the not-unusual request to have a revision "thoroughly edited by a native English speaker". My confirmation bias suggests that this is way more common when the first author has an apparently Asian surname.

It would be interesting to see a similar balanced test for scientific writing and review, wouldn't it?

My second thought was.... Ginther. Is this not another one of the thousand cuts contributing to African-American PIs' lower success rates and need to revise the proposal extra times? Seems as though it might be. 

22 responses so far

Thought of the Day

Apr 09 2016 Published by under Fixing the NIH, NIH, NIH Careerism

"Uppity" is a fascinating concept when it comes to NIH Grant award. 

We know the sentiment applies to newer and younger investigators. I've heard countless review and Program Officer comments which amount to "let's not get too big for your britches, young un!" in my day. 

I wonder how much of the Ginther effect is related to sentiments similar to "damn uppity [insert subvocalization]"? 

17 responses so far

MIRA Moaners

Mar 31 2016 Published by under Fixing the NIH, NIH, NIH Careerism

Jocelyn Kaiser reports that some people who applied for MIRA person-not-project support from NIGMS are now complaining. 

I have no* comment.

____
*printable

31 responses so far

How you know NIH officialdom is not being honest with you

Mar 10 2016 Published by under Anger, Fixing the NIH

Continue Reading »

18 responses so far

It's the pig-dog field scientists that are the problem

Mar 10 2016 Published by under Anger, Fixing the NIH, NIH, NIH Careerism

But clearly the laboratory based male scientists would never harass their female subordinates.

Field science is bad.

Lab science is good.

This is what the head of the Office of Extramural Research at the NIH seems to think.

8 responses so far

NIH on Diversity: First, make sure to feed the entrenched powers

Mar 08 2016 Published by under Academics, Fixing the NIH, NIH, Underrepresented Groups

I asked about your experiences with the transition from Master's programs to PhD granting programs because of the NIH Bridges to the Doctorate (R25) Program Announcement. PAR-16-109 has the goal:

... to support educational activities that enhance the diversity of the biomedical, behavioral and clinical research workforce. . ...To accomplish the stated over-arching goal, this FOA will support creative educational activities with a primary focus on Courses for Skills Development and Research Experiences.

Good stuff. Why is it needed?

Underrepresentation of certain groups in science, technology and engineering fields increases throughout the training stages. For example, students from certain racial and ethnic groups including, Blacks or African Americans, Hispanics or Latinos, American Indians or Alaska Natives, Native Hawaiians and other Pacific Islanders, currently comprise ~39% of the college age population (Census Bureau), but earn only ~17% of bachelor’s degrees and ~7% of the Ph.D.’s in the biological sciences (NSF, 2015). Active interventions are required to prevent the loss of talent at each level of educational advancement. For example, a report from the President’s Council of Advisors on Science and Technology recommended support of programs to retain underrepresented undergraduate science, technology, engineering and math students as a means to effectively build a diverse and competitive scientific workforce (PCAST Report, 2012).

  [I actually took this blurb from the related PAR-16-118 because it had the links]

And how is this Bridges to the Doctorate supposed to work?

The Bridges to Doctorate Program is intended to provide these activities to master's level students to increase transition to and completion of PhDs in biomedical sciences. This program requires partnerships between master's degree-granting institutions with doctorate degree-granting institutions.

Emphasis added. Oh Boy.

God forbid we have a NIH program that doesn't ensure that in some way the rich, already heavily NIH-funded institutions get a piece of the action.

So what is it really supposed to be funding? Well first off you will note it is pretty substantial "Application budgets are limited to $300,000 direct costs per year.The maximum project period is 5 years". Better than a full-modular R01! (Some of y'all will also be happy to note that it only comes with 8% overhead.) It is not supposed to fully replace NRSA individual fellowships or training grants.

Research education programs may complement ongoing research training and education occurring at the applicant institution, but the proposed educational experiences must be distinct from those training and education programs currently receiving Federal support. R25 programs may augment institutional research training programs (e.g., T32, T90) but cannot be used to replace or circumvent Ruth L. Kirschstein National Research Service Award (NRSA) programs.

That is less than full support to prospective trainees to be bridged. So what are the funds for?

Salary support for program administration, namely, the PDs/PIs, program coordinator(s), or administrative/clerical support is limited to 30% of the total direct costs annually.

Nice.

Support for faculty from the doctoral institution serving as visiting lecturers, offering lectures and/or laboratory courses for skills development in areas in which expertise needs strengthening at the master’s institution;
Support for faculty from the master’s degree-granting institution for developing or implementing special academic developmental activities for skills development; and
Support for faculty/consultants/role models (including Bridges alumni) to present research seminars and workshops on communication skills, grant writing, and career development plans at the master’s degree institution(s).

Cha-ching! One element for the lesser-light Master's U, one element for the Ivory Tower faculty and one slippery slope that will mostly be the Ivory Tower faculty, I bet. Woot! Two to one for the Big Dogs!

Just look at this specific goal... who is going to be teaching this, can we ask? The folks from the R1 partner-U, that's who. Who can be paid directly, see above, for the effort.

1. Courses for Skills Development: For example, advanced courses in a specific discipline or research area, or specialized research techniques.

Oh White Man's R1 Professor's Burden!
Ahem.

Okay, now we get to the Master's level student who wishes to benefit. What is in it for her?

2. Research Experiences: For example, for graduate and medical, dental, nursing and other health professional students: to provide research experiences and related training not available through formal NIH training mechanisms; for postdoctorates, medical residents and faculty: to extend their skills, experiences, and knowledge base.

Yeah! Sounds great. Let's get to it.

Participants may be paid if specifically required for the proposed research education program and sufficiently justified.

"May". May? MAY???? Wtf!??? Okay if this is setting up unpaid undergraduate style "research experience" scams, I am not happy. At all. This better not be what occurs. Underrepresented folks are disproportionally unlikely to be able to financially afford such shenanigans. PAY THEM!

Applicants may request Bridges student participant support for up to 20 hours per week during the academic year, and up to 40 hours/week during the summer at a pay rate that is consistent with the institutional pay scale.

That's better. And the pivot of this program, if I can find anything at all to like about it. Master's students can start working in the presumably higher-falutin' laboratories of the R1 partner and do so for regular pay, hopefully commensurate with what that University is paying their doctoral graduate students.

So here's what annoys me about this. Sure, it looks good on the surface. But think about it. This is just another way for fabulously well funded R1 University faculty to get even more cheap labor that they don't have to pay for from their own grants. 20 hrs a week during the school year and 40 hrs a week during the summer? From a Master's level person bucking to get into a doctoral program?

Sign me the heck UP, sister.

Call me crazy but wouldn't it be better just to lard up the underrepresented-group-serving University with the cash? Wouldn't it be better to give them infrastructure grants, and buy the faculty out of teaching time to do more research? Wouldn't it be better to just fund a bunch more research grants to these faculty at URGSUs? (I just made that up).

Wouldn't it be better to fund PIs from those underrepresented groups at rates equal to good old straight whitey old men at the doctoral granting institution so that the URM "participants" (why aren't they trainees in the PAR?) would have somebody to look at and think maybe they can succeed too?

But no. We can't just hand over the cash to the URGSUs without giving the already-well-funded University their taste. The world might end if NIH did something silly like that.

5 responses so far

A simple suggestion for Deputy Director for Extramural Research Lauer and CSR Director Nakamura

Nov 19 2015 Published by under Fixing the NIH, NIH, NIH Careerism

Michael S. Lauer, M.D., and Richard Nakamura, Ph.D. have a Perspective piece in the NEJM which is about "Reviewing Peer Review at the NIH". The motivation is captured at the end of the first paragraph:

Since review scores are seen as the proximate cause of a research project's failure to obtain support, peer review has come under increasing criticism for its purported weakness in prioritizing the research that will have the most impact.

The first half or more of the Perspective details how difficult it is to even define impact, how nearly impossible it is to predict in advance and ends up with a very true observation "There is a robust literature showing that expert opinion often fails to predict the future." So why proceed? Well, because

On the other hand, expert opinion of past and current performance has been shown to be a robust measure; thus, peer review may be more helpful when used to assess investigators' track records and renewal grants, as is typically done for research funded by the Howard Hughes Medical Institute and the NIH intramural program.

This is laughably illogical when it comes to NIH grant awards. What really predicts future performance and scientific productivity is who manages to land the grant award. The money itself facilitates the productivity. And no, they have never ever done this test I guarantee you. When have they ever handed a whole pile of grant cash to a sufficient sample of the dubiously-accomplished (but otherwise reasonably qualified) and removed most funding from a fabulously productive (and previously generously-funded) sample and looked at the outcome?

But I digress. The main point comes later when the pair of NIH honchos are pondering how to, well, review the peer review at the NIH. They propose reporting broader score statistics, blinding review*, scoring renewals and new applications in separate panels and correlating scores with later outcome measures.

Notice what is missing? The very basic stuff of experimental design in many areas of research that deal with human judgment and decision making.

TEST-RETEST RELIABILITY.

INTER-RATER RELIABILITY.

Here is my proposal for Drs. Lauer and Nakamura. Find out first if there is any problem with the reliability of review for proposals. Take an allocation of grants for a given study section and convene a parallel section with approximately the same sorts of folks. Or get really creative and split the original panels in half and fill in the rest with ad hocs. Whenever there is a SEP convened, put two or more of them together. Find out the degree to which the same grants get fundable scores.

That's just the start. After that, start convening parallel study sections to, again, review the exact same pile of grants except this time change the composition to see how reviewer characteristics may affect outcome. Make women-heavy panels, URM-heavy panels, panels dominated by the smaller University affiliations and/or less-active research programs. etc.

This would be a great chance to pit the review methods against each other too. They should review an identical pile of proposals in traditional face-to-face meetings versus phone-conference versus that horrible web-forum thing.

Use this strategy to see how each and every aspect of the way NIH reviews grants now might contribute to similar or disparate scores.

This is how you "review peer review" gentlemen. There is no point in asking if peer review predicts X, Y or Z outcome for a given grant when funded if it cannot even predict itself in terms of what will get funded.

__
*And by the way, when testing out peer review, make sure to evaluate the blinding. You have to ask the reviewers to say who they think the PIs are, their level of confidence, etc. And you have to actually analyze the results intelligently. It is not enough to say "they missed most of the time" if either the erroneous or correct guesses are not randomly distributed.

Additional Reading: Predicting the future

In case you missed it, the Lauer version of Rock Talk is called Open Mike.

Cite:
Reviewing Peer Review at the NIH
Michael S. Lauer, M.D., and Richard Nakamura, Ph.D.
N Engl J Med 2015; 373:1893-1895November 12, 2015
DOI: 10.1056/NEJMp1507427

35 responses so far

« Newer posts Older posts »