Rigor, reproducibility and the good kid

(by drugmonkey) Feb 09 2018

I was the good kid.

In my nuclear family, in school and in pre-adult employment.

At one point my spouse was in a very large lab and observed how annoying it is when the PI reads everyone the riot act about the sins of a few lab-jerks.

Good citizens find it weird and off-putting when they feel criticized for the sins of others.

They find it super annoying that their own existing good behavior is not recognized.

And they are enraged when the jerko is celebrated for finally, at last managing to act right for once.

Many of us research scientists feel this way when the NIH explains what they mean by their new initiative to enhance "rigor and reproducibility".


"What? I already do that, so does my entire subfield. Wait.....who doesn't do that?" - average good-kid scientist response to hearing the specifics of the R&R initiative.

9 responses so far

SABV in NIH Grant Review

(by drugmonkey) Feb 08 2018

We're several rounds of grant submission/review past the NIH's demand that applications consider Sex As a Biological Variable (SABV). I have reviewed grants from the first round of this obligation until just recently and have observed a few things coming into focus. There's still a lot of wiggle and uncertainty but I am seeing a few things emerge in my domains of grants that include vertebrate animals (mostly rodent models).

1) It is unwise to ignore SABV.

2) Inclusion of both sexes has to be done judiciously. If you put a sex comparison in the Aim or too prominently as a point of hypothesis testing you are going to get the full blast of sex-comparisons review. Which you want to avoid because you will get killed on the usual- power, estrus effects that "must" be there, various caveats about why male and female rats aren't the same - behaviorally, pharmacokinetically, etc etc - regardless of what your preliminary data show.

3) The key is to include both sexes and say you will look at the data to see if there appears to be any difference. Then say the full examination will be a future direction or slightly modify the subsequent experiments.

4) Nobody seems to be fully embracing the SABV concept coming from the formal pronouncements about how you use sample sizes that are half males and half females into perpetuity if you don't see a difference. I am not surprised. This is the hardest thing for me to accept personally and I know for certain sure manuscript reviewers won't go for it either.

Then there comes the biggest categorical split in approach that I have noticed so far.

5a) Some people appear to use a few targeted female-including (yes, the vast majority still propose males as default and females as the SABV-satisfying extra) experiments to check main findings.

5b) The other take is just to basically double everything up and say "we'll run full groups of males and females". This is where it gets entertaining.

I have been talking about the fact that the R01 doesn't pay for itself for some time now.
A full modular, $250K per year NIH grant doesn't actually pay for itself.

the $250K full modular grant does not pay for itself. In the sense that there is a certain expectation of productivity, progress, etc on the part of study sections and Program that requires more contribution than can be afforded (especially when you put it in terms of 40 hr work weeks) within the budget.

The R01 still doesn't pay for itself and reviewers are getting worse

I have reviewed multiple proposals recently that cannot be done. Literally. They cannot be accomplished for the price of the budget proposed. Nobody blinks an eye about this. They might talk about "feasibility" in the sense of scientific outcomes or preliminary data or, occasionally, some perceived deficit of the investigators/environment. But I have not heard a reviewer say "nice but there is no way this can be accomplished for $250K direct".

Well, "we're going to duplicate everything in females" as a response to the SABV initiative just administered the equivalent of HGH to this trend. There is approximately zero real world dealing with this in the majority of grants that slap in the females and from what I have seen no comment whatever from reviewers on feasibility. We are just entirely ignoring this.

What I am really looking forward to is the review of grants in about 3 years time. At that point we are going to start seeing competing continuation applications where the original promised to address SABV. In a more general sense, any app from a PI who has been funded in the post-SABV-requirement interval will also face a simple question.

Has the PI addressed SABV in his or her work? Have they taken it seriously, conducted the studies (prelim data?) and hopefully published some things (yes, even negative sex-comparisons)?

If not, we should, as reviewers, drop the hammer. No more vague hand wavy stuff like I am seeing in proposals now. The PI had better show some evidence of having tried.

What I predict, however, is more excuse making and more bad faith claims to look at females in the next funding interval.

Please prove me wrong, scientists in my fields of study.

Additional Reading:
NIH's OER blog Open Mike on the SABV policies.
NIH Reviewer Guidance [PDF]

3 responses so far

Undue influence of frequent NIH grant reviewers

(by drugmonkey) Feb 07 2018

A quotation

Currently 20% of researchers perform 75-90% of reviews, which is an unreasonable and unsustainable burden.

referencing this paper on peer review appeared in a blog post by Gary McDowell. It caught my eye when referenced on the twitts.

The stat is referencing manuscript / journal peer review and not the NIH grant review system but I started thinking about NIH grant review anyway. Part of this is because I recently had to re-explain one of my key beliefs about a major limitation of the NIH grant review system to someone who should know better.

NIH Grant review is an inherently conservative process.

The reason is that the vast majority of reviews of the merit of grant applications are provided by individuals who already have been chosen to serve as Principal Investigators of one or more NIH grant awards. They have had grant proposals selected as meritorious by the prior bunch of reviewers and are now are contributing strongly to the decision about the next set of proposals that will be funded.

The system is biased to select for grant applications written in a way that looks promising to people who have either been selected for writing grants in the same old way or who have been beaten into writing grants that look the same old way.

Like tends to beget like in this system. What is seen as meritorious today is likely to be very similar to what has been viewed as meritorious in the past.

This is further amplified by the social dynamics of a person who is newly asked to review grants. Most of us are very sensitive to being inexperienced, very sensitive to wanting to do a good job and feel almost entirely at sea about the process when first asked to review NIH grants. Even if we have managed to stack up 5 or 10 reviews of our proposals from that exact same study section prior to being asked to serve. This means that new reviewers are shaped even more by the culture, expectations and processes of the existing panel, which is staffed with many experienced reviewers.

So what about those experienced reviewers? And what about the number of grant applications that they review during their assigned term of 4 (3 cycles per year, please) or 6 (2 of 3 cycles per year) years of service? With about 6-10 applications to review per round this could easily be highly influential (read: one of the three primary assigned reviewers) review of 100 applications. The person has additional general influence in the panel as well, both through direct input on grants under discussion and on the general tenor and tone of the panel.

When I was placed on a study section panel for a term of service I thought the SRO told us that empaneled reviewers were not supposed to be asked for extra review duties on SEPs or as ad hoc on other panels by the rest of the SRO pool. My colleagues over the years have disabused me of the idea that this was anything more than aspirational talk from this SRO. So many empaneled reviewers are also contributing to review beyond their home review panel.

My question of the day is whether this is a good idea and whether there are ethical implications for those of us who are asked* to review NIH grants.

We all think we are great evaluators of science proposals, of course. We know best. So of course it is all right, fair and good when we choose to accept a request to review. We are virtuously helping out the system!

At what point are we contributing unduly to the inherent conservativeness of the system? We all have biases. Some are about irrelevant characteristics like the ethnicity** of the PI. Some are considered more acceptable and are about our preferences for certain areas of research, models, approaches, styles, etc. Regardless these biases are influencing our review. Our review. And one of the best ways to counter bias is the competition of competing biases. I.e., let someone else's bias into the mix for a change, eh buddy?

I don't have a real position on this yet. After my term of empaneled service, I accepted or rejected requests to review based on my willingness to do the work and my interest in a topic or mechanism (read: SEPs FTW). I've mostly kept it pretty minimal. However, I recently messed up because I had a cascade of requests last fall that sucked me in- a "normal" panel (ok, ok, I haven't done my duty in a while), followed by a topic SEP (ok, ok I am one of a limited pool of experts I'll do it) and then a RequestThatYouDon'tRefuse. So I've been doing more grant review lately than I have usually done in recent years. And I'm thinking about scope of influence on the grants that get funded.

At some point is it even ethical to keep reviewing so damn much***? Should anyone agree to serve successive 4 or 6 year terms as an empaneled reviewer? Should one say yes to every SRO request that comes along? They are going to keep asking so it is up to us to say no. And maybe to recommend the SRO ask some other person who is not on their radar?

*There are factors which enhance the SRO pool picking on the same old reviewers, btw. There's a sort of expectation that if you have review experience you might be okay at it. I don't know how much SROs talk to each other about prospective reviewers and their experience with the same but there must be some chit chat. "Hey, try Dr. Schmoo, she's a great reviewer" versus "Oh, no, do not ever ask Dr. Schnortwax, he's toxic". There are the diversity rules that they have to follow as well- There must be diversity with respect to the geographic distribution, gender, race and ethnicity of the membership. So people that help the SROs diversity stats might be picked more often than some other people who are straight white males from the most densely packed research areas in the country working on the most common research topics using the most usual models and approaches.

**[cough]Ginther[cough, cough]

***No idea what this threshold should be, btw. But I think there is one.

18 responses so far

Thought of the Day

(by drugmonkey) Feb 07 2017

I started blogging in a fit of anger about some aspect of the grant-funded scientific research career or other.

In the course of venting a lot of spleen about things that were bothering me, I met a lot of new people, virtually and eventually in real life.

These people taught me a lot of interesting stuff. Some career related, some scientific, some just plain old life.

It has been an interesting decade.

Thanks. To all my Readers and Commenters over the years.

Thank you.

34 responses so far


(by drugmonkey) Jan 31 2017

It is sort of interesting that just as my decade blogging as DrugMonkey is expiring, all this political jazz erupts.

20 responses so far


(by drugmonkey) Jan 31 2017


Indecent people do not stop because they realize they are wrong or went too far.

They only stop when the decent people stop them.

3 responses so far

Scientists' march on Washington

(by drugmonkey) Jan 26 2017


26 responses so far

The past is prologue: Political NIH interference edition

(by drugmonkey) Jan 24 2017

From a prestigious general science journal:

"Important elements in both Senate and the House are showing increasing dissatisfaction over Congress's decade-long honeymoon with medical research....critics are dissatisfied...with the NIH's procedures for supervising the use of money by its research grantees....NIH officials..argued, rather, that the most productive method in financing research is to pick good people with good projects and let them carry out their work without encumbering them...its growth has been phenomenal....[NIH director}: nor do we believe that most scientific groups in the country have an asking and a selling price for their product which is research activity...we get a realistic appraisal of what they need to do the job..the supervisory function properly belongs to the universities and other institutions where the research takes place....closing remarks of the report are:...Congress has been overzealous in appropriating money for health research".

D.S. Greenberg, Medical Research Funds: NIH Path Through Congress Has Developed Troublesome Bumps, Science 13 Jul 1962, Vol. 137, Issue 3524, pp. 115-119
DOI: 10.1126/science.137.3524.115 [link]
Previously posted.

9 responses so far

The Trump Administration is gagging science and we are next

(by drugmonkey) Jan 24 2017

The Environmental Protection Agency (EPA) has been told to stop funding grants and stop talking about scientific findings. Via Reuters:

U.S. President Donald Trump's administration has asked the Environmental Protection Agency to temporarily halt all contracts, grants and interagency agreements pending a review, according to sources.

The White House sent a letter to the EPA's Office of Administration and Resources Management ordering the freeze on Monday, an EPA staffer told Reuters. "Basically no money moving anywhere until they can take a look," the staffer said, asking not to be named.

Via The Verge:

Also, employees have been banned from providing updates to reporters or on social media. The internal memo specifies that no press releases will go out to external audiences, there will be “no blog messages” and media requests will be carefully screened.

The United States Department of Agriculture (USDA) has been told not to talk about their science. Via Buzzfeed News:

According to an email sent Monday morning and obtained by BuzzFeed News, the department told staff — including some 2,000 scientists — at the agency’s main in-house research arm, the Agricultural Research Service (ARS), to stop communicating with the public about taxpayer-funded work.

“Starting immediately and until further notice, ARS will not release any public-facing documents,” Sharon Drumm, chief of staff for ARS, wrote in a department-wide email shared with BuzzFeed News.

“This includes, but is not limited to, news releases, photos, fact sheets, news feeds, and social media content,” she added.

I'm sure I do not need to lead you by the hand to realize that this is a political putsch directed against scientific entities that tend to relate data that is discordant with Republican worldviews and preferred policies.

This is just the beginning. Each successful gag/intimidation will just fuel the next one.

NIH and NSF, dear to many in my audience, are most assuredly next, people. I hope not too many of you are counting on grant awards that are due to be issued in the next, oh, six months or so. Especially if you work on scientific questions that the Republicans have attacked before. Such as HIV/AIDS, drug abuse, mental health, LGBT issues, anything to do with family structure or dynamics, much of psychology and sociology, etc. And lord help you if you work with obviously ridiculous models such as fruit flies or obviously ridiculous therapies, like exercise.

How would you like some posturing Congress Critter to de-fund your grant?

50 responses so far

Thing I learned on Twitter

(by drugmonkey) Jan 18 2017

Valid #NIHgrant review is determined only by the aspects of the application that I excel at and any other influence is unfair bias.

2 responses so far

« Newer posts Older posts »