Archive for the 'Sex Differences' category

Reviewing SABV grant applications for the NIH

May 09 2016 Published by under NIH, NIH funding, Sex Differences

The first study section rounds that are obliged to grapple with the new SABV policy are upon us.
SROs are instructing panels and issuing grant assignments to reviewers.

If you are reviewing, what are your thoughts?

Me, I see more of the entirely predictable ahead- people ignoring it (or accepting thin excuses for not studying both sexes, in reality) or brandishing it as a cudgel in highly variable fashion. I'm cynical, perhaps unduly so, eh?

The opportunity to beat a panel into better agreement will come far too late for most applications. There is no way that guilt over consistency will drag the triaged apps up for discussion. 

I still seek consistency. In my own reviewing and in any panels I serve. I think this a virtue to strive for.

And I think that consideration and discussion of the approach to tricky review issues is the way to advance toward that goal. 

I also think that when you accept a reviewer position, you are agreeing to give the NIH what it is requesting, to the best of your ability. If you fight against the SABV push, you are doing it very wrong, IMO.

So....what do you think? How are you approaching the SABV mandate? Now that you have a few examples of how applicants have dealt with it, have you learned anything useful for us to consider?

--
Open Mike blog on SABV mandate

12 responses so far

Re-Repost: The funding is the science II, "Why do they always drop the females?"

The NIH has recently issued the first round of guidance on inclusion of Sex as a Biological Variable in future NIH research grants. I am completely behind the spirit of the initiative but I have concerns about how well this is going to work in practice. I wrote a post in 2008 that detailed some of the reasons that have brought us to the situation where the Director of the NIH felt he had to coauthor an OpEd on this topic. I believe these issues are still present, will not be magically removed with new instructions to reviewers and need to be faced head-on if the NIH is to make any actual progress on ensuring SABV is considered appropriately going forward.

The post originally appeared December 2, 2008.


The title quote came from one of my early, and highly formative, experiences on study section. In the course of discussing a revised application it emerged that the prior version of the application had included a sex comparison. The PI had chosen to delete that part of the design in the revised application, prompting one of the experienced members of the panel to ask, quite rhetorically, "Why do they always drop the females?"

I was reminded of this when reading over Dr. Isis' excellent post [Update: Original Sb post lost, I think the repost can be found here] on the, shall we say less pernicious, ways that the course of science is slanted toward doing male-based research. Really, go read that post before you continue here, it is a fantastic description.

What really motivated me, however, was a comment from the always insightful Stephanie Z:

Thank you. That's the first time I've seen someone address the reasons behind ongoing gender disparities in health research. I still can't say as it thrills me (or you, obviously), but I understand a bit better now.

Did somebody ring?

As I pointed out explicitly at least once ([Update: Original 2007 post]), research funding has a huge role in what science actually gets conducted. Huge. In my book this means that if one feels that an area of science is being systematically overlooked or minimized, one might want to take a close look at the manner by which science is funded and the way by which science careers are sustained as potential avenues for systematic remedy.

Funding

There are a couple of ways in which the generalized problems with NIH grant review lead to the rhetorical comment with which I opened the post. One very common StockCritique of NIH grant review is that of an "over ambitious" research plan. As nicely detailed in Isis' post, the inclusion of a sex comparison doubles the groups right off the bat but even more to the point, it requires the inclusion of various hormonal cycling considerations. This can be as simple as requiring female subjects to be assessed at multiple points of an estrous cycle. It can be considerably more complicated, often requiring gonadectomy (at various developmental timepoints) and hormonal replacement (with dose-response designs, please) including all of the appropriate control groups / observations. Novel hormonal antagonists? Whoops, the model is not "well established" and needs to be "compared to the standard gonadectomy models", LOL >sigh<.

manWomanControlPanel.jpg
Grant reviewers prefer simplicity
Keep in mind, if you will, that there is always a more fundamental comparison or question at the root of the project, such as "does this drug compound ameliorate cocaine addiction?" So all the gender comparisons, designs and groups need to be multiplied against the cocaine addiction/treatment conditions. Suppose it is one of those cocaine models that requires a month or more of training per group? Who is going to run all those animals ? How many operant boxes / hours are available? and at what cost? Trust me, the grant proposal is going to take fire for "scope of the project".

Another StockCritique to blame is "feasibility". Two points here really. First is the question of Preliminary Data- of course if you have to run more experimental conditions to establish that you might have a meritorious hypothesis, you are less likely to do it with a fixed amount of pilot/startup/leftover money. Better to work on preliminary data for two or three distinct applications over just one if you have the funds. Second aspect has to do with a given PIs experience with the models in question. More opportunity to say "The PI has no idea what s/he is doing methodologically" if s/he has no prior background with the experimental conditions, which are almost always the female-related ones. As we all know, it matters little that the hormonal assays or gonadectomy or whatever procedures have been published endlessly if you don't have direct evidence that you can do it. Of course, more latitude is extended to the more-experienced investigator....but then s/he is less likely to jump into gender-comparisons in a sustained way in contrast to a newly minted PI.

Then there are the various things under grantspersonship. You have limited space in a given type of grant application. The more groups and comparisons, the more you have to squeeze in with respect to basic designs, methods and the interpretation/alternative approaches part. So of course you leave big windows for critiques of "hasn't fully considered...." and "it is not entirely clear how the PI will do..." and "how the hypothesis will be evaluated has not been sufficiently detailed...".

Career

Although research funding plays a huge role in career success, it is only part of the puzzle. Another critical factor is what we consider to be "great" or "exciting" science in our respective fields.

The little people can fill in the details. This is basically the approach of GlamourMagz science. (This is a paraphrase of something the most successful GlamourMagz PI I know actually says.) Cool, fast and hot is not compatible with the metastasizing of experimental conditions that is an inevitable feature of gender-comparison science. Trouble is, this approach tends to trickle down in various guises. Lower (than GlamourMag) impact factor journals sometimes try to upgrade by becoming more NS-like (Hi, J Neuro!). Meticulous science and exacting experimental designs are only respected (if at all) after the fact. Late(r) in someone's career they start getting props on their grant reviews for this. Early? Well the person hasn't yet shown the necessity and profit for the exhaustive designs and instead they just look...unproductive. Like they haven't really shown anything yet.

As we all know splashy CNS pubs on the CV trump a sustained area of contribution in lower journals six ways to Sunday. This is not to say that nobody will appreciate the meticulous approach, they will. Just to say that high IF journal pubs will trump. Always.

So the smart young PI is going to stay away from those messy sex-differences studies. Everything tells her she should. If he does dip a toe, he's more likely to pay a nasty career price.
This is why NIH efforts to promote sex-comparison studies are necessary. Promoting special funding opportunities are the only way to tip the equation even slightly more favorable to the sex-differences side. The lure of the RFA is enough to persuade the experienced PI to write in the female groups. To convince the new PI that she might just risk it this one time.

My suspicion is that it is not enough. Beyond the simple need to take a stepwise approach to the science as detailed by Isis, the career and funding pressures are irresistible forces.

9 responses so far

NIH Mandate to Consider Sex as a Biological Variable in Grant Apps

Jun 09 2015 Published by under NIH, NIH funding, Sex Differences

The NIH has published NOT-OD-15-102 Consideration of Sex as a Biological Variable in NIH-funded Research which informs us:

This notice focuses on NIH's expectation that scientists will account for the possible role of sex as a biological variable in vertebrate animal and human studies. Clarification of these expectations is reflected in plans by NIH's Office of Extramural Research (OER) to update application instructions and review questions; once approved by the Office of Management and Budget (OMB), these updates will take effect for applications submitted for the January 25, 2016, due date and thereafter.

Also:

Accounting for sex as a biological variable begins with the development of research questions and study design. It also includes data collection and analysis of results, as well as reporting of findings. Consideration of sex may be critical to the interpretation, validation, and generalizability of research findings. Adequate consideration of both sexes in experiments and disaggregation of data by sex allows for sex-based comparisons and may inform clinical interventions. Appropriate analysis and transparent reporting of data by sex may therefore enhance the rigor and applicability of preclinical biomedical research.4

NIH expects that sex as a biological variable will be factored into research designs, analyses, and reporting in vertebrate animal and human studies. Strong justification from the scientific literature, preliminary data, or other relevant considerations must be provided for applications proposing to study only one sex. Investigators are strongly encouraged to discuss these issues with NIH program staff prior to submission of applications.

Additional information is provided in a three page PDF overview:

Literature review. Consider and describe how sex and gender may influence the research question(s) at hand. Conduct a review of the human clinical literature and any relevant preclinical literature. If there are differences between males and females in previous preclinical or clinical studies, this would provide a strong rationale for building consideration of sex into the research design and analyses of data. The absence of previous study data in an area of research does not, by itself, constitute strong justification to study only one sex.

Very nice. So helpful. Look NIH, clearly this is going to be a place where applicants who do not wish to incorporate SABV into the design are going to seek a loophole. What would be helpful here would be a more assertive statement about what does and, most importantly does not, constitute a "strong justification to study only one sex". Uncontrolled, this will devolve back to the reviewers who are already failing (going by your highly effortful and high profile new initiative) to appropriately favor* SABV in research grant proposals. They are the ones that will decide that the tiniest fig leaf of excuse making is acceptable "justification" if you give them half a chance to do so. This part needs strengthening.

and later on in the document:

Single-sex studies. Applicants must provide strong justification for applications proposing to study only one sex. Such justification may include the study of sex-specific conditions or phenomena (e.g., ovarian or prostate cancer), acutely scarce resources (e.g., non-human primates), or investigations in which the study of one sex is scientifically appropriate. The absence of evidence regarding sex differences in an area of research does not constitute strong justification to study only one sex.

Sex-specific conditions or phenomena, check. Good. Will hard-to-breed mice constitute "acutely scarce resources"? Human drug abusers of various characteristics that make it hard to recruit female or male participants? The devil will be in the detail. But "scientifically appropriate"? Again, this holds open a big old loophole of escape. And a repeat of the absence-of-evidence statement. What does this mean? What are the limits on this strong justification? How are you going to get reviewers on board with this, instead of leaving them to accept any old excuse?

Research design, data analysis, and reporting.
....Where little or no sex-specific data is available, sex-specific hypotheses may not be possible, whereas previously observed sex differences may prompt sex-specific hypotheses.

Dude what? Are you kidding with this? We all know there must be a supported hypothesis in the research plan. And if there has not been any sex-differences research in the past, well, there are no hypotheses we can advance. And therefore, so sorry, we must avoid proposing anything that investigates SABV because the study section will kill us for lack of a clear hypothesis**. Another whoppingly huge escape clause for the SABV resistant PI.

Acknowledge limitations in the applicability of findings that may arise from the samples, methods, and analyses used, in the research plan as well as in progress reports and publications.

Emphasis added. HAHAHAHHAHHAA!!!! Yeah RIGHT! Every NIH Grant awardee who does not explicitly include SABV in a paper must make sure to add the caveat in the Discussion that their results cannot be extended to the other sex. Sure that's going to happen. Sure.

Finally, one for my peers who already conduct SABV research with regularity.

Researchers working with animal models should consider if and how the female estrous cycle is relevant for experimental design and analysis; it may be relevant for some research questions and not others

This one is pointed straight at the buzz saw of the sex-differences aficionado Stock Criticism of grant applications. One of the ways that sex-differences gets stamped out of research proposals is that the "real" experts start in on "YOU MUST DO THE SEX-COMPARISONS RIGHT AND AS WE HAVE DONE". This may include cycle synchronization, gonadectomy, pharmacologico-hormonal manipulations, endless groups, etc, etc, etc.

There is little tolerance from these people for "First, let's give it a go in female (or male) animals/cells/tissues and see what we turn up" exploratory fishing expeditions.

I would argue that tolerance for fishing expeditions is precisely what the NIH needs if they want to jumpstart real change. You have to make the barrier low and, especially in this day and age, of low cost. Demanding that it has to be SABV design 101eleventy at all times or it is not worth doing is going to motivate resistance. Resistance on the part of PIs doing their grant proposing and on the part of peers doing the grant reviewing.

I propose that a NIH policy of "Any old Third Aim that will engage in sex-differences comparisons is good enough and a total freebie for the first five years***" is what is necessary.

_
*oh yes, believe you me there are puh-lenty of investigators who propose SABV aspects in proposals and get it beat out of them at review.

**StockCritiqueTM

***that may have to be slightly more formal

36 responses so far

The NIH says investigators must incorporate sex-differences analyses in their studies

May 14 2014 Published by under Drug Abuse Science, NIH, NIH funding, Sex Differences

For some reason I am having a DOI error on the actual comment from Clayton and Collins. So until that is resolved, the sourcing is from the journalists who got the embargoed version.

Apparently Janine Clayton and Francis Collins have issued a commentary on a new policy that the NYT describes as:

The N.I.H. is directing scientists to perform their experiments with both female and male animals and include both sexes in sufficient numbers to see statistically significant differences. Grant reviewers will be instructed to take the sex balance of each study design into account when awarding grants.

Yeah, that sounds pretty clear. My studies just doubled...which means really that they were just cut in half. I'm cool with that. I actually agree that it would be good if we did almost everything as a sex-differences study.

There's the money though. Sex difference studies in a behaving animal are not just a doubling as it happens (and as I inaccurately described it just above). From a prior post on this topic entitled: The funding is the science II, "Why do they always drop the females?"

As nicely detailed in Isis' post, the inclusion of a sex comparison doubles the groups right off the bat but even more to the point, it requires the inclusion of various hormonal cycling considerations. This can be as simple as requiring female subjects to be assessed at multiple points of an estrous cycle. It can be considerably more complicated, often requiring gonadectomy (at various developmental timepoints) and hormonal replacement (with dose-response designs, please) including all of the appropriate control groups / observations. Novel hormonal antagonists? Whoops, the model is not "well established" and needs to be "compared to the standard gonadectomy models", LOL >sigh<.

The money and the progress.

Keep in mind, if you will, that there is always a more fundamental comparison or question at the root of the project, such as "does this drug compound ameliorate cocaine addiction?" So all the gender comparisons, designs and groups need to be multiplied against the cocaine addiction/treatment conditions. Suppose it is one of those cocaine models that requires a month or more of training per group? Who is going to run all those animals ? How many operant boxes / hours are available? and at what cost?

Oh, don't worry bench jockeys. According to the NYT article:

Researchers who work with cell cultures are also being encouraged to study cells derived from females as well as males, and to do separate analyses to tease out sex differences at the cellular level.

“Every cell has a sex,” Dr. Clayton said in a telephone interview. “Each cell is either male or female, and that genetic difference results in different biochemical processes within those cells.”

“If you don’t know that and put all of the cells together, you’re missing out, and you may also be misinterpreting your data,” Dr. Clayton added. For example, researchers recently discovered that neurons cultured from males are more susceptible to death from starvation than those from females, because differences in the ways their cells process nutrients.

"Encouraged". Okay, maybe you CultureClowns have an escape clause here. Animal model folks are facing "demanded" language.

Final observations are ridiculous:

But [the new policies] are likely to be met with resistance from scientists who fear increased costs and difficulty in performing their experiments. Studying animals of both sexes may potentially double the number required in order to get significant results.

“There’s incredible inertia among people when it comes to change, and the vast majority of people doing biological research are going to think this is a huge inconvenience,” Dr. Zucker said.

...

Margaret McCarthy, a neuroscientist at University of Maryland School of Medicine who studies sex differences, agreed. “The reactions will range from hostile — ‘You can’t make me do that’ — to ‘Oh, I don’t want to control for the estrous cycle,'” she said.

This has nothing to do with whether a scientist "wants" to or not.

Let me be clear, I want to do sex-differences studies. I am delighted that this will be a new prescription. I agree with the motivating sentiments.

What I "fear" is that grant applications will be kicked in the teeth if they include sex differences comparisons. What I "fear" is that my research programs will be even less productive on the main area of interest, to the tune of a lot of extra work that will simply confirm a lot of what we already know. For example, female rats tend to self-administer more drug than males do. A lot of my colleagues have been working on these topics for a long time. The identification of those areas where it actually matters (i.e., sex difference effects that haven't yet been detected) are going to come along with a lot of negative findings. What I "fear" is that when we are interested in a certain thing, there is a bit of sex-differences literature and the hypothesis is going to be "males and females are the same" or even "females are more/less sensitive to drug" that this is going to bring down the holy hells of reviewer wrath over what hypothesis we are testing.

I fear a lot of things about this. What I don't fear is my own interest in the topic. What I don't fear is the "inconvenience". I don't even fear "difficulty". It just isn't that difficult to add female groups to my studies.

What it takes is additional grant funding. Or tolerance on the part of P&T committees, hiring committees and grant review panels for apparently reduced progress on a scientific topic of interest. And those things are not at all easy to come by.

The funny thing is, we've been taking steps in the lab toward this direction in the past year anyway. So I should be grateful I have at least that little tiny bit of a head start on this stuff.

21 responses so far