NIH Mandate to Consider Sex as a Biological Variable in Grant Apps

Jun 09 2015 Published by under NIH, NIH funding, Sex Differences

The NIH has published NOT-OD-15-102 Consideration of Sex as a Biological Variable in NIH-funded Research which informs us:

This notice focuses on NIH's expectation that scientists will account for the possible role of sex as a biological variable in vertebrate animal and human studies. Clarification of these expectations is reflected in plans by NIH's Office of Extramural Research (OER) to update application instructions and review questions; once approved by the Office of Management and Budget (OMB), these updates will take effect for applications submitted for the January 25, 2016, due date and thereafter.

Also:

Accounting for sex as a biological variable begins with the development of research questions and study design. It also includes data collection and analysis of results, as well as reporting of findings. Consideration of sex may be critical to the interpretation, validation, and generalizability of research findings. Adequate consideration of both sexes in experiments and disaggregation of data by sex allows for sex-based comparisons and may inform clinical interventions. Appropriate analysis and transparent reporting of data by sex may therefore enhance the rigor and applicability of preclinical biomedical research.4

NIH expects that sex as a biological variable will be factored into research designs, analyses, and reporting in vertebrate animal and human studies. Strong justification from the scientific literature, preliminary data, or other relevant considerations must be provided for applications proposing to study only one sex. Investigators are strongly encouraged to discuss these issues with NIH program staff prior to submission of applications.

Additional information is provided in a three page PDF overview:

Literature review. Consider and describe how sex and gender may influence the research question(s) at hand. Conduct a review of the human clinical literature and any relevant preclinical literature. If there are differences between males and females in previous preclinical or clinical studies, this would provide a strong rationale for building consideration of sex into the research design and analyses of data. The absence of previous study data in an area of research does not, by itself, constitute strong justification to study only one sex.

Very nice. So helpful. Look NIH, clearly this is going to be a place where applicants who do not wish to incorporate SABV into the design are going to seek a loophole. What would be helpful here would be a more assertive statement about what does and, most importantly does not, constitute a "strong justification to study only one sex". Uncontrolled, this will devolve back to the reviewers who are already failing (going by your highly effortful and high profile new initiative) to appropriately favor* SABV in research grant proposals. They are the ones that will decide that the tiniest fig leaf of excuse making is acceptable "justification" if you give them half a chance to do so. This part needs strengthening.

and later on in the document:

Single-sex studies. Applicants must provide strong justification for applications proposing to study only one sex. Such justification may include the study of sex-specific conditions or phenomena (e.g., ovarian or prostate cancer), acutely scarce resources (e.g., non-human primates), or investigations in which the study of one sex is scientifically appropriate. The absence of evidence regarding sex differences in an area of research does not constitute strong justification to study only one sex.

Sex-specific conditions or phenomena, check. Good. Will hard-to-breed mice constitute "acutely scarce resources"? Human drug abusers of various characteristics that make it hard to recruit female or male participants? The devil will be in the detail. But "scientifically appropriate"? Again, this holds open a big old loophole of escape. And a repeat of the absence-of-evidence statement. What does this mean? What are the limits on this strong justification? How are you going to get reviewers on board with this, instead of leaving them to accept any old excuse?

Research design, data analysis, and reporting.
....Where little or no sex-specific data is available, sex-specific hypotheses may not be possible, whereas previously observed sex differences may prompt sex-specific hypotheses.

Dude what? Are you kidding with this? We all know there must be a supported hypothesis in the research plan. And if there has not been any sex-differences research in the past, well, there are no hypotheses we can advance. And therefore, so sorry, we must avoid proposing anything that investigates SABV because the study section will kill us for lack of a clear hypothesis**. Another whoppingly huge escape clause for the SABV resistant PI.

Acknowledge limitations in the applicability of findings that may arise from the samples, methods, and analyses used, in the research plan as well as in progress reports and publications.

Emphasis added. HAHAHAHHAHHAA!!!! Yeah RIGHT! Every NIH Grant awardee who does not explicitly include SABV in a paper must make sure to add the caveat in the Discussion that their results cannot be extended to the other sex. Sure that's going to happen. Sure.

Finally, one for my peers who already conduct SABV research with regularity.

Researchers working with animal models should consider if and how the female estrous cycle is relevant for experimental design and analysis; it may be relevant for some research questions and not others

This one is pointed straight at the buzz saw of the sex-differences aficionado Stock Criticism of grant applications. One of the ways that sex-differences gets stamped out of research proposals is that the "real" experts start in on "YOU MUST DO THE SEX-COMPARISONS RIGHT AND AS WE HAVE DONE". This may include cycle synchronization, gonadectomy, pharmacologico-hormonal manipulations, endless groups, etc, etc, etc.

There is little tolerance from these people for "First, let's give it a go in female (or male) animals/cells/tissues and see what we turn up" exploratory fishing expeditions.

I would argue that tolerance for fishing expeditions is precisely what the NIH needs if they want to jumpstart real change. You have to make the barrier low and, especially in this day and age, of low cost. Demanding that it has to be SABV design 101eleventy at all times or it is not worth doing is going to motivate resistance. Resistance on the part of PIs doing their grant proposing and on the part of peers doing the grant reviewing.

I propose that a NIH policy of "Any old Third Aim that will engage in sex-differences comparisons is good enough and a total freebie for the first five years***" is what is necessary.

_
*oh yes, believe you me there are puh-lenty of investigators who propose SABV aspects in proposals and get it beat out of them at review.

**StockCritiqueTM

***that may have to be slightly more formal

36 responses so far

  • Grumble says:

    "Resistance on the part of PIs doing their grant proposing and on the part of peers doing the grant reviewing."

    Not to mention resistance on the part of the actual scientists doing the science, which is the most important thing, isn't it? I can totally see a beautiful male vs female experiment showing up in every single one of my grants, and then never actually doing anything remotely similar.

    As long as that particular way of telling NIH to fuck off with their stupid unfunded mandate exists, I will use it.

  • drugmonkey says:

    And come renewal time?

  • Junius Ponds says:

    Will the negligible chance that a grant gets renewed really be improved by investigators choosing to do what they said they were going to do?

  • girlparts says:

    As a reproductive biologist, of course, sex is always a variable. But I've never understood all the hand wringing about cycles in females. In rodents, at least, it is pretty darn easy to check estrus cycle stages and pick one for collections if that is really an important variable.

  • Dave says:

    In rodents, at least, it is pretty darn easy to check estrus cycle stages and pick one for collections if that is really an important variable.

    Yeh, fuck that noise. It's hard enough to get enough mice with some knockouts/transgenics without having to worry about testing the effect of the bloody estrus cycle.

    In the grant I just submitted, I mentioned that there was a difference in phenotype between males and females in preliminary studies (showed some data). I just said I will test everything in male and female mice. Didn't go into too much detail. Might get dinged for it, but we'll see.

    I can't see how a grant score will rest on this issue.....

  • Asst Prof says:

    It is because of bureaucratic bullshit like this that otherwise sane people become tea party republicans.

  • Ass(isstant) Prof says:

    I'm sort of with Dave. My recent submission just says that we will use both male and female animals to account for *potential* sex differences. This was in a data-driven (RFA asks for exploratory) study where we are looking at more generalizable variables that aren't influenced by sex or estrous cycle stage. We'll see if I get dinged for it.

  • Jessica Tollkuhn says:

    @girlparts: yes! THANK YOU! it is annoying, but not impossible to do same-day vaginal swabbing to ensure you test all your females at the same stage of the estrous cycle. Also, lots of sex diffs are actually cycle-independent.

    I know most of the readers here are neuro-inclined and *of course* you all check the sex of your animals when running behaviors. But a lot of more molecular people don't check at all (see latest ENCODE kerfuffle) and I think that's what is driving a lot of this recent NIH noise.

  • qaz says:

    @JessicaTollkuhn - What's driving this politically is the recent Ambien result that dosages differ between males and females.

    But a lot of what is driving the problem is that in the necessity of efficiency, most people are restricting themselves to the minimum N that they can get away with. This means that you want to eliminate variability. This means that you restrict to one sex. Many primate experiments restrict to females only (because males are more aggressive), while many rodent experiments restrict to males only (because they are marginally easier to work with than females - even if the estrus checking is easy, it's obviously harder than not doing it at all).

    What's funny about this is that including both sexes but not looking for a difference is apparently not OK either. Why not? I don't know. (In fact, why do we even have to check estrus cycles? Why can't we just assume that's an additional noise factor? If we see a sex-difference, if the females are more variable, then we run an experiment to check estrus. But if they're not (or no more variable than the noise already in the study), why do we even need to check estrus cycles? You're leaving variability on the table, but maybe sex or estrus isn't driving a lot of variance in your study. ) In part, the kerfuffle is shown in what I saw at a recent study section in which a proposal proposed to do both males and females and mix them (with a reasonable total N) and said "if we see trends to differences, we'll write a new grant to study that difference." This led to a very vigorous debate in the study section with all three reviewers complaining that the study didn't have enough N to see sex-differences and the rest of the study section saying "hey, why isn't this a reasonable thing to do?"

    I think a lot of the problem here is that NIH wants us to include both sexes, but people who actually work on sex differences have developed an entire set of controls of how to do this right, which means that meeting this regulation entails the addition of a whole new set of complex operations, leaving you open to more Stock Critiques.

    What I don't understand is that, given the fact that we're looking at different species already do we really feel that it is critical to test dosages in male and female mice? People need to understand that translation is a theory-driven enterprise (meaning it depends on explanation and understanding) not simply copying results from one species to another. Clinical trials should include the full range of humankind. But does a rat study need to include the full range of ratkind*? Does a yeast study need to include the full range of yeastkind?

    * Rat strain makes a huge difference in study outcomes.

  • Grumble says:

    @JT "it is annoying, but not impossible to do same-day vaginal swabbing to ensure you test all your females at the same stage of the estrous cycle."

    For some studies, it is far more than just "annoying." Consider certain behavioral studies in which many variables have to align in just the right way before one can even do an experiment. Something like optogenetic stimulation of a defined population of neurons at some defined time point while doing electrophysiological recordings in animals performing a decision-making, set-shifting or other complex task. The animal must behave as expected, the optical stimulation must work, and the ephys must be producing data with good signal-to-noise ratio. Out of 5 experiments, you might get one that yields usable data. Now add in the requirement that the animal also has to be at just the right stage of the estrus cycle, and the yield gets even lower. This is not "annoying." It is a brutal impediment to progress.

    @qaz "I think a lot of the problem here is that NIH wants us to include both sexes, but people who actually work on sex differences have developed an entire set of controls of how to do this right, which means that meeting this regulation entails the addition of a whole new set of complex operations, leaving you open to more Stock Critiques."

    This is precisely the problem. On study section, I've seen grants that, trying to follow the NIH gender-inclusiveness zeitgeist, have added groups of female subjects -- only to be torn apart by sex-difference experts whose standards for those sorts of experiments are far higher than the rest of ours.

    @DM "And come renewal time?"
    The key to renewal is to be PRODUCTIVE, not to do everything you said you were going to do. It's trivial to say something like "While Aim 2 proposed to do XYZ, we decided instead to follow up on our exciting results from experiment PQW by doing experiments MNO, which resulted in our recent paper." Come on, DM, you know this.

  • drugmonkey says:

    No I don't. What I know is that reviewers and study sections take different approaches. Some would tear up a renewal that had published lots of papers but with little of the originally proposed work completed. Others don't care a whit, so long as there are cool papers.

    I wouldn't like the renewal odds of someone that proposed sex-diffs and didn't do anything comparing the sexes.

  • Pinko Punko says:

    It really depends how the previous aims were written but even more how the summary statement was written because that is what the reviewers will see. It is not likely that what was proposed will be all that clear. Plus you can get everything done and say that and still get dinged for impact. Renewal is a multi variable problem with highly conditional solutions. Oy.

  • drugmonkey says:

    Reviewers can glean a lot from the summary statement. Also, it is occasionally true that the same reviewers will be on the panel.

    But yeah, nothing annoys a reviewer like seeing a clear problem identified in the prior review go unaddressed in the resulting papers *and* return as a flaw in the competing continuation app.

  • drugmonkey says:

    And to be clear, the continuation app is supposed to have a section listing the prior Aims and how they were addressed. An opportunity for the applicant to advance their argument but also a reminder to the reviewers

  • qaz says:

    DM - interesting. This must be a study-section by study-section thing because I've never seen a grant renewal review care about what the original aims were. All they care is that the person did good science with it. In all of the renewals I've ever seen, the "progress report" section starts with a paragraph saying something like "We set out to do general project XYZ and look at the great stuff we did." I was taught to spend the progress report showing two things (1) Productivity [which means results and publications] and (2) New questions that came out of the results [which hopefully sets up the current proposal].

    I know that my old colleagues always put "Aim 1 was X. Aim 2 was Y. Aim 3 was Z." but that was before my time, back in the old days when doing decent work on a project meant your renewal would be funded, particularly once it came up to its place in line after the holding pattern. But even then, I had a colleague who discovered something new in year 1, chased it for five years, and wrote in his renewal that he'd done lots of productivity and the original aims were still interesting and please fund him to chase them. Apparently it worked. I don't know if it still would.

    Of course, you still have to be in the same ballpark or program will yell at you. (And possibly pull your funding in year 3 or 4!) I wouldn't want to defend switching from bunny hopping to space elevators.

    Aren't you always saying that the grant is not a contract? So it shouldn't matter if you do the actual aims, right?

  • Dave says:

    Yeh and if you're aims get scooped, then what? I'm not going to run off and repeat an experiment that someone else did faster than me. I'm moving on.....

  • bacillus says:

    Why stop with gender differences? What about strain differences, age differences etc? I have experimental vaccines that protect some mouse strains but not others against challenge with the virulent pathogen. Which best mimicks humans? I know what I promote. Furthermore, what mouse strains best represent black people or Asians etc. Moreover, there are papers showing that performing exactly the same experiments in different labs can yield widely different results, as can animals of the same strain from different suppliers in the same lab. Lest we forget, even diferent lab staff in the same lab can get different results, especially in behavioural experiments. My favourite personal experience of this was the fact that a pair of male rhesus monkeys would start wanking like crazy whenever a particular male entered the lab. No one else male or female turned them on. As they say in "Bridge Over the River Kwai" "Madness" "Madness"

  • jmz4gtu says:

    "My favourite personal experience of this was the fact that a pair of male rhesus monkeys would start wanking like crazy whenever a particular male entered the lab."
    -Getting to the truth behind that is a grant I would vigorously support.

  • drugmonkey says:

    So it shouldn't matter if you do the actual aims, right?

    Most importantly, everyone needs to understand that opinions vary. There are personal factors and there are study-section-specific factors. I had an experience where we were reasonably productive along what was supposed to be a single "we'd better check on this" control/validation type of experiment in the proposal. The continuation got absolutely hammered for not doing what we proposed to do, in large part. Despite a clear explanation for why it was so important to keep pullling the threads on what we'd become distracted with.

    Is it justifiable to hold a PI somewhere near the ball park of what they promised to do? I think yes, yes it is, even if I'm really pissed about my scenario (b/c I thought it was ballpark enough, obv).

    Otherwise we're just funding people, not projects. And you know how I feel about that.

  • Grumble says:

    "I've never seen a grant renewal review care about what the original aims were."

    Neither have I. But I am willing to entertain the idea that the study sections DM sits on, and submits his grants to, are composed of people who are even more concerned with trivia and minutia than those who populate the ones I'm involved with.

  • Comradde PhysioProffe says:

    If you write your grants properly, the reviewers of your competing renewal can't know what your original specific aims were, except in very broad terms that you control. This is especially so with the bullet format of written critiques. All they see are the reviews and abstract of the prior funded grant.

  • Grumble says:

    The abstract, of course, can give some of it away. The question is, 7+ years from now, when I've gotten a grant that says I'm going to do this that and the other n female animals, and then I don't actually use a single female animal, will grant reviewers for the renewal say "the progress report did not describe experiments on female subjects"? Or, worse yet, "the PI's publications exclude female subjects"?

  • physioprof says:

    Who says you need to put anything about the sex of animals in your abstract?

  • Grumble says:

    You don't. But will reviewers deduce that there *should have been* female subjects in the previous grant period (because of the NIH mandate), and criticize a renewal because there is no evidence that I ever did a female subjects experiment?

  • Comradde PhysioProffe says:

    Fucke if I know. I'm just glad this is limited to vertebrate animals.

  • drugmonkey says:

    Sounds like a specific review section is planned.

    Clarification of these expectations is reflected in plans by NIH's Office of Extramural Research (OER) to update application instructions and review questions

  • drugmonkey says:

    criticize a renewal because there is no evidence that I ever did a female subjects experiment?

    "track record". boom.

  • Grumble says:

    That does it, I'm switching to fruit flies.

  • drugmonkey says:

    I'll notify Sarah Palin immediately.

  • Dr Becca says:

    I've seen grants that, trying to follow the NIH gender-inclusiveness zeitgeist, have added groups of female subjects -- only to be torn apart by sex-difference experts whose standards for those sorts of experiments are far higher than the rest of ours.

    How is this different from an expert in any other methodology tearing into a proposal because it doesn't include his specific pet control? This is not an issue unique to sex differences.

  • drugmonkey says:

    How is this different from an expert in any other methodology tearing into a proposal because it doesn't include his specific pet control? This is not an issue unique to sex differences.

    Yes and no.

    With this new SABV initiative at the NIH, you have to look at it slightly differently. The argument sounds to me like "We are in a big hole here and it is a huge problem. We need to fix this in a very broad way, accordingly, but our starting baseline is low. Sometimes the results are going to be negative, in many cases we have no idea what the power issues will be, what the critical SABV outcome measures will be - this is tied up in the reason we have come to this conclusion in the first place. So get all damn hands on deck and start figuring this stuff out".

    Holding all of NIH hostage to having perfect sex differences designs is going to seriously compromise the underlying agenda.

    I argue, "permitting" investigators to start at the beginning ("First off, run some female animals in your usual assays") is the best way to move forward. Remember, none of this is coming with any new money that I can see. Grants are going to be proposed around the non-SABV thing that is under study in the labs. So the smart PI will need to move forward judiciously to see where the likely productive avenues are in his or her research. I argue that "Let's check this in the other sex" is ground zero stuff.

    I understand the rage of those out there who have always focused on SABV and struggled to land funding nevertheless. Your point is "JUST FUND US DAMMIT". I get that. And there is a lot of truth in it. I think if NIH really wanted to address this that they would find a way to start picking up sex-differences grants right now this minute. But suppose all of the SABV folks were funded by Dec....this still doesn't have the broad impact that the NIH appears to be pursuing here.

  • Dr Becca says:

    I'm not defending a requirement for exhaustive, every-estrous-phase-plus-gonadectomy experimental designs, here. I'm all for exploratory Aims with well thought out follow-ups should sex differences arise.

    However, I guarantee you that many of those whining about being held to rigorous scientific standards when it comes to SABV would ding others' grants for failing to include whatever their magic number of drug doses, brain regions, etc is. It's hypocritical and demonstrates exactly why people are resistant to SABV beyond the $$/time issue - they just don' t think it matters that much.

  • drugmonkey says:

    You are verging into "it's not faaaair, waaaah" territory.

  • Grumble says:

    " It's hypocritical and demonstrates exactly why people are resistant to SABV beyond the $$/time issue - they just don' t think it matters that much."

    In the interest of being a non-hypocrite, I'll come right out and say that for many forms of basic (NOT applied) science, SABV doesn't matter very much. Not that it doesn't matter at all - just that if you are looking at certain biological processes shared by males and females, there is no a priori reason to assume that they are different. If the exprimenter has a reasonable hypothesis that it does matter, then that hypothesis should be tested as rigorously as possible. But that's not what the NIH is saying. They are saying, "you need to consider SABV even if the hypothesis that there are sex differences in whatever process you are examining is not a reasonable one that could be advanced on the basis of existing evidence."

    It sounds, then, like NIH is asking us to produce a bunch of SABV pilot experiments. Those really should NOT be held to the same standard as full-fledged, hypothesis-driven experiments. Yet that is exactly what I saw happening at study section. The sex differences researchers emerged with blood-stained fangs from all the easy kills.

  • […] first study section rounds that are obliged to grapple with the new SABV policy are upon us. SROs are instructing panels and issuing grant assignments to […]

  • […] be a long day. I'll come back and think about the foci, and try & write about how to cope. For Sex as a Biological Variable (SABV) go see the […]

Leave a Reply