One-and-done NIH Grants: A bug or a feature?

(by drugmonkey) Oct 29 2014

There is an interesting new survival analysis for PIs that have been awarded at least one R01 posted at RockTalk.

Briefly, what you are looking at is as follows:


We chose three cohorts of first-time R01-equivalent awardees — those who received their first R01-equivalent award in 1989, 1997, or 2003....We used data on these three cohorts for a Kaplan-Meier analysis to look at rates of retention. ...So we used this to analyzing the number of years between the first year of R01-equivalent funding, and the last time an individual receives any additional research project grant (RPG) funding – whether it be from the non-competing continuation of their 1st R01 or another RPG award.

The bulk of Sally Rockey's analytical comments focus on the sharp dropout associated with the first (presumed) interval of support, i.e., 3-5 years after the very first R01-equivalent award. And her conclusion seems to be that this is a problem that needs fixing.

These data seem to support the concept that if there is an intervention needed in retaining scientists in research, it would need to come at the renewal stage of the first award, or as some call it the “second” award. Indeed, we are giving increased focus to this stage through some of our new award mechanisms, such as the National Cancer Institute’s Outstanding Investigator award, and will continue to seek ways of keeping our talent from leaking out of the pipeline.

I am not sure that I agree with this general conclusion from the data presented.

Remember, we are talking system-wide statistics here, not the fate of your five closest colleagues, your training mentor or yourself.

The extreme case would be that once anyone manages to land an R01-equivalent award as a PI, that the NIH should move heaven and earth to keep them funded for the duration of their career. That is an arguable position, but I think it is wrong.

It is wrong for two reasons, which are related to each other. The first reason is that if we re-adjust the system to keep everyone in once they have entered, this will sharply reduce the entrants. It will reduce the number of people who get a chance to prove themselves. This may seem fine and dandy once you have passed the first-R01 hurdle yourself, but this is mind bogglingly forgetful of the position one was in before this and mind bogglingly arrogant in assuming you would have been one of the lucky few.

From a system perspective, this cuts down on scientific diversity. It cuts down on the ability to try out a range of scientific ideas and approaches to see which ones stick. It substitutes the limitations of advance prediction for the virtues of empirical testing.

This would also increase stagnation and slow progress. It would. When you have sinecure funding, I'm sorry but the pressures are not as high to be creative, productive and to diversity your scientific thinking. Yes, from our current vantage point of the amount of time spent securing funds versus doing the science, this may look better than the usual. But take the longer view here. Our competitive system has its virtues in terms of clearing out the dead wood and encouraging better efforts from those who are actively participating.

I think the real question here is the appropriate balance. The desired survival rate.

And we should be very clear that we expect there to be some amount of PI dropout.

Personally, I think that having the major reduction in PIs after the first interval of funding makes a lot of sense. Better than at year 11 or 16, for example. A given PI has had a chance to try out her ideas. He has been given the opportunity to show what he can do. Again, on a system-wide basis, some of these individuals are going to fail so badly that they are never funded again. Do recognize that many will suffer intervals of no-R01 and then come back. Datahound addressed that in a post earlier this year. But many will disappear. Wouldn't it make more sense to have it happen before the NIH has wasted another 5 year interval on them?

Now before I get too far down the path, I will recognize that this is only the start of the data analysis. We want to know a few more things about who is shelled out of the system, never to return, who is able to fight back in after a gap and who is able to sail along with continual funding. This will allow us to see what may be undesired effects or categories of PI/applications that we wish to specifically protect. Is this part of the perfect storm that hits women particularly hard? Human subjects research? Physician-scientist PIs? Ecology or sociology?

Nevertheless, when I look at these initial survival curves, I just don't see the problem that seems so obvious to Dr. Rockey. I don't see how this is the next problem that requires special fixes from the highest offices at NIH.

22 responses so far

Your Grant in Review: Follow the Reviewers' Style Guide

(by drugmonkey) Oct 27 2014

The NIH grant application has a tremendous amount of room for stylistic choice. No, I'm not talking about Georgia font again, nor your points-leaving choice to cite your references with numbers instead of author-date.

Within the dictated structure of Aims, Significance, Innovation, etc, there is a lot of freedom.

Where do I put the Preliminary Data now that there is no defined section? What comes first in the Approach- Aim 1? The Timeline? A bunch of additional rationale/background? Do you start every Aim with a brief Rationale and then list a bunch of Experiments? Which methods are "general" enough to put them at the end of Aim 3?

Do I include Future Directions?

What about discussion of Possible Pitfalls and Alternate Considerations and all that jazz?

Is the "Interpretation" for each Aim supposed to be an extensive tretise on results that you don't even have yet?

In all of this there is one certainty.

Ideally you are submitting multiple applications to a single study section over time. If not that, then you are likely submitting a revised version of an application that was not funded to the same study section that reviewed it in the first place. Study sections tend to have an evolved and transmissible culture that changes only slowly. There is a tendency for review to focus (overfocus, but there you have it) on certain structural expectations, in part as a way to be fair* to all the applications. There is a tendency for the study section to be the most comfortable with certain of these optional, stylistic features of a grant application included in juuuust the way that they expect.

So, and here is the certainty, if a summary statement suggests your application is deficient in one of these stylistic manners just suck it up and change your applications to that particular study section accordingly.

Is a Timeline silly when you've laid out a very simple and time-estimated set of experiments in a linear organization throughout the Aims? Perhaps. Is it idiotic to talk about alternatives when you conduct rapid, vertically ascending eleventy science and everything you propose right now is obsolete by the time Year 2 funds? Likely. Why do you need to lead the reviewers by the hand when your Rationale and experimental descriptions make it clear how the hypothesis will be tested and what it would mean? Because.

So when your summary statement suggests a stylistic variant that you wouldn't otherwise prefer...just do it.
__
Additional Your Grant in Review posts.

*If the section has beaten up several apps because they did not appropriately discuss the Possible Pitfalls, or include Future Directions, well, they have to do it for all the apps. So the tendency goes anyway.

52 responses so far

Once again, it is not pre-publication peer review that is your problem

(by drugmonkey) Oct 24 2014

The perennial discussion arose on the Twitts yesterday, probably sparked by renewed ranting from dear old @mbeisen.

For the uninitiated, a brief review of the components that go into the pre-publication approval of a scientific manuscript. In general, authors select a journal and submit a manuscript that they feel is ready for publication*. At this point an Editor (usually there is one Editor in Chief and a handful or so of sub-Editors usually referred to as Associate Editors; AEs) gives it a look and decides whether to 1) accept it immediately, 2) reject it immediately or 3) send it to peer scientists for review. The first option is exceptionally rare. The second option depends on both the obvious "fit" of the manuscript for the journal in terms of topic (e.g., "Mathematical Proof of Newton's Fifth and Sixth Laws" submitted to "Journal of Trangender Sociology") and some more-rarified considerations not immediately obvious to the home viewer.
Continue Reading »

22 responses so far

Datahound on productivity

(by drugmonkey) Oct 23 2014

This final figure from Datahound's post on K99/R00 recipients who have managed to win R01 funding is fascinating to me. This is a plot of individual investigators, matching their number of published papers against a weighted sum of publication. The weighting is for the number of authors on each paper as follows: "One way to correct for the influence of an increased number of authors on the number of publications is to weight each publication by 1/(number of authors) (as was suggested by a comment on Twitter). In this scenario, a paper with two authors would be worth 1/2 while a paper with 10 authors would be worth 1/10."

Doing this adjustment to the unadjusted authors/papers relationship tightens up the relationship from a correlation coefficient of 0.47 to 0.83.

Ultimately this post shows pretty emphatically that when you operate in a subfield or niche or laboratory that tends to publish papers with a lot of authors, you get more author credits. This even survives the diluting effect of dividing each paper by the number of authors on it. There are undoubtedly many implications.

I think the relationship tends to argue that increasing the author number is not a reflection of the so-called courtesy or guest authorships that seem to bother a lot of a people in science. If you get more papers produced, even when you divide by the number of authors on each paper, then this tends to suggest that authors are contributing additional science. The scatter plots even seem to show a fairly linear relationship so we can't argue that it tails off after some arbitrary cutoff of author numbers.

Another implication is for the purely personal. If we can generate more plots like this one across subfields or across PI characteristics (there may be something odd about the K99/R00 population of investigators for example), there may be a productivity line against which to compare ourselves. Do we (or the candidate) have more or fewer publications than would be predicted from the average number of authors? Does this suggest that you can identify slackers from larger labs (that happen to have a lot of pubs) and hard chargers from smaller labs (that have fewer total pubs, but excel against the expected value)?

15 responses so far

Pet Peeve: "the literature"

(by drugmonkey) Oct 22 2014

One precious tic of academic writing I implore you to avoid is "...in the literature".

"Novel contribution to the literature..."

"Unknown in the literature..."

"Fill a gap in the literature..."

You know what I mean.

The impression you create is that this is some silly self-referential game with only internal measures of importance.

Whether this is how you see science or not.....avoid creating this impression.

Talk about knowledge or understanding instead.

23 responses so far

Query for Readers: Are Blogrolls Over?

(by drugmonkey) Oct 21 2014

What do you think? Do you ever click on the blogroll anymore? Is this useful to any of you?

18 responses so far

NIH's rapid growth has let in a bunch of riff-raff!

(by drugmonkey) Oct 21 2014

I am sure Dr. McKnight realizes that when he asserts that "Biomedical research in the 1960s and 1970s was a spartan game" and "Biomedical research is a huge enterprise now; it attracts riff-raff who never would have survived as scientists in the 1960s and 1970s" he is in fact lauding the very scientists "When I joined the molecular cytology study section in the 1980s.. all kinds of superb scientists" who were the riff-raff the prior generation complained about.

From a very prestigious general Science journal in 1962:

Some of [this change] arises from expressions of concern within the scientific community itself over whether the NIH's rapid growth has sacrificed quality to achieve quantity.

The astute reader will also pick up on another familiar theme we are currently discussing.

And some of it reflects nothing more than the know-nothing ramblings of scientific illiterates, who conclude that if the title of a research project is not readily comprehensible to them, some effort to swindle the government must be involved.

1962, people. 1962.
__
Greenberg DS. NIH Grants: Policies Revised, but Critics Not Likely To Turn Away. Science. 1962 Dec 28;138(3548):1379-80.

3 responses so far

Congress is dissatisfied with NIH's spending priorities!

(by drugmonkey) Oct 21 2014

This passage appeared in a highly prestigious journal of science.

"Important elements in both Senate and the House are showing increasing dissatisfaction over Congress's decade-long honeymoon with medical research....critics are dissatisfied...with the NIH's procedures for supervising the use of money by its research grantees....NIH officials..argued, rather, that the most productive method in financing research is to pick good people with good projects and let them carry out their work without encumbering them...its growth has been phenomenal....[NIH director}: nor do we believe that most scientific groups in the country have an asking and a selling price for their product which is research activity...we get a realistic appraisal of what they need to do the job..the supervisory function properly belongs to the universities and other institutions where the research takes place....closing remarks of the report are:...Congress has been overzealous in appropriating money for health research".

Continue Reading »

2 responses so far

Bash Science with Gay AND Fat-shaming? It's like a rightwing three-fer

(by drugmonkey) Oct 20 2014

We recently discussed how the Origami Condom project supported under the Small Business Innovation Research Congressional mandate had quite obvious public health implications in a prior post. This was in response to the gleeful Republican bashing of NIH funding priorities in the wake of NIH Director Francis Collins' rather poorly considered claims* that Ebola research has been held back by the flatlining of the NIH budget over the past ten years.

Today we take on another one of these claims that the NIH has not been using its appropriations wisely. Fox news provides a handy example of the claim:

The National Institutes of Health (NIH) has spent more than $39 million on obese lesbians

As the wags are posting on various social media outlets, more Americans have been dumped by [insert popular entertainment personality] than have been killed by Ebola.

In striking contrast, obesity is a big killer of Americans. According to one review of the evidence:

Using data on all eligible subjects from all six studies, Allison et al. estimated that 280,184 obesity-attributable deaths occurred in the U.S. annually. When risk ratios calculated for nonsmokers and never-smokers were applied to the entire population (assuming these ratios to produce the best estimate for all subjects, regardless of smoking status, i.e., that obesity would exert the same deleterious effects across all smoking categories), the mean estimate for deaths due to obesity was 324,940.

Additional analyses were performed controlling for prevalent chronic disease at baseline using data from the CPS1 and NHS. After controlling for preexisting disease, the mean annual number of obesity-attributable deaths was estimated to be 374,239 (330,324 based on CPS1 data and 418,154 based on NHS data).

Over 350,000 Americans die annually of obesity. For the Republican Congresspersons in the audience, "annually" means every year. Last year, this year, next year. Over 350,000.

No biggie, right?
Whoops, maybe it is worse than we thought?

Researchers found that obesity accounted for nearly 20 percent of deaths among white and black Americans between the ages of 40 and 85. Previously, many scientists estimated that about 5 percent of deaths could be attributed to obesity.

And is coming close to beating smoking as the top preventable killer of American citizens?
Flegel et al 2004 and Flegel et al 2013 provide some handy context to estimating mortality causes for the nerdier types. From the 2013 meta-analysis:

[overweight (BMI of 25-<30), obesity (BMI of ≥30), grade 1 obesity (BMI of 30-<35), and grades 2 and 3 obesity (BMI of ≥35) ] ..
CONCLUSIONS AND RELEVANCE: Relative to normal weight, both obesity (all grades) and grades 2 and 3 obesity were associated with significantly higher all-cause mortality. Grade 1 obesity overall was not associated with higher mortality, and overweight was associated with significantly lower all-cause mortality.

So. Just this easily we can confirm that obesity is a major public health concern from mortality alone. This doesn't even get into non-mortal effect of obesity on personal well-being. Major public health concerns are the very province of NIH-funded academic research.

So once again, the applicability of grants that are targeted at reducing obesity (even if it is just understanding the causes of obesity) to the goals of the NIH, as mandated by Congress, is not in question. At all. This is not a frivolous expenditure.

That leaves us with the specific projects in question. I trotted over to RePORTER and pulled up 6 current awards- two are K-mechanism mentored training awards so we'll focus on the R-mechanism research projects.

R01 HD066963: SEXUAL ORIENTATION AND OBESITY: TEST OF A GENDERED BIOPSYCHOSOCIAL MODEL

nearly three-quarters of adult lesbians overweight or obese, compared to half of heterosexual women. In stark contrast, among men, heterosexual males have nearly double the risk of obesity compared to gay males. Despite clear evidence from descriptive epidemiologic research that sexual orientation and gender markedly pattern obesity disparities, there is almost no prospective, analytic epidemiologic research into the causes of these disparities. It will be impossible to develop evidence-based preventive interventions unless we first answer basic questions about causal pathways, as we plan to do.

I bolded a key part, from my perspective. You waste a ton of money, often public money, if you go off with solutions to problems without having a clear understanding of the things causing or following from this problem. Epidemiological and sociological research guides not just public policy but also additional studies of physiology, genetic liabilities, etc. So this specific project would seem to be of considerable use.

R01 DK099360:TYPE 2 DIABETES AND SEXUAL ORIENTATION DISPARITIES IN WOMEN

lesbian and bisexual (LB) women may be at elevated risk for developing T2D because they are more likely than heterosexual women to experience obesity and other risk factors linked with T2D such as cigarette smoking, violence victimization, and depressive distress. Nonetheless, knowledge of T2D and how it may disproportionately affect LB women is severely limited. Studies using longitudinal designs that have comprehensively examined how lifestyle, diet, and psychosocial risk factors for T2D may differ between LB and heterosexual women across the life course are virtually nonexistent.

This project emphasizes non-mortal morbidity, i.e., Type 2 Diabetes (T2D). And again, the abstract describes how we know almost nothing about the reasons for the obesity disparity between lesbian and heterosexual women. If we are going to disentangle potential social, behavioral, cultural, physiological and genetic contributors to the disparity, we need information. And very likely, through this research we will come to know more about how these variables affect obesity risk for all Americans, across all subpopulations. This will help us design better interventions to reduce the obesity burden. Clearly this is another grant that is clearly non-frivolous and fits into the public health mandate of the NIH.

R21 HD073120: UNDERSTANDING DISPARITIES IN OBESITY AND WEIGHT BEHAVIORS BY SEXUAL IDENTITY

Previous research indicates that lesbian, gay, bisexual and transgender (LGBT) adults experience more adverse health outcomes than their peers. Findings from the few studies examining weight disparities among adults suggest that lesbian women are more likely to be overweight or obese compared to their heterosexual peers, though less is known about gay men and bisexuals. Given the scant research to date in this area, the Institute of Medicine (IOM) recently issued a call for additional research on LGBT health. Furthermore, IOM highlighted the need to utilize a life-course framework when examining health disparities by sexual identity, acknowledging the unique influence of various life stages on health

What's this now? Even the US Institute of Medicine has reported on how important it is to combat obesity in US citizens? I mean dang, guys, it's the IOM.

The Institute of Medicine (IOM) is an independent, nonprofit organization that works outside of government to provide unbiased and authoritative advice to decision makers and the public.

Established in 1970, the IOM is the health arm of the National Academy of Sciences, which was chartered under President Abraham Lincoln in 1863. Nearly 150 years later, the National Academy of Sciences has expanded into what is collectively known as the National Academies, which comprises the National Academy of Sciences, the National Academy of Engineering, the National Research Council, and the IOM.

And they do investigations, review evidence, compare the facts...

anyway, this R21 is going to focus on young adults and do studies under the following Aims:

(1) Quantify disparities in obesity, dietary intake, physical activity, unhealthy weight control behavior, body satisfaction and other weight-related health outcomes among LGB and heterosexual students; (2) Identify major weight-related health behavioral patterns, or profiles, and the extent to which these behavioral profiles differ by sexual identity and gender; and (3) Characterize these behavioral profiles by demographic factors and health outcomes (e.g., age, socioeconomic status, health care coverage, obesity, and health status). We hypothesize that LGB students engage in more adverse behaviors than their heterosexual peers and exhibit differential behavioral patterning.

Yep, more psycho-social research but I continue to assert that without this evidence, we run the risk of wasting more money pursuing directions that could have been falsified by the epidemiological and social science studies of this type.

The final research project is an R15/AREA grant:
R15 AA020424: MINORITY STRESS, ALCOHOL USE, AND INTIMATE PARTNER VIOLENCE AMONG LESBIANS

Ok, going by the Abstract this one is indeed focused on Alcohol abuse and intimate partner violence and I don't see why it is being triggered by the obesity keyword on the search. But still, I think we can see that this one ALSO would draw right wing fire. Even though, once again, alcoholism and intimate partner violence are huge health issues in the US.

As with the Origami Condom NIH Grant, we can find with relatively little thinking that the "National Institutes of Health (NIH) has spent more than $39 million on obese lesbians" comment is wrongly placed in an article addressing "wasteful" spending on the part of the NIH. These projects address the causes of obesity, which is basically a top predator of Americans at the moment. Obesity causes excess mortality and morbidity, which is of course associated with financial costs. Costs to the individual and costs to us all as a society that shares some degree of social support for the health care of our fellow citizens. It is in our direct and obvious interests to conduct research that will help us reduce this burden of obesity. As far as studying subpopulations who appear to be at increased risk for obesity goes, there is no reason not to want to help African-Americans, Southern Americans, Flyoverlandia Americans or...Lesbian-Americans. Right? And while it may take a little bit of a leap of faith for those who haven't thought hard about it, understanding the causes of a major health condition in those other people over there helps to understand the causes in people who are just like ourselves. By subtraction if by no other means.

For my regular Readers I'll close with a plea. Use analysis like this one to beat back this stupid meme that is going around about "frivolous" NIH expenditures. This is not just about this current Ebola fervor. This is about the normal operations of the NIH as it has progressed over decades. There are always those wanting to score cheap political points by bashing science as trivial or obviously ridiculous. Nine times out of ten, these charges are easily rebutted. So take the time to do so, even if it just posting some text pulled from the grant abstract and a link to a morbidity report on whichever health concern happens to be under discussion.

__
*"poorly considered" meaning he didn't apparently anticipate handing such a bunch of base-bait to the Republicans.

26 responses so far

The Origami Condom and NIH Ebola funding

(by drugmonkey) Oct 16 2014

One of the NIH funded research projects that has been bandied about with much glee from the right wing, in the wake of Francis Collins' unfortunate assertion about Ebola research and the flatlined NIH budget, is the "Origami Condom". It shows why NIH Director Collins should have known better. The Origami Condom sounds trivial and ridiculous, right? "Origami". hahah. Oooh, "condom". Wait, what are we, 12 year olds?

Rand Paul provides a convenient example.
Continue Reading »

18 responses so far

Older posts »