NIH's long sordid history of failing to launch new investigators fairly and cleanly

(by drugmonkey) May 03 2018

Actually, they call it "A History of Commitment"

It starts with the launch of the R23 in 1977, covers the invention and elimination of the R29 FIRST and goes all the way to the 2017 announcement that prior ESI still need help, this time for their second and third rounds of funding as "Early Established Investigators".

pssst, guys. FIX STUDY SECTIONS and PO BEHAVIOR.

Updated to add:
Mike Lauer is wringing his hands on the blog about The Issue that (allegedly) keeps us (NIH officialdom) awake at night [needs citation].

We pledge to do everything we can to incorporate those recommendations, along with those of the NASEM panel, in our ongoing efforts to design, test, implement, and evaluate policies that will assure the success of the next generation of talented biomedical researchers.

5 responses so far

NIH reminds Universities not to keep paying harasser PIs from grant funds while suspended

(by drugmonkey) May 03 2018

On the May 1, 2018 the NIH issued NOT-OD-18-172 to clarify that:

NIH seeks to remind the extramural community that prior approval is required anytime there is a change in status of the PD/PI or other senior/key personnel where that change will impact his/her ability to carry out the approved research at the location of, and on behalf of, the recipient institution. In particular, changes in status of the PI or other senior/key personnel requiring prior approval would include restrictions that the institution imposes on such individuals after the time of award, including but not limited to any restrictions on access to the institution or to the institution’s resources, or changes in their (employment or leave) status at the institution. These changes may impact the ability of the PD/PI or other senior/key personnel to effectively contribute to the project as described in the application; therefore, NIH prior approval is necessary to ensure that the changes are acceptable.

Hard on the heels of the news breaking about long term and very well-funded NIH grant Principal Investigators Thomas Jessel and Inder Verma being suspended from duties at Columbia University and The Salk Institute for Biological Studies, respectively, one cannot help but draw the obvious conclusion.

I don't know what prompted this Notice but I welcome it.

Now, I realize that many of us would prefer to see some harsher stuff here. Changing the PI of a grant still keeps the sweet sweet indirects flowing into the University or Institute. So there is really no punishment when an applicant institution is proven to have looked the other way for years (decades) when their well-funded PIs are accused repeatedly of sexual harassment, gender-based discrimination, retaliation on whistleblowers and the like.

But this Notice is still welcome. It indicates that perhaps someone is actually paying a tiny little bit of attention now in this post-Weinstein era.

3 responses so far

Time to N-up!

(by drugmonkey) May 02 2018

Chatter on the Twitts today brought my attention to a paper by Weber and colleagues that had a rather startlingly honest admission.

Weber F, Hoang Do JP, Chung S, Beier KT, Bikov M, Saffari Doost M, Dan Y.Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons. Nat Commun. 2018 Jan 24;9(1):354. doi: 10.1038/s41467-017-02765-w.

If you page all the way down to the end of the Methods of this paper, you will find a statement on sample size determination. I took a brief stab at trying to find the author guidelines for Nature Communications because a standalone statement of how sample size was arrived upon is somewhat unusual to me. Not that I object, I just don't find this to be common in the journal articles that I read. I was unable to locate it quickly so..moving along to the main point of the day. The statement reads partially:

Sample sizes

For optogenetic activation experiments, cell-type-specific ablation experiments, and in vivo recordings (optrode recordings and calcium imaging), we continuously increased the number of animals until statistical significance was reached to support our conclusions.

Wow. WOW!

This flies in the face of everything I have ever understood about proper research design. In the ResearchDesign 101 approach, you determine* your ideal sample size in advance. You collect your data in essentially one go and then you conduct your analysis. You then draw your conclusions about whether the collected data support, or fail to support, rejection of a null hypothesis. This can then allow you to infer things about the hypothesis that is under investigation.

In the real world, we modify this a bit. And what I am musing today is why some of the ways that we stray from ResearchDesign orthodoxy are okay and some are not.

We talk colloquially about finding support for (or against) the hypothesis under investigation. We then proceed to discuss the results in terms of whether they tend to support a given interpretation of the state of the world or a different interpretation. We draw our conclusions from the available evidence- from our study and from related prior work. We are not, I would argue, supposed to be setting out to find the data that "support our conclusions" as mentioned above. It's a small thing and may simply reflect poor expression of the idea. Or it could be an accurate reflection that these authors really set out to do experiments until the right support for a priori conclusions has been obtained. This, you will recognize, is my central problem with people who say that they "storyboard" their papers. It sounds like a recipe for seeking support, rather than drawing conclusions. This way lies data fakery and fraud.

We also, importantly, make the best of partially successful experiments. We may conclude that there was such a technical flaw in the conduct of the experiment that it is not a good test of the null hypothesis. And essentially treat it in the Discussion section as inconclusive rather than a good test of the null hypothesis.

One of those technical flaws may be the failure to collect the ideal sample size, again as determined in advance*. So what do we do?

So one approach is simply to repeat the experiment correctly. To scrap all the prior data, put fixes in place to address the reasons for the technical failure, and run the experiment again. Even if the technical failure hit only a part of the experiment. If it affected only some of the "in vivo recordings", for example. Orthodox design mavens may say it is only kosher to re run the whole shebang.

In the real world, we often have scenarios where we attempt to replace the flawed data and combine it with the good data to achieve our target sample size. This appears to be more or less the space in which this paper is operating.

"N-up". Adding more replicates (cells, subjects, what have you) until you reach the desired target. Now, I would argue that re-running the experiment with the goal of reaching the target N that you determined in advance* is not that bad. It's the target. It's the goal of the experiment. Who cares if you messed up half of them every time you tried to run the experiment? Where "messed up" is some sort of defined technical failure rather than an outcome you don't like, I rush to emphasize!

On the other hand, if you are spamming out low-replicate "experiments" until one of the scenarios "looks promising", i.e. looks to support your desired conclusions, and selectively "n-up" that particular experiment, well this seems over the line to me. It is much more likely to result in false positives. Well, I suppose running all of these trial experiments at the full power is just as likely it is just that you are not able to do as many trial experiments at full power. So I would argue the sheer number of potential experiments is greater for the low-replicate, n-up-if-promising approach.

These authors appear to have done this strategy even one worse. Because their target is not just an a priori determined sample size to be achieved only when the pilot "looks promising". In this case they take the additional step of only running replicates up to the point where they reach statistical significance. And this seems like an additional way to get an extra helping of false-positive results to me.

Anyway, you can google up information on false positive rates and p-hacking and all that to convince yourself of the math. I was more interested in trying to probe why I got such a visceral feeling that this was not okay. Even if I personally think it is okay to re-run an experiment and combine replicates (subjects in my case) to reach the a priori sample size if it blows up and you have technical failure on half of the data.

__
*I believe the proper manner for determining sample size is entirely apart from the error the authors have admitted to here. This isn't about failing to complete a power analysis or the like.

27 responses so far

Diversity statements from faculty candidates are an unfair test of ideology?

(by drugmonkey) Apr 30 2018

Someone on the twitts posted an objection:

to UCSD's policy of requiring applicants for faculty positions to supply a Statement of Contribution to Diversity with their application.

Mark J Perry linked to his own blog piece posted at the American Enterprise Institute* with the following observation:

All applicants for faculty positions at UCSD now required to submit a Contribution to Diversity Statement (aka Ideological Conformity Statements/Pledge of Allegiance to Left-Liberal Orthodoxy Statements)

Then some other twitter person chimed in with opinion on how this policy was unfair because it was so difficult for him to help his postdocs students with it.


Huh? A simple google search lands us on UCSD's page on this topic.

The Contributions to Diversity Statement should describe your past efforts, as well as future plans to advance diversity, equity and inclusion. It should demonstrate an understanding of the barriers facing women and underrepresented minorities and of UC San Diego’s mission to meet the educational needs of our diverse student population.

The page has links to a full set of guidelines [PDF] as well as specific examples in Biology, Engineering and Physical Sciences (hmm, I wonder if these are the disciplines they find need the most help?). I took a look at the guidelines and examples. It's pretty easy sailing. Sorry, but any PI who is complaining that they cannot help their postdocs figure out how to write the required statement are lying being disingenuous. What they really mean is that they disagree with having to prepare such a statement at all.

Like this guy Bieniasz, points for honesty:


I am particularly perplexed with this assertion that "The UCSD statement instructions (Part A) read like a test of opinions/ideology. Not appropriate for a faculty application".

Ok, so is it a test of opinion/ideology? Let's go to the guidelines provided by UCSD.

Describe your understanding of the barriers that exist for historically under-represented groups in higher education and/or your field. This may be evidenced by personal experience and educational background. For purposes of evaluating contributions to diversity, under-represented groups (URGs) includes under-represented ethnic or racial minorities (URM), women, LGBTQ, first-generation college, people with disabilities, and people from underprivileged backgrounds.

Pretty simple. Are you able to understand facts that have been well established in academia? This only asks you to describe your understanding. That's it. If you are not aware of any of these barriers *cough*Ginther*cough*cough*, you are deficient as a candidate for a position as a University professor.

So the first part of this is merely asking if the candidate is aware of things about academia that are incredibly well documented. Facts. These are sort of important for Professors and any University is well within it's rights to probe factual knowledge. This part does not ask anything about causes or solutions.

Now the other parts do ask you about your past activities and future plans to contribute to diversity and equity. Significantly, it starts with this friendly acceptance: "Some faculty candidates may not have substantial past activities. If such cases, we recommend focusing on future plans in your statement.". See? It isn't a rule-out type of thing, it allows for candidates to realize their deficits right now and to make a statement about what they might do in the future.

Let's stop right there. This is not different in any way to the other major components of a professorial hire application package. For most of my audience, the "evidence of teaching experience and philosophy" is probably the more understandable example. Many postdocs with excellent science chops have pretty minimal teaching experience. Is it somehow unfair to ask them about their experience and philosophy? To give credit for those with experience and to ask those without to have at least thought about what they might do as a future professor?

Is it "liberal orthodoxy" if a person who insists that teaching is a waste of time and gets in the way of their real purpose (research) gets pushed downward on the priority list for the job?

What about service? Is it rude to ask a candidate for evidence of service to their Institutions and academic societies?

Is it unfair to prioritize candidates with a more complete record of accomplishment than those without? Of course it is fair.

What about scientific discipline, subfield, research orientations and theoretical underpinnings? Totally okay to ask candidates about these things.

Are those somehow "loyalty pledges"? or a requirement to "conform to orthodoxy"?

If they are, then we've been doing that in the academy a fair bit with nary a peep from these right wing think tank types.

__
*"Mark J. Perry is concurrently a scholar at AEI and a professor of economics and finance at the University of Michigan's Flint campus." This is a political opinion-making "think-tank" so take that into consideration.

84 responses so far

When Congress boosts the NIH budget midyear weird stuff happens

(by drugmonkey) Apr 19 2018

Interesting comment about NIGMS recent solicitation of supplement applications for capital equipment infrastructure.

If true this says some interesting things about whether NIH will ever do anything to reduce churn, increase paylines and generally make things more livable for their extramural workforce.

9 responses so far

Another day, another report on the postdocalypse

(by drugmonkey) Apr 13 2018

As mentioned in Science, a new report from the US Academies of Sciences, Engineering, and Medicine have deduced we have a problem with too many PhDs and not enough of the jobs that they want.

The report responds to many years of warning signs that the U.S. biomedical enterprise may be calcifying in ways that create barriers for the incoming generation of researchers. One of the biggest challenges is the gulf between the growing number of young scientists who are qualified for and interested in becoming academic researchers and the limited number of tenure-track research positions available. Many new Ph.D.s spend long periods in postdoctoral positions with low salaries, inadequate training, and little opportunity for independent research. Many postdocs pursue training experiences expecting that they will later secure an academic position, rather than pursuing a training experience that helps them compete for the range of independent careers available outside of academia, where the majority will be employed. As of 2016, for those researchers who do transition into independent research positions, the average age for securing their first major NIH independent grant is 43 years old, compared to 36 years old in 1980.

No mention (in the executive summary / PR blurb) that the age of first R01 has been essentially unchanged for a decade despite the NIH ESI policy and the invention of the K99 which is limited by years-since-PhD.

No mention of the reason that we have so many postdocs, which is the uncontrolled production of ever more PhDs.

On to the actionable bullet points that interest me.

Work with the National Institutes of Health to increase the number of individuals in staff scientist positions to provide more stable, non-faculty research opportunities for the next generation of researchers. Individuals on a staff scientist track should receive a salary and benefits commensurate with their experience and responsibilities.

This is a recommendation for research institutions but we all need to think about this. The NCI launched the R50 mechanism in 2016 and they have 49 of them on the books at the moment. I had some thoughts on why this is a good idea here and here. The question now, especially for those in the know with cancer research, is whether this R50 is being used to gain stability and independence for the needy awardee or whether it is just further larding up the labs of Very Important Cancer PIs.

Expand existing awards or create new competitive awards for postdoctoral researchers to advance their own independent research and support professional development toward an independent research career. By July 1, 2023, there should be a fivefold increase in the number of individual research fellowship awards and career development awards for postdoctoral researchers granted by NIH.

As we know the number of NIH fellowships has remained relatively fixed relative to the huge escalation of "postdocs" funded on research grant mechanisms. We really don't know the degree to which independent fellowships simply annoint the chosen (population wise) versus aid the most worthy and deserving candidates to stand out. Will quintupling the F32s magically make more faculty slots available? I tend to think not.

As we know, if you really want to grease the skids to faculty appointment the route is the K99/R00 or basically anything that means the prospective hire " comes with money". Work on that, NIH. Quintuple the K99s, not the F32s. And hand out more R03 or R21 or invent up some other R-mechanism that prospective faculty can apply for in place of "mentored" K awards. I just had this brainstorm. R-mechs (any really) that get some cutesy acronym (like B-START) and can be applied for by basically any non-faculty person from anywhere. Catch is, it works like the R00 part of the K99/R00. Only awarded upon successful competition for a faculty job and the offer of a competitive startup.

Ensure that the duration of all R01 research grants supporting early-stage investigators is no less than five years to enable the establishment of resilient independent research programs.

Sure. And invent up some "next award" special treatment for current ESI. and then a third-award one. and so on.

Or, you know, fix the problem for everyone which is that too many mouths at the trough have ruined the cakewalk that experienced investigators had during the eighties.

Phase in a cap – three years suggested – on salary support for all postdoctoral researchers funded by NIH research project grants (RPGs). The phase-in should occur only after NIH undertakes a robust pilot study of sufficient size and duration to assess the feasibility of this policy and provide opportunities to revise it. The pilot study should be coupled to action on the previous recommendation for an increase in individual awards.

This one got the newbie faculty all het up on the twitters.


and


being examples if you are interested.

They are, of course, upset about two things.

First, "the person like me". Which of course is what drives all of our anger about this whole garbage fire of a career situation that has developed. You can call it survivor guilt, self-love, arrogance, whatever. But it is perfectly reasonable that we don't like the Man doing things that mean people just like us would have washed out. So people who were not super stars in 3 years of postdoc'ing are mad.

Second, there's a hint of "don't stop the gravy train just as I passed your damn insurmountable hurdle". If you are newb faculty and read this and get all angree and start telling me how terrible I am.....you need to sit down an introspect a bit, friend. I can wait.

New faculty are almost universally against my suggestion that we all need to do our part and stop training graduate students. Less universally, but still frequently, against the idea that they should start structuring their career plans for a technician-heavy, trainee-light arrangement. With permanent career employees that do not get changed out for new ones every 3-5 years like leased Priuses either.

Our last little stupid poll confirmed that everyone things 3-5 concurrent postdocs is just peachy for even the newest lab and gee whillikers where are they to come from?

Aaaanyway.
This new report will go nowhere, just like all the previous ones that reach essentially the same conclusion and make similar recommendations. Because it is all about the

1) Mouths at the trough.
and
2) Available slops.

We continue to breed more mouths PHDs.

And the American taxpayers, via their duly appointed representatives in Congress, show no interest in radically increasing the budget for slops science.

And even if Congress trebled or quintupled the NIH budget, all evidence suggests we'd just to the same thing all over again. Mint more PhDs like crazee and wonder in another 10-15 years why careers still suck.

63 responses so far

Don't get too big for your britches, jr faculty: trainee edition

(by drugmonkey) Apr 10 2018

As you know I am not a super big fan of NIH grant review sentiments which boil down to "tut, tut, Dr. Junior Faculty, let's not get too big for your britches. Try this small starter award and see how you do with that before you get to play with the big kids."

I believe things like size of grant and number of grants (and relatedly, overall total direct costs) should be taken on a case by case basis. And I believe that modern "junior" faculty are pretty old, phenomenally broadly experienced and generally pretty capable compared to junior faculty minted in, say, the early to mid eighties.

The question of the day, however, has more to do with lab size and specifically to do with the number of academic trainees.

Is there a limit to the number of grad students, postdocs or grad students plus postdocs that most junior faculty should be training?

My gut take is "heck yes". I don't know that I've ever had to act up this. I can't recall a time when I ever had to judge a R-mechanism or F-mechanism where the PI or supervisor (respectively) was seemingly overburdened with trainees. But my gut says that this is possible. There would be times where I might raise an eyebrow about how many concurrent trainees a junior (or senior, but that's another argument) PI might be proposing to have. Whether that be due to taking a look at the "training environment" for a F32/F31 application or in looking at relative commitment levels for a new Rproposal there are seemingly times that this might come up. Conceivably.

My gut feeling on this is guided by my own experience which, as we know, is wildly out of touch with y'all.

We have had one or two conversations about what people think of as a small, medium or large lab. My takeaway from these is that people think a 6-7 person lab is average, medium, normal and basically expected value.

To me this is "on the larger side".

I have run anywhere from 0-4 concurrent academic trainees and when I am at 4 postdocs I definitely feel a bit stretched.

I have been doing this gig for some time now. When I was a wee newbie PI I thought that two concurrent trainees was pretty much good. Three was not something that I thought was sustainable.

Whatcha think, Dear Reader?

Can most junior PIs handle 5 or more concurrent academic trainees? Should they just take as many as possible?

__
*I solemnly swear this is not a troll to further complain about the training of too many PhDs.

24 responses so far

The Alleged Profession of Journalism Sleazy Techniques Strike Again.

(by drugmonkey) Apr 04 2018

One of the nastiest things that the alleged profession of journalism has been caught doing is photoshopping pictures to engage the prejudices of their readers. Probably the most famous of these was when TIME was caught darkening the appearance of OJ Simpson's mugshot during his murder trial.

In June of 1994, in the midst of OJ Simpson’s murder trial, both TIME magazine and Newsweek featured Simpson’s mugshot on their covers.
...
The two magazines were placed side by side on newsstands and the public immediately saw that TIME’s cover had considerably darkened Simpson’s skin. The photo, representing a case already laced with racial tension, caused massive public outcry.

In this they walk in lockstep with the sorts of sleazy tricks played by political advertising geniuses such as those that tried to play on racial prejudice in opposing President Obama.

Campaign ads have used darker images of Obama to appeal to voters' racial prejudice, a new study has revealed.

Researchers analyzed 126 ads from the campaign in 2008, and found that digital editing had changed the appearances of both Barack Obama and Republican opponent John McCain.

Sometimes they appeared more washed out, but the McCain campaign often used images in which Obama's skin appeared darker when they were attempting to link him with crime.

I was struck by the image used recently on STAT to head an article on the Director of the NIAAA, George Koob*.

Looks kinda sinister to me. The article, by Sharon Begley and Andrew Joseph, is one of a pair (so far) of articles which appear to be accusing Koob of being under the sway of the beverage industry to the extent that it is influencing what grants he approves for funding as NIAAA Director. That's a topic for another post, perhaps, but the issue of today is the sleazy way that the alleged profession of journalism is fully willing to use pictures to create an impression consistent with their accusations. Just the way TIME did with the OJ mugshot. Just the way Republican political operatives did with pictures of President Obama.

The goal is to engage the prejudices of the reader so as to push them down the road to believing the case that you are supposedly making on more objective grounds.

Here's what a quick Google image search has to say about Koob's appearance.
[click to enlarge]

You can compare the distribution of Koob's appearances to the one included in the STAT piece for yourself.

Now, where did STAT get the image? STAT credits it to themselves as an "illustration" and it looks sourced from an AP credited photo from this article in japantimes.com. So yes, presumably their art department combed the web to find the picture that they wanted to use, selecting it from among all the available pictures of their subject, and then pshopped it into this "illustration".

Point being that they chose this particular image out of many. It's intentional.
_
*Disclaimer: I've been professionally acquainted with Koob since about 1991, at times fairly well-acquainted. I've heard him hold forth on the problems of alcohol and other substance misuse/dependence/addiction numerous times and have read a fair number of his reviews. I find him to be a pretty good guy, overall, with a keen intent to reduce the suffering associated with alcoholism and other substance dependencies. These recent accusations that he is somehow under the sway of the beverage industry strike me as really discordant with my experience of him over the past 27 years. Take my comments on this topic with that in mind.

24 responses so far

Ludicrous academics for $200, Alex

(by drugmonkey) Apr 02 2018

Just when I think I will not find any more ridiculous things hiding in academia.....

A recent thread on twitter addressed a population of academics (not sure if it was science) who are distressed when the peer review of their manuscripts is insufficiently vigorous/critical.

This is totally outside of my experience. I can't imagine ever complaining to an Editor of a journal that the review was too soft after getting an accept or invitation to revise.

People are weird though.

5 responses so far

Question of the Day

(by drugmonkey) Apr 02 2018

How do you assess whether you are too biased about a professional colleague and/or their work?

In the sense that you would self-elect out of reviewing either their manuscripts for publication or their grant applications.

Does your threshold differ for papers versus grants?

Do you distinguish between antipathy bias and sympathy bias?

8 responses so far

« Newer posts Older posts »