Time to N-up!

(by drugmonkey) May 02 2018

Chatter on the Twitts today brought my attention to a paper by Weber and colleagues that had a rather startlingly honest admission.

Weber F, Hoang Do JP, Chung S, Beier KT, Bikov M, Saffari Doost M, Dan Y.Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons. Nat Commun. 2018 Jan 24;9(1):354. doi: 10.1038/s41467-017-02765-w.

If you page all the way down to the end of the Methods of this paper, you will find a statement on sample size determination. I took a brief stab at trying to find the author guidelines for Nature Communications because a standalone statement of how sample size was arrived upon is somewhat unusual to me. Not that I object, I just don't find this to be common in the journal articles that I read. I was unable to locate it quickly so..moving along to the main point of the day. The statement reads partially:

Sample sizes

For optogenetic activation experiments, cell-type-specific ablation experiments, and in vivo recordings (optrode recordings and calcium imaging), we continuously increased the number of animals until statistical significance was reached to support our conclusions.

Wow. WOW!

This flies in the face of everything I have ever understood about proper research design. In the ResearchDesign 101 approach, you determine* your ideal sample size in advance. You collect your data in essentially one go and then you conduct your analysis. You then draw your conclusions about whether the collected data support, or fail to support, rejection of a null hypothesis. This can then allow you to infer things about the hypothesis that is under investigation.

In the real world, we modify this a bit. And what I am musing today is why some of the ways that we stray from ResearchDesign orthodoxy are okay and some are not.

We talk colloquially about finding support for (or against) the hypothesis under investigation. We then proceed to discuss the results in terms of whether they tend to support a given interpretation of the state of the world or a different interpretation. We draw our conclusions from the available evidence- from our study and from related prior work. We are not, I would argue, supposed to be setting out to find the data that "support our conclusions" as mentioned above. It's a small thing and may simply reflect poor expression of the idea. Or it could be an accurate reflection that these authors really set out to do experiments until the right support for a priori conclusions has been obtained. This, you will recognize, is my central problem with people who say that they "storyboard" their papers. It sounds like a recipe for seeking support, rather than drawing conclusions. This way lies data fakery and fraud.

We also, importantly, make the best of partially successful experiments. We may conclude that there was such a technical flaw in the conduct of the experiment that it is not a good test of the null hypothesis. And essentially treat it in the Discussion section as inconclusive rather than a good test of the null hypothesis.

One of those technical flaws may be the failure to collect the ideal sample size, again as determined in advance*. So what do we do?

So one approach is simply to repeat the experiment correctly. To scrap all the prior data, put fixes in place to address the reasons for the technical failure, and run the experiment again. Even if the technical failure hit only a part of the experiment. If it affected only some of the "in vivo recordings", for example. Orthodox design mavens may say it is only kosher to re run the whole shebang.

In the real world, we often have scenarios where we attempt to replace the flawed data and combine it with the good data to achieve our target sample size. This appears to be more or less the space in which this paper is operating.

"N-up". Adding more replicates (cells, subjects, what have you) until you reach the desired target. Now, I would argue that re-running the experiment with the goal of reaching the target N that you determined in advance* is not that bad. It's the target. It's the goal of the experiment. Who cares if you messed up half of them every time you tried to run the experiment? Where "messed up" is some sort of defined technical failure rather than an outcome you don't like, I rush to emphasize!

On the other hand, if you are spamming out low-replicate "experiments" until one of the scenarios "looks promising", i.e. looks to support your desired conclusions, and selectively "n-up" that particular experiment, well this seems over the line to me. It is much more likely to result in false positives. Well, I suppose running all of these trial experiments at the full power is just as likely it is just that you are not able to do as many trial experiments at full power. So I would argue the sheer number of potential experiments is greater for the low-replicate, n-up-if-promising approach.

These authors appear to have done this strategy even one worse. Because their target is not just an a priori determined sample size to be achieved only when the pilot "looks promising". In this case they take the additional step of only running replicates up to the point where they reach statistical significance. And this seems like an additional way to get an extra helping of false-positive results to me.

Anyway, you can google up information on false positive rates and p-hacking and all that to convince yourself of the math. I was more interested in trying to probe why I got such a visceral feeling that this was not okay. Even if I personally think it is okay to re-run an experiment and combine replicates (subjects in my case) to reach the a priori sample size if it blows up and you have technical failure on half of the data.

__
*I believe the proper manner for determining sample size is entirely apart from the error the authors have admitted to here. This isn't about failing to complete a power analysis or the like.

27 responses so far

Diversity statements from faculty candidates are an unfair test of ideology?

(by drugmonkey) Apr 30 2018

Someone on the twitts posted an objection:

to UCSD's policy of requiring applicants for faculty positions to supply a Statement of Contribution to Diversity with their application.

Mark J Perry linked to his own blog piece posted at the American Enterprise Institute* with the following observation:

All applicants for faculty positions at UCSD now required to submit a Contribution to Diversity Statement (aka Ideological Conformity Statements/Pledge of Allegiance to Left-Liberal Orthodoxy Statements)

Then some other twitter person chimed in with opinion on how this policy was unfair because it was so difficult for him to help his postdocs students with it.


Huh? A simple google search lands us on UCSD's page on this topic.

The Contributions to Diversity Statement should describe your past efforts, as well as future plans to advance diversity, equity and inclusion. It should demonstrate an understanding of the barriers facing women and underrepresented minorities and of UC San Diego’s mission to meet the educational needs of our diverse student population.

The page has links to a full set of guidelines [PDF] as well as specific examples in Biology, Engineering and Physical Sciences (hmm, I wonder if these are the disciplines they find need the most help?). I took a look at the guidelines and examples. It's pretty easy sailing. Sorry, but any PI who is complaining that they cannot help their postdocs figure out how to write the required statement are lying being disingenuous. What they really mean is that they disagree with having to prepare such a statement at all.

Like this guy Bieniasz, points for honesty:


I am particularly perplexed with this assertion that "The UCSD statement instructions (Part A) read like a test of opinions/ideology. Not appropriate for a faculty application".

Ok, so is it a test of opinion/ideology? Let's go to the guidelines provided by UCSD.

Describe your understanding of the barriers that exist for historically under-represented groups in higher education and/or your field. This may be evidenced by personal experience and educational background. For purposes of evaluating contributions to diversity, under-represented groups (URGs) includes under-represented ethnic or racial minorities (URM), women, LGBTQ, first-generation college, people with disabilities, and people from underprivileged backgrounds.

Pretty simple. Are you able to understand facts that have been well established in academia? This only asks you to describe your understanding. That's it. If you are not aware of any of these barriers *cough*Ginther*cough*cough*, you are deficient as a candidate for a position as a University professor.

So the first part of this is merely asking if the candidate is aware of things about academia that are incredibly well documented. Facts. These are sort of important for Professors and any University is well within it's rights to probe factual knowledge. This part does not ask anything about causes or solutions.

Now the other parts do ask you about your past activities and future plans to contribute to diversity and equity. Significantly, it starts with this friendly acceptance: "Some faculty candidates may not have substantial past activities. If such cases, we recommend focusing on future plans in your statement.". See? It isn't a rule-out type of thing, it allows for candidates to realize their deficits right now and to make a statement about what they might do in the future.

Let's stop right there. This is not different in any way to the other major components of a professorial hire application package. For most of my audience, the "evidence of teaching experience and philosophy" is probably the more understandable example. Many postdocs with excellent science chops have pretty minimal teaching experience. Is it somehow unfair to ask them about their experience and philosophy? To give credit for those with experience and to ask those without to have at least thought about what they might do as a future professor?

Is it "liberal orthodoxy" if a person who insists that teaching is a waste of time and gets in the way of their real purpose (research) gets pushed downward on the priority list for the job?

What about service? Is it rude to ask a candidate for evidence of service to their Institutions and academic societies?

Is it unfair to prioritize candidates with a more complete record of accomplishment than those without? Of course it is fair.

What about scientific discipline, subfield, research orientations and theoretical underpinnings? Totally okay to ask candidates about these things.

Are those somehow "loyalty pledges"? or a requirement to "conform to orthodoxy"?

If they are, then we've been doing that in the academy a fair bit with nary a peep from these right wing think tank types.

__
*"Mark J. Perry is concurrently a scholar at AEI and a professor of economics and finance at the University of Michigan's Flint campus." This is a political opinion-making "think-tank" so take that into consideration.

84 responses so far

When Congress boosts the NIH budget midyear weird stuff happens

(by drugmonkey) Apr 19 2018

Interesting comment about NIGMS recent solicitation of supplement applications for capital equipment infrastructure.

If true this says some interesting things about whether NIH will ever do anything to reduce churn, increase paylines and generally make things more livable for their extramural workforce.

9 responses so far

Another day, another report on the postdocalypse

(by drugmonkey) Apr 13 2018

As mentioned in Science, a new report from the US Academies of Sciences, Engineering, and Medicine have deduced we have a problem with too many PhDs and not enough of the jobs that they want.

The report responds to many years of warning signs that the U.S. biomedical enterprise may be calcifying in ways that create barriers for the incoming generation of researchers. One of the biggest challenges is the gulf between the growing number of young scientists who are qualified for and interested in becoming academic researchers and the limited number of tenure-track research positions available. Many new Ph.D.s spend long periods in postdoctoral positions with low salaries, inadequate training, and little opportunity for independent research. Many postdocs pursue training experiences expecting that they will later secure an academic position, rather than pursuing a training experience that helps them compete for the range of independent careers available outside of academia, where the majority will be employed. As of 2016, for those researchers who do transition into independent research positions, the average age for securing their first major NIH independent grant is 43 years old, compared to 36 years old in 1980.

No mention (in the executive summary / PR blurb) that the age of first R01 has been essentially unchanged for a decade despite the NIH ESI policy and the invention of the K99 which is limited by years-since-PhD.

No mention of the reason that we have so many postdocs, which is the uncontrolled production of ever more PhDs.

On to the actionable bullet points that interest me.

Work with the National Institutes of Health to increase the number of individuals in staff scientist positions to provide more stable, non-faculty research opportunities for the next generation of researchers. Individuals on a staff scientist track should receive a salary and benefits commensurate with their experience and responsibilities.

This is a recommendation for research institutions but we all need to think about this. The NCI launched the R50 mechanism in 2016 and they have 49 of them on the books at the moment. I had some thoughts on why this is a good idea here and here. The question now, especially for those in the know with cancer research, is whether this R50 is being used to gain stability and independence for the needy awardee or whether it is just further larding up the labs of Very Important Cancer PIs.

Expand existing awards or create new competitive awards for postdoctoral researchers to advance their own independent research and support professional development toward an independent research career. By July 1, 2023, there should be a fivefold increase in the number of individual research fellowship awards and career development awards for postdoctoral researchers granted by NIH.

As we know the number of NIH fellowships has remained relatively fixed relative to the huge escalation of "postdocs" funded on research grant mechanisms. We really don't know the degree to which independent fellowships simply annoint the chosen (population wise) versus aid the most worthy and deserving candidates to stand out. Will quintupling the F32s magically make more faculty slots available? I tend to think not.

As we know, if you really want to grease the skids to faculty appointment the route is the K99/R00 or basically anything that means the prospective hire " comes with money". Work on that, NIH. Quintuple the K99s, not the F32s. And hand out more R03 or R21 or invent up some other R-mechanism that prospective faculty can apply for in place of "mentored" K awards. I just had this brainstorm. R-mechs (any really) that get some cutesy acronym (like B-START) and can be applied for by basically any non-faculty person from anywhere. Catch is, it works like the R00 part of the K99/R00. Only awarded upon successful competition for a faculty job and the offer of a competitive startup.

Ensure that the duration of all R01 research grants supporting early-stage investigators is no less than five years to enable the establishment of resilient independent research programs.

Sure. And invent up some "next award" special treatment for current ESI. and then a third-award one. and so on.

Or, you know, fix the problem for everyone which is that too many mouths at the trough have ruined the cakewalk that experienced investigators had during the eighties.

Phase in a cap – three years suggested – on salary support for all postdoctoral researchers funded by NIH research project grants (RPGs). The phase-in should occur only after NIH undertakes a robust pilot study of sufficient size and duration to assess the feasibility of this policy and provide opportunities to revise it. The pilot study should be coupled to action on the previous recommendation for an increase in individual awards.

This one got the newbie faculty all het up on the twitters.


and


being examples if you are interested.

They are, of course, upset about two things.

First, "the person like me". Which of course is what drives all of our anger about this whole garbage fire of a career situation that has developed. You can call it survivor guilt, self-love, arrogance, whatever. But it is perfectly reasonable that we don't like the Man doing things that mean people just like us would have washed out. So people who were not super stars in 3 years of postdoc'ing are mad.

Second, there's a hint of "don't stop the gravy train just as I passed your damn insurmountable hurdle". If you are newb faculty and read this and get all angree and start telling me how terrible I am.....you need to sit down an introspect a bit, friend. I can wait.

New faculty are almost universally against my suggestion that we all need to do our part and stop training graduate students. Less universally, but still frequently, against the idea that they should start structuring their career plans for a technician-heavy, trainee-light arrangement. With permanent career employees that do not get changed out for new ones every 3-5 years like leased Priuses either.

Our last little stupid poll confirmed that everyone things 3-5 concurrent postdocs is just peachy for even the newest lab and gee whillikers where are they to come from?

Aaaanyway.
This new report will go nowhere, just like all the previous ones that reach essentially the same conclusion and make similar recommendations. Because it is all about the

1) Mouths at the trough.
and
2) Available slops.

We continue to breed more mouths PHDs.

And the American taxpayers, via their duly appointed representatives in Congress, show no interest in radically increasing the budget for slops science.

And even if Congress trebled or quintupled the NIH budget, all evidence suggests we'd just to the same thing all over again. Mint more PhDs like crazee and wonder in another 10-15 years why careers still suck.

63 responses so far

Don't get too big for your britches, jr faculty: trainee edition

(by drugmonkey) Apr 10 2018

As you know I am not a super big fan of NIH grant review sentiments which boil down to "tut, tut, Dr. Junior Faculty, let's not get too big for your britches. Try this small starter award and see how you do with that before you get to play with the big kids."

I believe things like size of grant and number of grants (and relatedly, overall total direct costs) should be taken on a case by case basis. And I believe that modern "junior" faculty are pretty old, phenomenally broadly experienced and generally pretty capable compared to junior faculty minted in, say, the early to mid eighties.

The question of the day, however, has more to do with lab size and specifically to do with the number of academic trainees.

Is there a limit to the number of grad students, postdocs or grad students plus postdocs that most junior faculty should be training?

My gut take is "heck yes". I don't know that I've ever had to act up this. I can't recall a time when I ever had to judge a R-mechanism or F-mechanism where the PI or supervisor (respectively) was seemingly overburdened with trainees. But my gut says that this is possible. There would be times where I might raise an eyebrow about how many concurrent trainees a junior (or senior, but that's another argument) PI might be proposing to have. Whether that be due to taking a look at the "training environment" for a F32/F31 application or in looking at relative commitment levels for a new Rproposal there are seemingly times that this might come up. Conceivably.

My gut feeling on this is guided by my own experience which, as we know, is wildly out of touch with y'all.

We have had one or two conversations about what people think of as a small, medium or large lab. My takeaway from these is that people think a 6-7 person lab is average, medium, normal and basically expected value.

To me this is "on the larger side".

I have run anywhere from 0-4 concurrent academic trainees and when I am at 4 postdocs I definitely feel a bit stretched.

I have been doing this gig for some time now. When I was a wee newbie PI I thought that two concurrent trainees was pretty much good. Three was not something that I thought was sustainable.

Whatcha think, Dear Reader?

Can most junior PIs handle 5 or more concurrent academic trainees? Should they just take as many as possible?

__
*I solemnly swear this is not a troll to further complain about the training of too many PhDs.

24 responses so far

The Alleged Profession of Journalism Sleazy Techniques Strike Again.

(by drugmonkey) Apr 04 2018

One of the nastiest things that the alleged profession of journalism has been caught doing is photoshopping pictures to engage the prejudices of their readers. Probably the most famous of these was when TIME was caught darkening the appearance of OJ Simpson's mugshot during his murder trial.

In June of 1994, in the midst of OJ Simpson’s murder trial, both TIME magazine and Newsweek featured Simpson’s mugshot on their covers.
...
The two magazines were placed side by side on newsstands and the public immediately saw that TIME’s cover had considerably darkened Simpson’s skin. The photo, representing a case already laced with racial tension, caused massive public outcry.

In this they walk in lockstep with the sorts of sleazy tricks played by political advertising geniuses such as those that tried to play on racial prejudice in opposing President Obama.

Campaign ads have used darker images of Obama to appeal to voters' racial prejudice, a new study has revealed.

Researchers analyzed 126 ads from the campaign in 2008, and found that digital editing had changed the appearances of both Barack Obama and Republican opponent John McCain.

Sometimes they appeared more washed out, but the McCain campaign often used images in which Obama's skin appeared darker when they were attempting to link him with crime.

I was struck by the image used recently on STAT to head an article on the Director of the NIAAA, George Koob*.

Looks kinda sinister to me. The article, by Sharon Begley and Andrew Joseph, is one of a pair (so far) of articles which appear to be accusing Koob of being under the sway of the beverage industry to the extent that it is influencing what grants he approves for funding as NIAAA Director. That's a topic for another post, perhaps, but the issue of today is the sleazy way that the alleged profession of journalism is fully willing to use pictures to create an impression consistent with their accusations. Just the way TIME did with the OJ mugshot. Just the way Republican political operatives did with pictures of President Obama.

The goal is to engage the prejudices of the reader so as to push them down the road to believing the case that you are supposedly making on more objective grounds.

Here's what a quick Google image search has to say about Koob's appearance.
[click to enlarge]

You can compare the distribution of Koob's appearances to the one included in the STAT piece for yourself.

Now, where did STAT get the image? STAT credits it to themselves as an "illustration" and it looks sourced from an AP credited photo from this article in japantimes.com. So yes, presumably their art department combed the web to find the picture that they wanted to use, selecting it from among all the available pictures of their subject, and then pshopped it into this "illustration".

Point being that they chose this particular image out of many. It's intentional.
_
*Disclaimer: I've been professionally acquainted with Koob since about 1991, at times fairly well-acquainted. I've heard him hold forth on the problems of alcohol and other substance misuse/dependence/addiction numerous times and have read a fair number of his reviews. I find him to be a pretty good guy, overall, with a keen intent to reduce the suffering associated with alcoholism and other substance dependencies. These recent accusations that he is somehow under the sway of the beverage industry strike me as really discordant with my experience of him over the past 27 years. Take my comments on this topic with that in mind.

24 responses so far

Ludicrous academics for $200, Alex

(by drugmonkey) Apr 02 2018

Just when I think I will not find any more ridiculous things hiding in academia.....

A recent thread on twitter addressed a population of academics (not sure if it was science) who are distressed when the peer review of their manuscripts is insufficiently vigorous/critical.

This is totally outside of my experience. I can't imagine ever complaining to an Editor of a journal that the review was too soft after getting an accept or invitation to revise.

People are weird though.

5 responses so far

Question of the Day

(by drugmonkey) Apr 02 2018

How do you assess whether you are too biased about a professional colleague and/or their work?

In the sense that you would self-elect out of reviewing either their manuscripts for publication or their grant applications.

Does your threshold differ for papers versus grants?

Do you distinguish between antipathy bias and sympathy bias?

8 responses so far

NIH to crack down on violations of confidential peer review

(by drugmonkey) Mar 30 2018

Nakamura is quoted in a recent bit in Science by Jeffrey Brainard.

I'll get back to this later but for now consider it an open thread on your experiences. (Please leave off the specific naming unless the event got published somewhere.)

I have twice had other PIs tell me they reviewed my grant. I did not take it as any sort of quid pro quo beyond *maybe* a sort of "I wasn't the dick reviewer" sort of thing. In both cases I barely acknowledged and tried to move along. These were both scientists that I like both professionally and personally so I assume I already have some pro-them bias. Obviously the fact these people occurred on the review roster, and that they have certain expertise, made them top suspects in my mind anyway.

Updated:

“We hope that in the next few months we will have several cases” of violations that can be shared publicly, Nakamura told ScienceInsider. He said these cases are “rare, but it is very important that we make it even more rare.”

Naturally we wish to know how "rare" and what severity of violation he means.

Nakamura said. “There was an attempt to influence the outcome of the review,” he said. The effect on the outcome “was sufficiently ambiguous that we felt it was necessary to redo the reviews.”

Hmmm. "Ambiguous". I mean, if there is ever *any* contact from an applicant PI to a reviewer on the relevant panel it could be viewed as an attempt to influence outcome. Even an invitation to give a seminar or invitation to join a symposium panel proposal could be viewed as currying favor. Since one never knows how an implicit or explicit bias is formed, how would it ever be anything other than ambiguous? But if this is something clearly actionable by the NIH doesn't it imply some harder evidence? A clearer quid pro quo?

Nakamura also described the types of violations of confidentiality NIH has detected. They included “reciprocal favors,” he said, using a term that is generally understood to mean a favor offered by a grant applicant to a reviewer in exchange for a favorable evaluation of their proposal.

I have definitely heard a few third hand reports of this in the past. Backed up by a forwarded email* in at least one case. Wonder if it was one of these type of cases?

Applicants also learned the “initial scores” they received on a proposal, Nakamura said, and the names of the reviewers who had been assigned to their proposal before a review meeting took place.

I can imagine this happening** and it is so obviously wrong, even if it doesn't directly influence the outcome for that given grant. I can, however, see the latter rationale being used as self-excuse. Don't.

Nakamura said. “In the past year there has been an internal decision to pursue more cases and publicize them more.” He would not say what triggered the increased oversight, nor when NIH might release more details.

This is almost, but not quite, an admission that NIH is vaguely aware of a ground current of violations of the confidentiality of review. And that they also are aware that they have not pursued such cases as deeply as they should. So if any of you have ever notified an SRO of a violation and seen no apparent result, perhaps you should be heartened.

oh and one last thing:

In one case, Nakamura said, a scientific review officer—an NIH staff member who helps run a review panel—inappropriately changed the score that peer reviewers had given a proposal.

SROs and Program Officers may also have dirt on their hands. Terrifying prospect for any applicant. And I rush to say that I have always seen both SROs and POs that I have dealt with directly to be upstanding people trying to do their best to ensure fair treatment of grant applications. I may disagree with their approaches and priorities now and again but I've never had reason to suspect real venality. However. Let us not be too naive, eh?

_
*anyone bold enough to put this in email....well I would suspect this is chronic behavior from this person?

**we all want to bench race the process and demystify it for our friends. I can see many entirely well-intentioned reasons someone would want to tell their friend about the score ranges. Maybe even a sentiment that someone should be warned to request certain reviewers be excluded from reviewing their proposals in the future. But..... no. No, no, no. Do not do this.

29 responses so far

PI seeks postdoc

(by drugmonkey) Mar 23 2018

Every PI wants only the most brilliant, creative and motivated trainees that will put in insane levels of effort to advance the lab agenda.

We know this because it is how they write their postdoc solicitation blurbs.

This is not what is consistently available.

I know this because a consistent backchannel theme of my dubious life online as science careers nerd features PIs complaining about their trainees.

My usual response is to point out that they became PI due to being much better than average. So of course most of their trainees aren't going to be as good as they are*.

__

*were

17 responses so far

« Newer posts Older posts »