Personal jihads and distinguishing better/worse science from wrong science

Jul 22 2016 Published by under Conduct of Science

This is relevant to posts by a Russ Poldrack flagellating himself for apparent methodological lapses in fMRI analysis.

The fun issue is as summarized in the recent post:

Student [commenter on original post] is exactly right that I have been a coauthor on papers using methods or reporting standards that I now publicly claim to be inappropriate. S/he is also right that my career has benefited substantially from papers published in high profile journals prior using these methods that I now claim to inappropriate. ... I am in agreement that some of my papers in the past used methods or standards that we would now find problematic...I also appreciate Student's frustration with the fact that someone like myself can become prominent doing studies that are seemingly lacking according to today's standards, but then criticize the field for doing the same thing.

I made a few comments on the Twitts to the effect that this is starting to smell of odious ladder pulling behavior.

One key point from the original post:

I would note that points 2-4 were basically standard practice in fMRI analysis 10 years ago (and still crop up fairly often today).

And now let us review the original critiques to which he is referring:

  • There was no dyslexic control group; thus, we don't know whether any improvements over time were specific to the treatment, or would have occurred with a control treatment or even without any treatment.
  • The brain imaging data were thresholded using an uncorrected threshold.
  • One of the main conclusions (the "normalization" of activation following training") is not supported by the necessary interaction statistic, but rather by a visual comparison of maps.
  • The correlation between changes in language scores and activation was reported for only one of the many measures, and it appeared to have been driven by outliers.

As I have mentioned on more than one occasion I am one that finds value in the humblest papers and in the single reported experiment. Often times it is such tiny, tiny threads of evidence that helps our science and the absence of any information on something whatever that hinders us.

I find myself mostly able to determine whether the proper controls were used. More importantly, I find myself more swayed by the strength of the data and the experiment presented than I am by the claims made in the Abstract or Discussion about the meaning of the reported work. I'd rather be in a state of "huh, maybe this thing might be true (or false), pending these additional controls that need to be done" then a state of "dammit, why is there no information whatsoever on this thing I want to know about right now".

Yes, absolutely, I think that there are scientific standards that should be generally adhered to. I think the PSY105: Experimental Design (or similar) principles regarding the perfect experiment should be taken seriously....as aspirations.

But I think the notion that you "can't publish that" because of some failure to attain the Gold Plated Aspiration of experimental design is stupid and harmful to science as a hard and fast rule. Everything, but everything, should be reviewed by the peers considering a manuscript for publication intelligently and thoughtfully. In essence, taken on it's merits. This is much as I take any published data on their own merits when deciding what I think they mean.

This is particularly the case when we start to think about the implications for career arcs and the limited resources that affect our business.

It is axiomatic that not everyone has the same interests, approaches and contingencies that affect their publication practices. This is a good thing, btw. In diversity there is strength. We've talked most recently around these parts about LPU incrementalism versus complete stories. We've talked about rapid vertical ascent versus riff-raff. Open Science Eleventy versus normal people. The GlamHounds versus small town grocers. ...and we almost invariably start in on how subfields differ in any of these discussions. etc.

Threaded through many of these conversations is the notion of gate keeping. Of defining who gets to play in the sandbox on the basis of certain standards for how they conduct their science. What tools they use. What problems they address. What journals are likely to take their work for publication.

The gates control the entry to paper publication, job appointment and grant funding, among other things. You know, really frickin important stuff.

Which means, in my not at all humble opinion, that we should think pretty hard about our behavior when it touches on this gate keeping.

We need to be very clear on when our jihadist "rules" for how science needs to be done affect right from wrong versus mere personal preference.

I do agree that we want to keep the flagrantly wrong out of the scientific record. Perhaps this is the issue with the triggering post on fMRI but the admission that these practices still continue casts some doubt in my mind. It seems more like a personal preference. Or a jihad.

I do not agree that we need to put in strong controls so that all of science adheres to our personal preferences. Particularly when our personal preferences are for laziness and reflect our unwillingness to synthesize multiple papers or to think hard about the nature of the evidence behind the Abstract's claim. Even more so when our personal preferences really are coming from a desire to winnow a competitive field and make our own lives easier by keeping out the riff raff.

9 responses so far

  • neuromusic says:

    This discussion reminded me of Millikan's Oil Drop experiment and the famed (apocryphal?) attempts at replication... each slowly drifting closer to the true value of the charge of an electron. Feynman uses the example to highlight the pressure of each subsequent scientist to conform to Millikan's original results and basically criticizes them for caving to this pressure.

    But what about Millikan himself? By publishing first, despite apparent error, his name is the one that goes down in the history books. One could even imagine him reviewing papers attempting replication and holding them to a higher standard than his own for instrument precision or control experiments or statistical power or whatever, "pulling up the ladder" as you say.

    It strikes me as a very clever strategy for a science career... if you are the First (especially when it comes to a new model or employing a new technology), you get to both claim precedence AND avoid being held to the same scrutiny of those trying to follow in your path.

  • drugmonkey says:

    ..hmm. I am not sure I see it as closely related. The need to "get the right result or you are clearly not able to publish" is really terrible. Far far more disturbing for science than the attempt to impose the "right" orthodox methods for a given assay or type of experiment.

  • GM says:

    It strikes me as a very clever strategy for a science career... if you are the First (especially when it comes to a new model or employing a new technology), you get to both claim precedence AND avoid being held to the same scrutiny of those trying to follow in your path.

    I work in a field that basically revolves around new methods (more so than actual new scientific discoveries). I've seen what you describe happening many times and I've had that thought myself repeatedly -- if you publish first you get both the credit for doing it first and avoid the "But was it done well?" scrutiny (because you were first and there is no standard to compare you to). And I think that it's not just me, people are well aware of that fact and their publication tactics and practices reflect it.

    There is also the question of hiring practices. My impression is that these days the requirements for getting an interview at a major institution are the following:

    1) having a unique method/technique/system that nobody else does and that will generate papers and attract grants for the foreseeable future
    2) C/N/S papers, usually on that technique/system
    3) Academic lineage/Big shots having your back

    With 1) being the most important.

    I may be wrong but I think there is definitely an effect of people focusing more on method development than on answering questions, precisely because of these factors. Answering important questions is hard and is the kind of thing that in many cases you can only do once you have your own lab and can spend 10-15 years on one such question. You often can't afford pursuing that sort of thing without tenure, and you most definitely cannot afford to pursue it as a postdoc. So if you are in the business of answering questions in existing systems using existing techniques, you run a huge risk of placing yourself at a severe disadvantage come hiring time -- you might get lucky and strike gold, but more likely than not you will have answered some relatively small question and you will not have the unique technique/system that seems to be so important for getting hired at a top place. Therefore the best strategy is to go into method development. And not spend too much time thinking about questions.

    End result -- as usual, science suffers.

  • Laffer says:

    I'm not a 'cream off the top' kind of scientist. I have made a good career taking sharper tools made by someone else and revisiting 'old' results that clarify the previous work. As a result, these papers have to demonstrate both that the new methods are better and that the old results were wrong, leading to 'complete stories.'

    I enjoy your admission that both LPU and complete stories are necessary for the sake of scientific diversity and completeness. If we all pursued the LPU approach, it could be that none of these different, particular observations are consolidated. And if we all pursued complete stories, all of the negative results that didn't fit the model would be shelved and ignored.

  • AcademicLurker says:

    It strikes me as a very clever strategy for a science career... if you are the First (especially when it comes to a new model or employing a new technology), you get to both claim precedence AND avoid being held to the same scrutiny of those trying to follow in your path.

    One theory regarding Jan Hendrick Schon's epic run of fraudulent Science and Nature papers is that he believed that the effects he was reporting were "real" in the sense that someone else would eventually genuinely detect them (as opposed to fabricating the data). When that happened he would be assured of his place of honor as the "pioneer" who got the field started.

  • physioprof says:

    One additional problem with this "just publish the shit and let the field sort it out" philosophy beyond the rush for priority is that it can lead to fields being infected with "everyone knows that X is the case", and thus preventing "X is not the case" from every being published later even though it's the right answer.

  • GM says:

    One theory regarding Jan Hendrick Schon's epic run of fraudulent Science and Nature papers is that he believed that the effects he was reporting were "real" in the sense that someone else would eventually genuinely detect them (as opposed to fabricating the data)

    It's not just him, this happens in a number of cases of research misconduct -- the perpetrator truly believes his claims are correct and so he does not bother actually doing some of the controls or even the experiments and instead fabricates them. At least some of the time the hypothesis indeed turns out to be correct.

  • qaz says:

    It is REALLY important to differentiate fraud from a paper that says "We think that X is true, but the data is really messy" which includes the messy data accurately. It is important to publish those papers that say "We think that X is true, but the data is really messy" because it can encourage people to go see whether X is true. My favorite example of this is the first place cell paper, which is amazing in its lack of convincingness. But it opened the door rather than closing it, and it wasn't for half a decade or more (with many labs doing replications*) that people began to really believe the hippocampal place cell story.

    * Real replications - meaning that people didn't try to redo the experiment and see if they got the same results - instead people did other experiments that tested the theory and over the subsequent 30 years discovered the boundaries of the place cell hypothesis.

    It is also important for it to be OK to be wrong in your theory, as long as you are clear about what data you have and what data you don't have. A lot of the problem is not the fact that the methods are messy, but that people are so enamored with being not first but only, that all conclusions have to be "proven" rather than "suggested".

    I would argue that the problem is that the current sociology of science does not allow people to "be first but be sloppy (because of the limits of technology)" because scientific papers are treated as closing the door on conclusions. (See discussion of LPU and "complete stories" in previous threads.)

  • fjordmaster says:

    Your comments on gate-keeping reminded me of an episode from grad school:

    A big-name PI, who was also the editor of a top journal in the field, gave a talk describing their new work using a particular method. A method that they had previously and publicly declared lacked any real value in the field. The PI did admit that they had come around on the potential value of the method and punctuated the statement with a chuckle.

    I did not find it very funny because I knew multiple people, some early-stage, that had their manuscripts outright rejected from the PI's journal because they had incorporated the previously blacklisted method.

Leave a Reply