Does it matter how the data are collected?

Commenter jmz4 made a fascinating comment on a prior post:


It is not the journals responsibility to mete out retractions as a form of punishment(&). Only someone that buys into papers as career accolades would accept that. The journal is there to disseminate accurate scientific information. If the journal has evidence that, despite the complaint, this information is accurate,(%) then it *absolutely* should take that into account when deciding to keep a paper out there.

(&) Otherwise we would retract papers from leches and embezzlers. We don't.

That prior post was focused on data fraud, but this set of comments suggest something a little broader.

I.e., that fact are facts and it doesn't matter how we have obtained them.

This, of course, brings up the little nagging matter of the treatment of research subjects. As you are mostly aware, Dear Readers, the conduct of biomedical experimentation that involves human or nonhuman animal subjects requires an approval process. Boards of people external to the immediate interests of the laboratory in question must review research protocols in advance and approve the use of human (Institutional Review Board; IRB) or nonhuman animal (Institutional Animal Care and Use Committee; IACUC) subjects.

The vast majority (ok, all) journals of my acquaintance require authors to assert that they have indeed conducted their research under approvals provided by IRB or IACUC as appropriate.

So what happens when and if it is determined that experiments have been conducted outside of IRB or IACUC approval?

The position expressed by jmz4 is that it shouldn't matter. The facts are as they are, the data have been collected so too bad, nothing to be done here. We may tut-tut quietly but the papers should not be retracted.

I say this is outrageous and nonsense. Of course we should apply punitive sanctions, including retracting the paper in question, if anyone is caught trying to publish research that was not collected under proper ethical approvals and procedures.

In making this decision, the evidence for whether the conclusions are likely to be correct or incorrect plays no role. The journal should retract the paper to remove the rewards and motivations for operating outside of the rules. Absolutely. Publishers are an integral part of the integrity of science.

The idea that journals are just there to report the facts as they become known is dangerous and wrong.

__
Additional Reading: The whole board of Sweden's top-ranked university was just sacked because of the Macchiarini scandal

13 responses so far

  • nickwan says:

    I agree with this statement. How data are collected is important simply from a "garbage in, garbage out" perspective. Data collection transparency is important. From an ethical perspective, it's even more important -- sadly, not all research published follow irb/iacuc protocol (let alone, go through an ethical check). How data are collected is definitely something not to overlook.

  • jmz4 says:

    I suppose as long as the data remain searchable, flagged as credible, and available, it really isn't that important that the authors no longer get to keep it on their CV. But no one can cite it, so it still quells the dissemination of valid information, which is unfortunate.

    I will note that if someone was conducting research without IRB approval, a paper being retracted would be the least of their worries. They'd likely be suspended from receiving funding and could be fired outright by the University, which would also incur fines and penalties.

    Since we have alternative bodies and laws to govern these kinds of ethical violations, I don't see that there is a need for journals to do their own policing in those areas. I also don't overly trust them to do it effectively and without bias.

    Plagiarism might be a thornier issue, but I'd still say the appropriate response is to credit the original writer of the section as an author and leave a note explaining the authors are dipshittes.

    Imho, retraction should be only for when the information in the paper has been credibly called into doubt. Anything else is confusing and seems unnecessary.

    Getting published is not a prize, it is an obligation that comes with using public money for research. Taking potentially useful information out of the public arena (which retraction does to some degree), doesn't serve the broader interest of the scientific community. As you noted, it doesn't even hurt the people whose article was retracted that much. They've already gotten jobs and grants from it, and can spin the retraction story however they want. So retraction's utility as a deterrent is limited from the start.

  • qaz says:

    @DM, this is an old and long discussed ethical dilemma. Do we use the Nazi or Tuskeegee experimental results? My understanding is that the current accepted ethical decision is to use the data (taking into account that the data may be flawed because the science has generally been poor), but not allow credit, fame, or rewards to accrue.

    The problem @jmz4 is that getting published IS a prize. It is the coin of our realm. A Nature paper gets you a job. (So does a Cell paper, although I still don't understand why.) A Science paper gets you a grant. What we need is a way to say "This paper's data is correct, but it was obtained unacceptably and therefore the authors may not claim credit or win awards for it."

    Unfortunately, right now (as has been made clear through the wandering harasser situations) the retraction of an important paper would seem to be the bigger worry than local punishment because a famous enough scientist seems to be able to walk away. On the other hand, people thinking that you're not a reliable scientist (because your papers are "known to be bogus") is a very serious problem that can impact your ability to publish in the best journals.

  • Ola says:

    @JMZ24
    "I will note that if someone was conducting research without IRB approval, a paper being retracted would be the least of their worries. "

    Wrong. The retraction is the public face of wrong-doing. All that other crap (being suspended from IRB research, etc.) can be brushed under the rug never to be seen again. Quiet word behind closed doors between the University and faculty member attorneys, nothing to see here.

    A retraction is forever, not just for Christmas. Good luck getting that past the search committee next time you're out looking for a job.

  • odyssey says:

    @jmz4: Science is built on trust. You said on DM's original post you had read the Cell paper. Given that some of the data may have been falsified, how do you know what you should trust in that manuscript? How would you know to trust the work of someone who's willing to lie about IRB or IACUC approval? Sure we have to trust the bulk of what's out there, but in cases where there is proof of some kind of malfeasance? Retraction, imperfect though it might be, is all we have to remove/flag science that cannot be trusted from the record and to point out the bad actors in the community.

  • @jmz24
    Retraction does not remove a paper from circulation. I wish it did.

    In my courses, I often see students citing papers later retracted for fraud from Glam journals, because the retraction comes years later and is not obviously attached to the original. The original remains searchable. These are papers whose results don't replicate, yet they remain polluting the literature.

  • becca says:

    Well, of course it matters how the data are collected.

    That said, retractions for ethical norm violation in the absence of any scientific validity questions makes me uncomfortable. I suspect plenty of people wanted Milgram's career to suffer for the unethical aspects of his authority experiments. That, or the fact we shouldn't rerun them as originally performed, doesn't mean retracting his article is appropriate.

    Given modern methods of accessing information electronically, we can (and should) do a better job linking scientific works that have known issues with later criticism and commentary. We have opportunities to do something other than stamp a stigmatizing but vague red "R" on a paper and throw it in the trash.

  • Draino says:

    I was once caught doing research that was not conducted under the appropriate ethical-standards and it almost ended my postdoc.

    There I was, two and a half years into the project, with tons of great data and dreams of Glam. But then one of the vivarium workers noticed a notation on some of my mouse cages. I had written "Tam" and some dates. Of course it meant Tamoxifen, something I routinely used and something I had no reason to hide. The only problem was, nobody had ever thought to add Tamoxifen to the mouse protocol. I was naive and the PI screwed up and forgot. So the zealous vivarium worker reported to her boss, who reported it to the IACUC, who reported it all the way to the NIH. After a couple weeks of knots in our stomachs (mine and my PI's), the NIH came back and said we could still use the data but we would have to tell the journal what we had done. Luckily the journal didn't much care about the violation, twas an honest mistake really, and we were able to publish. In the methods section we did not say all the work was done with IACUC approval. We only said the protocols for making the mice were approved, which is true.

    I learned my lesson about approvals and data collection. Still have some PTSD from the experience. I always want to be best buds with the vet staff so just maybe they'll talk to me if they catch me in error, instead of running it all the way to the NIH.

  • serialmentor says:

    jmz4's comment seems to imply that it is the job of scientific journals to provide only correct, verified information, as in an encyclopedia. This is definitely not the case. We don't retract papers that later turn out to have favored an incorrect hypothesis, or where the authors made some poor experimental choices that led to incorrect conclusions.

    The accepted standard of scientific publishing is that we report accurately and honestly what was done and what was observed. As long as this standard is met, a paper should never be retracted, even if it later turns out that it was completely wrong.

    By contrast, if somebody falsified data or made them up, then this standard isn't met, and hence the paper should be retracted regardless of the ultimate truth of its message. The retraction is not to punish the author, it's to flag/remove a paper that has demonstrably not met the accepted standard of scientific reporting.

  • drugmonkey says:

    I suspect plenty of people wanted Milgram's career to suffer for the unethical aspects of his authority experiments. That, or the fact we shouldn't rerun them as originally performed, doesn't mean retracting his article is appropriate.

    If you are talking about changing standards over time (as in the HM case we recently discussed), or about monday-morning-quarterbacking the local IRB/IACUC, I'd say this is a bit different from violating the standard in place at the time of publication.

  • drugmonkey says:

    I often see students citing papers later retracted for fraud from Glam journals, because the retraction comes years later and is not obviously attached to the original

    I agree it is an absolutely fixable problem to attach "retracted" to pdfs and to the medline record, etc. This can and should be done. comprehensively.

    These are papers whose results don't replicate, yet they remain polluting the literature.

    Danger Will Robinson. Again, it is absolutely immaterial to any discussion of fraud/retraction whether the results can be replicated or not.

    ...NIH came back and said we could still use the data...Luckily the journal didn't much care about the violation, twas an honest mistake really, and we were able to publish.

    To me, this is an appropriate oversight process working as it should. There was a violation and an investigation external to the lab/scientists and even University went up and down the chain. My opinion, your opinion, my Dear Readers' opinions are all well and good but the important thing is the specific oversight process. Sounds like it worked.

    The accepted standard of scientific publishing is that we report accurately and honestly what was done and what was observed. As long as this standard is met, a paper should never be retracted, even if it later turns out that it was completely wrong.

    Spot. On. Target.

  • These are papers whose results don't replicate, yet they remain polluting the literature.

    Danger Will Robinson. Again, it is absolutely immaterial to any discussion of fraud/retraction whether the results can be replicated or not.

    True enough. The papers were retracted for falsifying data, as they should have been (even if they were later shown to be correct).

    Falsified papers that are also factually incorrect are the gift that keeps on giving pain, since they still show up in searches, causing newcomers to the field (especially students, who have less experience with literature searching) to sometimes be led down the rabbit hole for a while until they realize that they are wasting time on nonsense. Such papers are rarely cited after the retraction, so there is less of a literature paper trail pointing to problems, unlike when scientific understanding evolves over time.

  • Jonathan Badger says:

    Of course we should apply punitive sanctions, including retracting the paper in question

    I think the idea that a retraction is a "punishment" for the authors is absolutely the wrong way to think about it. The point of retracting a paper is to tell scientific community know that the results of a paper are not supported, period. This can be due to fraud or just honest error. Yes, a retracted paper on one's CV looks bad, but that isn't the point of the retraction.

Leave a Reply