Amgen continues their cherry picking on "reproducibility" agenda

Feb 05 2016 Published by under Conduct of Science, Replication, ReplicationCrisis

A report by Begley and Ellis, published in 2012, was hugely influential in fueling current interest and dismay about the lack of reproducibility in research. In their original report the authors claimed that the scientists of Amgen had been unable to replicate 47 of 53 studies.

Over the past decade, before pursuing a particular line of research, scientists (including C.G.B.) in the haematology and oncology department at the biotechnology firm Amgen in Thousand Oaks, California, tried to confirm published findings related to that work. Fifty-three papers were deemed 'landmark' studies (see 'Reproducibility of research findings'). It was acknowledged from the outset that some of the data might not hold up, because papers were deliberately selected that described something completely new, such as fresh approaches to targeting cancers or alternative clinical uses for existing therapeutics. Nevertheless, scientific findings were confirmed in only 6 (11%) cases. Even knowing the limitations of preclinical research, this was a shocking result.

Despite the limitations identified by the authors themselves, this report has taken on a life of truthy citation as if most of all biomedical science reports cannot be replicated.

I have remarked a time or two that this is ridiculous on the grounds the authors themselves recognize, i.e., a company trying to skim the very latest and greatest results for intellectual property and drug development purposes is not reflective of how science works. Also on the grounds that until we know exactly which studies and what they mean by "failed to replicate" and how hard they worked at it, there is no point in treating this as an actual result.

At first, the authors refused to say which studies or results were meant by this original population of 53.

Now we have the data! They have reported their findings! Nature announces breathlessly that Biotech giant publishes failures to confirm high-profile science.

Awesome. Right?

Well, they published three of them, anyway. Three. Out of fifty-three alleged attempts.

Are you freaking kidding me Nature? And you promote this like we're all cool now? We can trust their original allegation of 47/53 studies unreplicable?

These are the data that have turned ALL OF NIH UPSIDE DOWN WITH NEW POLICY FOR GRANT SUBMISSION!

Christ what a disaster.

I look forward to hearing from experts in the respective fields these three papers inhabit. I want to know how surprising it is to them that these forms of replication failure occurred. I want to know the quality of the replication attempts and the nature of the "failure"- was it actually failure or was it a failure to generalize in the way that would be necessary for a drug company's goals? Etc.

Oh and Amgen? I want to see the remaining 50 attempts, including the positive replications.
__

Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012 Mar 28;483(7391):531-3. doi: 10.1038/483531a.

20 responses so far

  • drugmonkey says:

    Note: it actually is not stated in the Nature news bit whether these are three additional studies or were from the original 53 described. I assumed that without any evidence.

  • boehninglab says:

    Those three papers/findings had already been self corrected in the literature.
    http://blogs.sciencemag.org/pipeline/archives/2016/02/05/three-not-so-reproducible-papers

  • drugmonkey says:

    Interesting. I missed that, obviously.

  • odyssey says:

    Nature continues its march towards becoming the National Enquirer of science...

  • Dave says:

    Those three papers/findings had already been self corrected in the literature.

    Hmmm, where are you getting that from for the GPR21 paper? Amgen hypothesized that the original GPR21 phenotype was caused by disruption of an overlapping gene (Rabgap1), but nobody has published this before. In their 'replication' attempt, however, they show that the expression of Rabgap1 is indeed increased. However, they present no evidence whatsoever that this gene accounted for the original phenotype. What is surprising is that they say this:

    Microarray results identified that Rabgap1 was the only gene that was changed in all the tissues analyzed

    As someone who has done a lot of array and RNA-Seq work in knockout tissue, that sounds like total nonsense. They don't show the full microarray dataset.

    I could go on and on, but the bottom line is that at least one attempt is a long way from a slam-dunk replication failure, and the quality of their attempt certainly doesn't match up to Amgens bravado on this.

  • MoBio says:

    @Dave

    The GPR21 results certainly are new and agree have not been 'self corrected in the literature'. Looking at where the original targeting construct was and how they revised it with the TALEN explains why Rabgap1 would be deleted as well in the original papers (one of which was by Amgen).

    I tend to believe the main finding of the GPR21 paper that the original finding was incorrect in that it implicated GPR21 as being important for the metabolic phenotype observed.

    As they mention it still begs the question as to what GPR21 might do. As someone who studies GPCRs I found this paper quite useful and reasonably informative.

    Remember though Amgen is correcting Amgen here.

  • drugmonkey says:

    lemme get this straight. original paper claims to knock out gene 1, attributes the phenotype to that. turns out, they knocked out at least two genes. and to add the frosting, MoBio claims a close examination of the original targeting construct could have been picked up in the first place as a problem.

    THIS. IS. NOT. A. FAILURE. TO. REPLICATE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  • MoBio says:

    Yes, 'not a failure to replicate'

  • Busy says:

    "a company trying to skim the very latest and greatest results for intellectual property and drug development purposes is not reflective of how science works. "

    I'm not sure what this has to do with replicability. If anything it seems they would be more interested in obtaining a positive result. Care to elaborate?

  • Dave says:

    Right. This is correcting a mistake, if true. I still wont be convinced until they can demonstrate that rabgap1 KO mice display the same phenotype as the original GPR21 papers. That would actually be replication....oddly.

  • Penfold says:

    You noticed the first 'replication' of the three involved giving a drug to rats, whereas the original study was in mice (including a transgenic disease model), right? And the assays were different. Different species is not replication (although it does seem that original effects may be doubtful from other , better replication studies)
    It is true that many results are not as broadly generalizable as we hope or state in our papers. But doesn't mean that 90% of the literature is flag out wrong as their original letter insinuates

  • uchicagoalum says:

    "I'm not sure what this has to do with replicability. If anything it seems they would be more interested in obtaining a positive result. Care to elaborate?"
    -I think he means that, because the results are new and flashy, that one shouldn't be surprised if they don't replicate at the rate for less novel lines of inquiry. If they did a series of replication studies on, say JBC papers, I wouldn't be surprised if they had a much better replicability score.

    OOOH, can replicability score replace JIF, please?

  • drugmonkey says:

    Absolutely Penfold. I have long suspected many of these alleged replication failures claimed by Amgen are of the lack-of-generalization variety. A drug company of course wishes they can take a single hot basic science result and turn that into a human medication. The fact this doesn't work out is not a failure of the original paper to replicate. And ffs the overblown translational claims of your typical Glam authors doesn't change this reality.

  • Point of information:

    The Nature news story states:

    "In 2012, Amgen researchers made headlines when they declared that they had been unable to reproduce the findings in 47 of 53 'landmark' cancer papers. Those papers were never identified — partly because of confidentiality concerns — and there are no plans to release details now either, says Kamb, who was not involved with that publication."

    The 3 data studies Amgen just posted are *not* related to the work that was published in 2012. We still don't know which were the 53 papers that Amgen were talking about in 2012 (and perhaps we may never know).

  • Biotech Boy says:

    Great points made by the author of this blog. It is unfortunate that these 3 data sets were chosen for release (given the methods used for "reproduction"), as they are not good examples of a phenomenon well known to all of us (academics and industrial types alike) - that is, the poor quality of the biomedical literature, making much of the data not reproducible or reliable. I, like everyone else here, would have loved to have seen data generated by identical methods failing to reproduce the results and conclusions of the original respective publications (I am sure there is a wealth of this data in internal R&D servers at large companies such as Amgen).

    I am surprised that Amgen allowed researchers to release replication efforts of any of these publications, due to the potential for insights into their R&D pipeline. Therefore, I suspect that the data released was chosen solely because they were generated in the course of projects that have now been canned. Perhaps this is why we are all so disappointed - we are not seeing the cream of the crop of reproducibility efforts generated by Amgen R&D.

  • drugmonkey says:

    BB- on what basis do you assume they have anything else of higher relevance to the so-called replication crisis? The better assumption is that all their complaining is of the same nature. 1) failure of generalization, 2) not working at it hard enough and 3) trying to skim the cream from results that are least likely to be as good as hoped.

  • qaz says:

    Let's say it again. There is no replication crisis. There is only a misunderstanding of how science works.

    Scientific experiments are by nature occurring at the edge of knowledge. (If they weren't, they'd be demos.) That means there are lots of factors that are not understood. We do our best to control those factors. When someone finds that a result doesn't replicate, saying there's a problem with the scientific literature is the incorrect response. The correct response is to figure out what the key factor is/was that differed unexpectedly between the experiments. Sometimes that key factor will surprise you. (Hint: That new and surprising factor - that's called a "discovery".)

    The problem comes from people taking single experiments and making judgements based on them. Particularly across generalizations (different species? different genes?) or with factors that are poorly controlled (humans with lots of different histories?).

    Real science progresses slowly. This is why it takes 30 years to go from discovery to implementation. It takes many replications (and many failures to replicate!) to find the edges and parameters of a result.

    If AmGen wants to participate in the scientific literature, they should report the differences between their experiments and the ones they didn't replicate, so that we can build on their discoveries. If they want to keep it all proprietary because scientific sharing will cut into their profit margin, then they should just shut the f*ck up.

  • drugmonkey says:

    Very well said, qaz.

  • fjordmaster says:

    Drugmonkey, agree with your points on failure of generalization. I would love for Amgen to release some metadata on those original 53 studies. We could see how close their attempts were to actual replications and they could maintain confidentiality. How many of their "replication" attempts were in a species or cell line that was different from the original publication? How many studies used different pharmacological tools that may be in the same class, but have different "off-target" effects?

    A lot of my work involves validating translation between rats and mice, and even between different strains within one or the other species. You can't take those seemingly small changes for granted. If I personally see a response in C57BL/6J mice and not in CD-1 mice. I would not call that a failure to replicate even within my lab. It would be ridiculous to make that claim across labs.

  • dsks says:

    I replicated 2435 landmark studies in my basement last night, and failed to reproduce the principal findings of any of them.

    True story.

Leave a Reply