Throwing punches about PubPeer

Sep 30 2015 Published by under Peer Review, Science Publication

Wow.

PS Brookes has posted a spirited critique of an Op-Ed offered by Michael R. Blatt, EIC of Plant Physiology.

 

[Blatt] then adds this beauty…

“So, whatever the shortfalls of the peer-review process, I do not accept the argument that it is failing, that it is a threat to progress, or that, as scientists, we need to retake control of our profession. Indeed, if there is a threat to the scientific process, I would argue that, unchecked, the most serious is the brand of vigilante science currently facilitated by PubPeer.”

So let’s get this straight – the problems facing science today are not: (i) a lack of funding,  (ii) rampant fakery, (iii) politicians seeking to defund things they don’t like, (iv) inadequate teaching of the scientific method in schools, (v) proliferation of the blood-sucking profiteering publishing industry, (vi) an obsession with impact factor and other outdated metrics, (vii) a broken training to job pipeline in academia, (viii) insert your favorite #scipocalypse cause here.

Go read the Editorial and then the takedown.

I had a few issues with Blatt's comments that did not get addressed by Brookes.

Blatt writes:

I concur with Hilda Bastian,3 who notes, on the one hand, the lack of reliable evidence to support the benefits of reviewer anonymity and, on the other, the importance of assessing whether commenters are “outside their areas of expertise … [or] have conflicts of interest.” Anonymity can conceal much mischief and do great damage. Even PubPeer acknowledges that “anonymity does allow low quality and bad faith comments to be made with impunity”

Ok, so first he mentions a lack of evidence to prove the benefits of anonymous peer-review. Which is in the territory of trying to prove a negative. Right? The main benefit of making peer-review anonymous is to get comments that are honest and un-modified by any threat of retaliation against the one offering honest, but critical, commentary that may irritate the authors. It is almost impossible to know, particularly in an opt-in reviewer selection system, the extent to which an open peer review system has lost honesty of review. It is likely to be hard to detect.

On the other hand, Blatt is making an assertion of a positive ("mischief" and "great damage") without any shred of evidence in support of the existence or scope of any such problem. Nor is he comparing the rate of mischief that is worked because of anonymity versus the rate of mischief that is worked through a sense of impunity that is associated with having a position of power in the field.

Hey, since we're making assertions and all, Editor Blatt, here's mine. "Positions of relative power in science can conceal much more mischief, and facilitate damage to science and careers alike on a grander scale, than anonymous comments ever have or will."

Wheee! Aren't data-free assertions fun, Dr. Blatt?

The second set of comments are rather bizarre and I invite your input, Dear Reader. Blatt writes:


However, the argument for anonymity in postpublication discussion fallaciously equates such discussion with prepublication peer review. The essence of prepublication review is a mutual agreement, a social contract between author and editor, who are known to one another. The author submitting a manuscript for review agrees to accept the judgment of the editor, and the editor agrees to judge the worthiness of the manuscript for publication. To make this assessment, the editor may obtain independent reviews of the author’s work before passing judgment, but (and here is the fundamental flaw in the argument for “post publication peer review” as championed by PubPeer) the reviewers and their anonymity are secondary to the contract between author and editor. In short, anonymity makes sense when reviews are offered in confidence to be assessed and moderated by an editor, someone whose identity is known and who takes responsibility for the decision informed by the reviews. Obviously, this same situation does not apply postpublication, not when the commenters enter into a discussion anonymously and the moderators are also unknown.

I agree that technically, the peer review of manuscripts is merely advisory to the Editor. So, from one viewpoint, he is not wrong here. But I assert that the process of peer review has become something different in practice. At least for many scientists and many editorial teams.

I feel as though I am contracting not merely with the AE and EIC of the journal but with reviewers to whom they send the manuscript for review. We are in a social agreement that they will stand in as proxy for my desired audience of eventual readers and provide honest commentary on the strengths and weaknesses of our work. It is not blind trust. There is variance in the system and I recognize that I will not always agree with reviewers.The reviewers may not agree with each other. And the AE or EIC may not agree with any or all.

It is a semi-chaotic agreement but an agreement nevertheless. I feel as though authors have an expectation of their peers to participate in the process and an expectation of Editors to perform their role.

I see no reason why it makes sense to Editor Blatt that only "someone whose identity is known" can be seen to "take(s) responsibility for the decision informed by the reviews". Or why it is needed that "responsibility" be taken.

As Brookes observed in response to Blatt's comments to the effect that nattering about gels and blots was unimportant:

The way that “real” scientists respond when their data is questioned, is to answer the damn question! Show the data. Produce the originals. In case you hadn’t noticed, the front page of PubPeer cycles once every 3-4 days – if there’s an innocent explanation, you WILL be vindicated and your career will not end if you engage with the commenters.

It doesn't matter that any one individual take responsibility for a question or criticism! Other scientists will have those thoughts as well. And if they are venal mischief, how can they do great damage if they are wrong? If anything this would cause less mischief because a broader public could evaluate the comment. In contrast the power of anonymous peer reviewers to do great mischief by misleading an Editor is much, much greater.

The arguments being made by Blatt's Editorial just don't make consistent sense to me.

28 responses so far

  • DJMH says:

    Yeah these nutjobs like Blatt seem to think that if PubPeer doesn't post anonymous comments blasting papers, then those thoughts/comments simply don't exist. As though people in private only ever say good things about papers, and the internet facilitates some sort of novel hating function.

    Whereas, I see it as the exact opposite: if people have read my paper and think there's a problem, I would FAR rather have a forum to address that possible problem, than for it to be discussed behind my back without an opportunity for my input.

  • physioprof says:

    Is there any interesting discussion on Pubpeer, or is it all just a bunch of loons masturbating over spliced western blots?

  • drugmonkey says:

    You could have saved Blatt a lot of time, PP.

  • SidVic says:

    I'm starting to like this PBrookes. He doesn't look like much but he is a real rabble-rouser!

  • Dave says:

    There is a lot of chat on Pubpeer about splicing/image problems etc, but there are many threads that I have read that are purely scientific discussions, and can be quite interesting to read.

  • Dave says:

    ...no journal club I ever organized or contributed to was so obsessed with the minutiae of data presentation

    That's an unbelievable statement to me. Bit concerning coming from a journal editor, honestly.

  • zb says:

    I think that anonymity plays an important role in addressing the balance of power in the essentially closed system of academic science. But, I think that in your efforts to come to the defense of anonymity for this goal, you underplay the role that anonymity can play in aggression and harassment (like the harassment of women law students at AutoAdmit).

    There are ways to mitigate that kind of harassment (I'm guessing you are editing your comments section to prevent it) and maybe pubpeer is, too, to some extent. As long as the commenters are sticking to the published paper they seem to avoid this issue of anonymity (yes, they might be commenting on data they don't understand, but presumably the paper can stand in its own defense). But there are ways that anonymity can be used to the wrong ends, and I don't know enough about pubpeer to address whether that's the case.

    (I do think I've never seen your comments section overstep that line in my mind).

  • qaz says:

    You may *feel* like you have a contract with reviewers, but you don't. You have a contract with the editor and the editor has a contract with the reviewers. If the editors are not willing to contradict a reviewer who is demanding excess experiments (which I hear you complain muchly about) or who is citing statistics incorrectly (which has happened to me), then that is a poor editor and you should find another.

    As an author,I have seen editors who have told me that I need to answer reviewers but that I don't need to accomodate them. As a reviewer, I have had editors ignore my rejection of a paper. These are both fine, because the reviewers do not make the decision to publish or not, the editors do.

    There are good editors and there are bad editors, just as there are good reviewers and bad reviewers. My understanding is that good editors know who the good reviewers are and use them when possible.

  • dr24hours says:

    "And if they are venal mischief, how can they do great damage if they are wrong?"

    Easily. Allegations are always more memorable than rebuttals. Doesn't take more than a single person on a study section to say, "Yeah, but I don't trust their integrity. I heard there was some dust up on PubPeer..." not having paid attention closely to the whole arc.

  • Jonathan Badger says:

    I find it interesting that the argument over PubPeer is now whether it is too powerful in its ability to do "vigilante science" -- I seem to remember not too long ago when the consensus was that it and similar things were more or less useless because "nobody" was going to bother to do post-publication review.

  • drugmonkey says:

    zb-

    I am fortunate to have a commentariat that rarely ever requires moderation of comments. There have been some that I've edited but probably fewer than a dozen. I appreciate this immensely.

  • MF says:

    I actually think that both Blatt and PBrooks make a number of valid points. What I am confused by is the assertion that Blatt and his journal are a part of the evil publishing industry that feels threatened by the online peer review. It is hard to see how an academic editor (with a full time academic job, it seems) of a society-published journal is really part of the "industry".

    Also, when Blatt writes dismissively about the minutiae of data presentation, I think he means that in most journal clubs people will be concerned not just with the appearance of spliced gel lanes etc but with the absence of adequate controls as well as the problems with data interpretation. I run a journal club, and my experience has been that we find just as many issues with whether a particular experiment was an adequate test of the authors hypothesis or whether the conclusions are too far-reaching, as with the data presentation.

  • dsks says:

    "Is there any interesting discussion on Pubpeer, or is it all just a bunch of loons masturbating over spliced western blots?"

    Huh. I've never even been to PubPeer, and after searching my field that's exactly what all the comments were about.

    I think it's time to just accept the fact that people who use westerns are naturally inclined towards shadiness and shenaniganry, and shouldn't be allowed to apply for federal funding.

  • jmz4 says:

    It's mostly people doing post-hoc image critiquing, and some of them seem unhealthily obsessed with image compression artifacts, spending hours on analysis of these mostly likely harmless aberrations.
    One does wonder what axe these people have to grind that motivates them to do all this uncompensated, tedious work, which only rarely finds cases of actual fraud. For instance, a thread just flared up on one of Steve Elledge's old (2005) papers, possibly due to his being a recent recipient of the Lasker award. I think it's best not to attribute entirely altruistic motivation to the people on pubpeer.

    That being said, Pubpeer hasn't penetrated deep enough into the scientific community's consciousness to be a threat to anyone. Many PI's and postdocs I've spoken to are unaware of its existence. Those that are expressed sentiments similar to physioprof's. In fairness, here's a good example on something other than a Western blot, a reanalysis of bioinformatics data that I hope becomes more prevalent: https://pubpeer.com/publications/51FF2043DA908ABD9D8AB9CF5B7EF5#fb37415

    I think the best compromise for them would be to retain the anonymity, but at least put some cursory safeguards in to ensure that all the commenters are from unique individuals (i.e., no spamming an article you don't like from multiple angles). Currently you can leave any number of unregistered comments posing as different people. Unique handles for all commenters would also help keep track of the discussions, which can get confusing.

  • lurker says:

    Blatt's implied "social contract" between author and editor is a steaming pile of gut microbiome. What about desk rejects? So-called professional editors who are basically paper and email pushers for the E.I.C.? Zero accountability on the editor even though their name is attached, since the editor can wipe his/hers hands clean by saying Reviewer#3 said so and so. Editors can make or break careers, and a reason why editors go to meetings is not to "brush up" on the field, but to be wined and dined and see who are the BSDs whose brand recognition helps insure a paper will be well cited and garner fanfare. I'll get a desk reject while BSD over there gets editor to revisit their manuscript after an initial review reject by 3 referees.

    I heard anecdata that some BSDs will court editors and EICs to find out what hot topics the Glam journal wants to showcase in 6 months, and the BSD lab will then direct their research so their paper matches what the Glam journal editor wants. The editor also has tremendous sway in the pre-pub review process by who s/he selects for reviewers, because the editorial management tools track how many Accepts and Rejects and Declines each reviewer has made. Not a full guarantee, but an editor scanning these metrics in the Reviewer Database can eyeball who is going to be a Reviewer#3 and who is going to be a softy.

    Surprised these issues have not been brought up by @mbeisen yet, but I don't think it's just my anecdata. Anyone else confirm what I've heard?

  • Philapodia says:

    @dsks

    "I think it's time to just accept the fact that people who use westerns are naturally inclined towards shadiness and shenaniganry, and shouldn't be allowed to apply for federal funding."

    What about:

    "I think it's time to just accept the fact that people who use single-cell patch clamps are naturally inclined towards shadiness and shenaniganry, and shouldn't be allowed to apply for federal funding."

    or

    "I think it's time to just accept the fact that people who use isothermal titration calorimetry are naturally inclined towards shadiness and shenaniganry, and shouldn't be allowed to apply for federal funding."

    or

    "I think it's time to just accept the fact that people who use screwdrivers are naturally inclined towards shadiness and shenaniganry, and shouldn't be allowed to apply for federal funding."

  • Dave says:

    @Philapodia - don't take the bait!!!

    It's worth remembering that many misconduct cases start because of problems with images, so I think we have to be careful criticizing those on Pubpeer who highlight genuine problems with them/image manipulation.

    Sure, there are plenty of nutters on there with axes to grind, but I do think Pubpeer has many excellent post-publication discussions of papers that are just science driven now. They also moderate the comments more aggressively these days, so hopefully that will weed out more of the crazies. Linking the comments to PMC (where anonymity is impossible) and journal websites (i.e. Nature) is also an interesting move that should focus discussions mostly on the science.

  • drugmonkey says:

    lurker- sure. Confirm. It's not really even a secret. The professional Glam editors cop to this. We had a famous comment years ago from a Glam editor type bragging about how they were directing the science and should be recognized as such.

  • AcademicLurker says:

    @Philapodia: Your suggestions are overly complicated. I recommend replacing them with:

    "I think it's time to just accept the fact that people who work in subfields other than AcademicLurker's are naturally inclined towards shadiness and shenaniganry, and shouldn't be allowed to apply for federal funding."

    That's so obviously fair and reasonable that I don't see how anyone could disagree.

  • lurker says:

    Thanks, DM! I think the shenanigans of professional Glam editors Are Not well known. At least not from my interactions with other profs, both noob and mid-career. Is this a dirty secret that "One does not discuss such things", like Fight Club? Do you recall the exact famous comment of the Glam editor's braggadocio? Maybe a Blast-from-Past blog post? Let's bow down to our Glad editor overlords!

  • dsks says:

    "I think it's time to just accept the fact that people who use single-cell patch clamps are naturally inclined towards shadiness and shenaniganry, and shouldn't be allowed to apply for federal funding."

    C'mon. Everybody knows patch clampers walk the incorruptible path of The Pure, The Righteous, and The Good-Looking.

  • Philapodia says:

    It is true that having your head in a Faraday cage adds significantly to your charisma. It's like beer-googles for science nerds.

  • Philapodia says:

    @Academic Lurker

    I double-dog dare you to write that in your next R01 abstract.

  • ROStressed says:

    As someone who has published in Plant Physiology over the years, I find the Blatt Editorial a bit baffling. PubPeer, which I had not heard of before today is as others have noted is clearly not a top threat to Science today. In fact there don't appear to be many Articles from Plant Physiology in PubPeer (only 6 my my scan and none with more than 4 comments). I find it disturbing that the Editor feels like this is the most important subject to highlight. In fact if anything I think that Blatt has increased the visibility of PubPeer to his readership.

  • Jaime A. Teixeira da Silva says:

    I am a plant scientist and I have seen the importance of anonymity in correcting the literature. Therefore, I decided to comment at PubMed Commons, in person, critiquing the Blatt editorial:
    http://www.ncbi.nlm.nih.gov/pubmed/26417050

    I also contacted Prof. Blatt and the entire editor board, and not anonymously, requesting him to reflect on his perspectives, and his fairly one-sided opinion about PPPR and PubPeer. In my personal opinion, I actually think that Prof. Blatt does not understand the belly of this beast, and speaks superficially. If one observes the make-up of the Plant Physiology editor board, one will see editors from research institutes that have come under scrutiny directly or indirectly at PubPeer, so I believe that this editorial has a much deeper purpose to vilify the anonymous voice.

    I also believe that editors like Blatt get irritated with the anonymous voice, because they'd like to tag them, hold the responsible for their opinions, but they fail to see the other side: the possible retributions against a whistle-blower or anonymous commentator. Most likely because Prof. Blatt lives in a cocoon of excellent academic protection, far from the ground-war of fraud and corruption that some of us observe in plant science literature literally daily.

    So, n my comment made on PubMed Commons, I have stuck my neck out. I have tried to be balanced. I have praised Prof. Blatt for contributing towards the overall discussion, but then I have been critical of him for two reason: a) he has not given a balanced perspective; b) he has failed to cite some of my literature, which is key and central to PPPR in plant science. The latter aspect makes me feel that he is editorially biased. Strong critique of Blatt's editorial by others seems to suggest this, too.

    Finally, I am surprised to see that my PubMed Commons comment has not been mirrored on PubPeer. Is this a time lag? I was under the impression that PubMed Commons comments go straight through to PubPeer. I will give it two more days before I get worried. I should also say that an anonymous comment I left yesterday (Oct. 1, 2015) with reference to the PubMed Commons/PubMed page was not published. So, this pseudo-arbitrage at PubPeer had better clean up its act of approving or manipulating comments, because it's not totally acceptable, or fair, how they filter through only what they want, e.g., excessive slant towards the cancer literature.

    Plant science is, despite these criticisms, made better by the Blatt editorial. Because he is a respected figure. And because many on his editor board are in powerful positions and institutes, even in GM corporations. So, he his bringing the naked truth out to the public. And this is good. Let us not be purely critical of Prof. Blatt (who I do not know personally), but let us also guide him and his editors to see the positive, and important, nature of anonymity. In fact, one of my anonymous comments addressed to Prof. Blatt that was not published yesterday by PubPeer (was it really so vile or stupid not to mention publication?) was to invite Prof. Blatt to offer critical commentary about a few papers he has seen with problems in the plant science literature. Unless, of course, he thinks that the literature is perfect and rosy, that is...

    Of course, the best person to address these criticisms of the editorial is Blatt himself. And to that end, he has given me an assurance, by e-mail, that he will respond to these queries (although he did not specify whether he would send them to me personally by e-mail, or to PubMed Commons, or to PubPeer), but he has provided assurances of a response, nonetheless, so may I suggest that a little more patience be exercised.

    I closing, I think that the title should be vigilant science, not vigilante science. Perhaps there was an innocent spelling error in the first word?

  • zb says:

    " I think the shenanigans of professional Glam editors Are Not well known. "

    At the risk of being mean, my guess is that people who think that are so far out of the glamor loop that they aren't in the same league or even the league that aspires to be in the same league. Mind you, I don't think this says anything about the science being done.

    Yes, glamor mags wine and dine (the relationship can go in both directions). They need the superstars (and potential super stars) to review papers and send their papers to them (including to Nature Neuro instead of Neuron). And they want to sit at the table with you when the hot science in your field is being discussed.

    Topics become hot (for example, when a someone wins the nobel prize and has a secret interest in a field peripherally related to their own -- say, a molecular biologist who has always been excited about hormone impact on behavior).

    It's naive to imagine that the system is in any way "fair" in the sense that only the really important stuff goes to the really big journals. I'm not one who thinks that the stuff in the glamor mags is worse, but I do think it's important as a reader/citer not to not foolishly imagine that the best stuff will end up in Science, so no need to read elsewhere.

  • […] “I suggest that many authors do not respond simply because they fail to see any real value in engaging in such exchange, not in a venue of this kind.” Michael Blatt, the editor-in-chief of Plant Physiology, doesn’t like PubPeer very much, particularly its anonymity. But Paul Brookes doesn’t think much of Blatt’s argument, and neither does DrugMonkey. […]

Leave a Reply