Not Getting the Point on "The Third Reviewer"

Jun 25 2010 Published by under Neuroscience, Science Publication, Tribe of Science

Zen, Zen, Zen. Oh, Zen. On The Third Reviewer:

My position: Anonymity doesn't improve things (see here and here).

Yeah, it does. And the proof is in the pudding proof of the pudding lies in the eating. We've been through the evidence before. More comments and more vigorous exchanges on sites which permit anonymous commenting. Sites which do not remain mired in low-traffic land with a limited group of participants trading puns and pictures of their cats.

Has science become that much like the mob?
Are we as a group that thin-skinned, petty and vindictive that we're going to put out a hit someone's grant or whack another scientist's pub because they didn't think we used the right statistical test?
And if the answer is yes, we should start asking ourselves why that bad behaviour is tolerated, and how we can get rid of it.

Anyone spot the error of logic here?


The point, my dear Zen, is not so much whether it is a good or bad thing that people fear career reprisals for what should be our stock in trade. The question is whether they indeed fear such reprisals (with justification or not) and whether that fear keeps them from engaging in a desirable behavior, i.e., open discussion of papers and data.
The Third Reviewer bypasses should to overcome what is. Is it not obvious that if it succeeds it will take us closer to the should goal?

Besides, anyone who believes they can safely remain anonymous on the web is fooling themselves. Own every word you speak.

Yes and no. Sure, you comment enough, tick off enough people and eventually they will be able to figure you out. But post one comment on The Third Reviewer and I don't think there is a huge risk.
Also, remember that in the vast majority of cases we are talking about fear squelching the willingness to comment, not the reality of sanction. It is very unlikely that ever author that perceives a critical comment is going to track down the "culprit". This anonymous comment site doesn't guarantee that you will stay hidden but it sure changes the odds.

I have a blog. If I have something substantive to say, I'll say it here.

Right. Did you actually read my post on the topic? Coverage. Range. Focus. ...and not everybody even wants to blog, holmes. This comment lacks pretty seriously in the imagination department.

I wish the guys behind the site luck, because I think they'll need it. The history of researchers commenting on published papers is... not encouraging.

Heh, heh. Which is the whole bloody point, Zen! Let's try this another way. What is your hypothesis as to why the history of comment on published papers is so dismal?

10 responses so far

  • Off Topic Nitpicking----The saying goes "The proof of the pudding is in the eating". There may be chocolate in the pudding, tapioca even, but proof, no.

  • I have a blog. If I have something substantive to say, I'll say it here.

    Yeah, to the five fucking people reading it.

  • DrugMonkey says:

    Now, now, PP. Play nice.

  • Autistic Lurker says:

    Results 1 - 10 of about 119,823 for the third reviewer.(google blogs)
    PP has a point. I don't even have the time to go through the first 20 pages of search results.

  • rork says:

    Do I fear reprisal: Yes.
    Is the fear justified: Yes.
    Will anonymity reduce my fear: Not much for papers with a limited audience. Might work for papers with large audiences.
    Another option I've always wondered about for papers that are heavy in stats or informatics (what I know), is a little like "Nicolas Bourbaki" was in math. A large group of co-authors would publish reviews of selected papers, pointing out the errors and fudging, lying, hiding, and dubious parts for example, but the group is so large (300 authors, every time) that no one person can really be shot, even though perhaps only 5 people really worked on any one review. Like a quality improvement in math-science, the QUIMS (gotta work on that). The advantage would be that each small group would be assigned the paper they would review, rather than self-selecting, which would reduce but not completely eliminate ax-grinding. Defamation might still be claimed against the consortium though. Need more thought.
    It's hard to tell the difference between a crafty cheater, and a blunderer selecting results (from many) that came from errors or misconceptions "in their favor". Which would you rather be considered?

  • HFM says:

    Since there's very little upside for participating in something like this, even the vaguest possibility of a downside will chase people away. If you leave a critical comment, and you sign it, you've got no winning options: either you're right and the authors now hate you, or you're wrong and you look like an ignoramus. (And if you leave a positive comment, you get what, a cookie?) It's like the pissing contests at seminars, only permanent and searchable.
    And by the way, I've gotten some non-trivial flack for mouthing off about science on the internet under my real name. As an undergrad, I posted a comment about my own work(!), in which I had some opinions about a particular controversy in the field. Journalists found the post, and it got play in the national media. Which...caused quite the flamewar. Oops.
    So yes, scientists are people, and I don't think it's paranoid to want anonymity. I'm not sure that anonymity will be enough, but at least it should help get things rolling.

  • bsci says:

    One other thing of note is that one might not want critiques of other people (even if they're positive & well written) to be the first thing that shows up when someone searches your name. Getting your name on a big website (i.e. Nature, PLOS) means that is going to be at the top of searches.
    I think the ideal is a moderated system that allows anonymity, but has with clear rules on what types of comments are appropriate. Who will write the "clear rules" and who will moderate is a bit trickier.
    (I would have commented on Zen's site, but it requires a logged in account to post)

  • Yep, the theory we're working on is that first we should get people actively commenting, as compared to the current dismal state of affairs on journal websites; and then if the site's descending into lagoons of backbiting we can always try to rein it back. But the first step is to get the discussion going. Zen, like everyone else out there, is encouraged to help establish the tone of comments by leaving a good one!

  • Zen Faulkes says:

    Post necromancy!

    My thinking on anonymity has softened. I still think anonymity is overused, and I favour transparency in general. But I can see more situations where anonymity can be beneficial.

    Hopefully, it's useful to see that academics can change their minds once in a while.

  • […] My position has moderated over time, partly because people have pushed me on this. For example, this post generated this pushback. […]

Leave a Reply