The Third Reviewer is....you!

I have occasionally mentioned that I really like the way that Nature Publishing Group (NPG) have promoted the online discussion of scientific research articles. After all, the publication of an article is merely the starting point and the authors' interpretations of their data are only part of a larger set. Science proceeds best when we collaborate with our data, our ideas, our interpretations and our conclusions. Internet technologies can assist with this process. Indeed, these technologies already are assisting and have been doing so for some time. How many times in the last month have you used email to discuss a figure or a paper with a colleague? A ubiquitous phenomenon, is it not? Yeah, well when I started graduate school there was no email*.
I have also, I confess, waxed slightly critical of the execution of online paper discussion. Although I mostly bash NPG because they leave so much tasty chum lying in the water, I am generally critical; PLoS hasn't really managed to do much better than the NPG titles when it comes to consistent online discussion.
Science blogs are slightly better at generating robust discussion of an article which in some cases feels a little more like journal club. This latter is a touchstone target for this behavior, IMNSHO. Science blogs suffer, however, from a lack of focus and a lack of comprehensive coverage. Researchblogging.org is a focal portal to select the journal article discussions out from the cacophony of a typical blog but again, it tends to suffer from coverage issues. The audience is presumed to be a general audience by most science bloggers and therefore they tend to select topics of general interest.
This brings me to a new internet creation: The Third Reviewer

ThirdReviewGrab.png

The first thing you will notice is the list of journals which publish scientific articles in the neurosciences in the tabs at the top. The site grabs a Table of Contents feed and lists each article as a commentable link/entry. The comprehensive coverage problem is solved.
The site allows anonymous commenting. This is huge. It solves what I think is the major problem with the approach of publishing houses to this topic. Like it or not, people are less likely to openly comment on papers in a way that could come back to nail them. Yes, even if they are totally and completely polite, their criticism is on the up and up and 80% of the field agrees with it.
The snooty nosed types allege that anonymous commenting will make such an effort descend into meaningless drivel, ad hominem attacks and nastiness. Those of us who actually discuss papers in online venues that permit anonymous commenting allege that such risks are vastly overblown and that a light hand of moderation, plus social tone-setting, takes care of any problems that might arise.
The Third Reviewer will test these competing hypotheses. And you know I'm excited about that!
__
*yes, it had been developed but it was not in widespread academic use at that point.
N.b. Tragically, the owners of the movie Downfall have gone after many of the YouTube mashups, including the one from which "The Third Reviewer" derives. Has anyone seen it pop up on another host?

16 responses so far

  • Yeah, this is gonna be fucking awesome! Let's nail those glamourfuckers to the fucking wall!

  • neurolover says:

    Looks nice. And, of the 6 comments on the "top" commented paper, 5 are good and substantive, only one naive, and all polite. 2 are from the same non-anonymous poster, but the others are anonymous.
    You've mentioned some problems with execution (and non-anonymous commenting is a big one), but I think another big problem getting post-publication "review" going has been the perceived lack of benefit to the people offering substantive comments. A key is that you need to have knowledgeable people putting their views up. That can draw in those hoping to learn, and then produce a community. But without that seeding, you end up with the blind leading the blind, and people don't join because they don't think they'll benefit.
    In a real life journal club, the professor and senior people provide the expertise, and the compulsion to come plays a role. Without the compulsion, how do you attract the people who are providing more expertise than they are getting, at least in the beginning? Eventually, you create a thriving community, but until then?

  • Dr. O says:

    Looks great; I really hope it catches on in other fields.
    Faculty1000 has done a decent job of post-reviewing articles, but it would be more complete to have a forum for non-"faculty" to get into the discussion as well. And I just don't believe that anonymous commenting, if moderated, is doomed to descend into the abyss.

  • whimple says:

    Very slick. I wonder who pays for it?

  • DrugMonkey says:

    I wonder who pays for it?
    how so?
    You mean the bandwidth and maintenance? it looks like someone associated with the site is commenting over at Neurotopia so perhaps this person will eventually wander over here and chime it.
    My understanding is that this is a privateer effort of some individuals who have been thinking along these lines and wanted to try to make it happen. Labor of love kinda thing.

  • Dan says:

    The snooty nosed types allege that anonymous commenting will make such an effort descend into meaningless drivel, ad hominem attacks and nastiness.
    If the goal is to recreate lab meetings, then a good bit of meaningless drivel, ad hominem attacks and nastiness is going to be required.

  • Thanks for the linklove, DM!
    Whimple, currently I am paying for the hosting myself; it's only ~$100/year. I may look into small grants to cover that amount, but it's hardly a backbreaking cost, even for an academic...
    DrO, yes, F1000 is nice but it's all top-down, and of course the faculty can only say fairly positive things. I've heard colleagues complain that they think a certain paper is misleading but they're intimidated by fear of reprisal from commenting at the journal site. Obviously those are important conversations too, not just the all-good-all-the-time of F1000. FYI, ThirdReviewer is contemplating expansion into other fields if there's sufficient interest; please contact me (neuroreview, gmail, com) if you'd like to talk about it further.

  • whimple says:

    Whimple, currently I am paying for the hosting myself; it's only ~$100/year. I may look into small grants to cover that amount, but it's hardly a backbreaking cost, even for an academic...
    No, who pays for the website design and coding? It's a slick presentation, looks professional grade.

  • @Whimple - it's an out of the box wordpress theme. Too bad I don't know anything about neuro 🙁

  • neurlover says:

    Third reviewer -- Is it possible to add Cerebral Cortex? That's a decent neuro focussed journal.
    the simple design is good, and it really does look like it's working. Now, everyone who believes it's a good thing go make some insightful comments.

  • What Christina said. I did it all myself, using a WordPress theme and some minor coding tweaks. And I'm no coding genius.
    Neurolover, I'll add it to the request list, thanks! It's probably about 3rd in line....

  • jojo says:

    I have a question for The Third Reviewer and maybe anyone else that's done this sort of thing: Are you currently having to upload the relevant article titles manually, or is there a way to pull down journal titles and links automatically? The main issue I see is that if its' manual a lot of papers will be missed and/or the amount of work involved could balloon quickly.

  • Julian says:

    Third Reviewer, how much work is it keeping the site up to date?
    I'd be very interested in related fields (cognitive neuroscience and such).

  • Hi jojo--I'm pulling them in by RSS feeds from each journal. There's a little tweaking here and there (e.g. for journals that publish papers in other fields, I have to either go manual or else import all the papers and delete the irrelevant ones) but it's pretty fast.
    Julian, solely for the journal upkeep it's no more than 1-2 hrs/wk, though of course right now I'm spending more time fiddling with appearances. Send me an email (neuroreview, gmail, com) if you'd like to talk about adding cog neuro either to this site or a subdomain or something.

Leave a Reply