Archive for the 'Peer Review' category

Grant in Review LOLzies

Oct 06 2015 Published by under Peer Review

Probably one of the most hilarious comments I've ever received in review of one of my grants boiled down to this.

"Your colleagues have boatloads of grant money to work on Topic X. Why have you not produced more publications on Topic X with their resources? .....anyway, so therefore your new application on Topic Y sucks. "

[My recollection is that my productivity on Topic Y was mentioned by other reviewers as a strength if anything. If not on that particular proposal, than on other ones around the same time.]

20 responses so far

Throwing punches about PubPeer

Sep 30 2015 Published by under Peer Review, Science Publication


PS Brookes has posted a spirited critique of an Op-Ed offered by Michael R. Blatt, EIC of Plant Physiology.


[Blatt] then adds this beauty…

“So, whatever the shortfalls of the peer-review process, I do not accept the argument that it is failing, that it is a threat to progress, or that, as scientists, we need to retake control of our profession. Indeed, if there is a threat to the scientific process, I would argue that, unchecked, the most serious is the brand of vigilante science currently facilitated by PubPeer.”

So let’s get this straight – the problems facing science today are not: (i) a lack of funding,  (ii) rampant fakery, (iii) politicians seeking to defund things they don’t like, (iv) inadequate teaching of the scientific method in schools, (v) proliferation of the blood-sucking profiteering publishing industry, (vi) an obsession with impact factor and other outdated metrics, (vii) a broken training to job pipeline in academia, (viii) insert your favorite #scipocalypse cause here.

Go read the Editorial and then the takedown.

Continue Reading »

28 responses so far

Completely uncontroversial graph preferences

I am sure that nobody has any opinions whatsoever on using the placement of significance symbols to...err....emphasize..... the magnitude of the effect.

9 responses so far

Applicant fantasizes about visiting the study section meeting during discussion of his grant application

Jul 28 2015 Published by under Peer Review

h/t: Namaste_ish

5 responses so far

Suggesting Reviewers

Who do you select when listing potential reviewers for your manuscripts? 

I go for suggestions that I think will be favorably inclined toward acceptance. This may be primarily because they work on similar stuff (otherwise they aren't going to be engaged at all) but also because I think* they are favorable towards my laboratory. 

Of course. 

(I have also taken to making sure I suggest at least 50% women but that is a different matter.)

I wouldn't suggest anyone that violates  the clearest statement of automatic COI that pertains to me, i.e. the NIH grant review 3-year window of collaboration.  

Where do you get your standards?


*I could always be wrong of course

30 responses so far

On Responding to Prior Critique

The life of the academic scientist includes responding to criticism of their ideas, experimental techniques and results, interpretations and theoretical orientations*.

This comes up pointedly and formally in the submission of manuscripts for potential publication and in the submission of grant applications for potential funding.

There is an original submission, a return of detailed critical comments and an opportunity to respond to those critiques with revisions to the manuscript / grant application and/or argumentative rebuttal.

As I have said repeatedly in this forum, one of my most formative scientific mentors told me that you should take each and every comment seriously. Consider what is being said, why it is being said and try to respond accordingly. This mentor told me that I would usually find that by considering even the most idiotic seeming comments seriously, the manuscript (or grant application) is improved.

I have found this to be a universal truth of my professional work.

My understanding of what I was told by my mentor, versus what I have filled in additionally in my similar comments to my own trainees is now very fuzzy. I cannot remember exactly how extensively this mentor stamped down what is now my current understanding. For example, it is helpful to me to consider that Reviewer #3 represents about 33% of peers instead of thinking of this person as the rare outlier. I think that one may be my own formulation. Regardless of the relative contributions of my mentor versus my lived experience, it is all REALLY valuable advice that I have internalized.

The paper and grant review process is not there, by any means, to prove to you beyond a shadow of a doubt** that the reviewer's position is correct and you are wrong. A reviewer that provides citations for a criticism is not by any means the majority of my experience...although you will see this occasionally. Even there, you could always engage cited statements from an antagonistic default setting. This is unwise.

The upshot of this critique-not-proof system means that as a professional, you have to be able to argue against yourself in proxy for the reviewer. This is why I say you need to consider each comment thoughtfully and try to imagine where it is coming from and what the person is really saying to you. Assume that they are acting in good faith instead of reflexively jumping behind paranoid suspicions that they are just out to get you for nefarious purposes.

This helps you to critically evaluate your own product.

Ultimately, you are the one that knows your product best, so you are the one in position to most thoroughly locate the flaws. In a lot of ways, nobody else can do that for you.

Professionalism demands that you do so.

*Not an exhaustive list.

**colloquially, they are leading you to water, not forcing you to drink.

3 responses so far

Learning and training

Jun 05 2015 Published by under NIH, NIH Careerism, Peer Review

Every aspect of human endeavor that involves teaching newcomers how to do something involves both didactic and practical experiences. 

That is just the way it works.

Grant review is one of those things. Formal instruction only gets the job partially done. More learning takes place in the doing.

6 responses so far

Citation Curmudgeonry

  • In response to a post at Potnia Theron's blog:

26 responses so far

That study of peer review perplexes me

Apr 24 2015 Published by under Grant Review, NIH, NIH Careerism, Peer Review

I just can't understand what is valuable about showing that a 1%ile difference in voted score leads to 2% difference in total citations of papers attributed to that grant award. All discussions of whether NIH peer review is working or broken center on the supposed failure to fund meritorious grants and the alleged funding of non-meritorious grants. 

Please show me one PI that is upset that her 4%ile funded grant really deserved a 2%ile and that shows that peer review is horribly broken. 

The real issue, how a grant overlooked by the system would fare *were it to be funded* is actually addressed to some extent by the graph on citations to clearly outlying grants funded by exception.

This is cast as Program rescuing those rare exception brilliant proposal. But again, how do we know the ones that Program fails to rescue wouldn't have performed well?

23 responses so far

New plan for angry authors

Two years after your paper is published in Journal of SocietyB send the citation report showing that it quadrupled the JIF of the JournalA that rejected it to the rejecting Editor. 

Let's make this a thing, people. 

25 responses so far

Older posts »