Repost: Peer Review: Friends and Enemies

Jun 15 2009 Published by under Grant Review, Grantsmanship, Mentoring


This was originally posted June 22, 2007.


Perhaps "advocates" and "detractors" are the better terms. This is one of those heuristics that might help with crafting responses to the Summary Statement or the paper review. Others have views that touch on the topic for example MWE&G has the following in a recent post:

if you've hooked your primary reviewer into being a passionate advocate for your proposal, that will likely come through as well. If the summary statement lacks any sign of someone going to bat for your work, then you did not make your case even to the reviewer who should have been most excited about your project.

The heuristic is this. In situations of scientific evaluation, whether this be manuscript peer-review, grant application review, job application or the tenure decision, one is going to have a set of advocates in favor of one's case and detractors who are against. The usual caveats apply to such a strict polarization. Sometimes you will have no advocates, in which case you are sunk anyway so that case isn't worth discussing. The same reviewer can simultaneously express pro and con views but as we'll discuss this is just a special case. Etc. Nevertheless there are a couple of points which arise from this heuristic which apply to all of the above situations and suggest concrete approaches to both original presentation and, where applicable, in revising the proposal/manuscript. We'll take the case in which one is crafting a revision to a grant in response to a prior critique as the example after the jump.

Give your advocates what they need to go to bat for you.

This is the biggie. In all things you have to give the advocate something to work with. It does not have to be overwhelming evidence, just something. Let's face it, how many times are you really in position in science to overwhelm objections with the stupendous power of your argument and data to the point where the most confirmed critic cries "Uncle". Right. Never happens. So why write a response that seems to feel you can do just this with "rebuttal" language?
Let's take some stock critiques as examples. "Productivity". The goal here is not to somehow rush 8 first author papers into press. Not at all. Just give them one or two more papers, that's enough. Sometimes reiterating the difficulty of the model or the longitudinal nature of the study might be enough. "Independence of untried PI with NonTenureTrackSoundin' title". Yes, you are still in the BigPIs lab, nothing to be done about that. But emphasize your role in supervising whole projects, running aspects of the program, etc. It doesn't have to be meticulously documented, just state it and show some sort of evidence. Like your string of first and second authorships on the papers from that part of the program. "Not hypothesis driven". Sure, well sometimes we propose methodological experiments, sometimes the outcome is truly a matter of empirical description and sometimes the results will be useful no matter how it comes out so why bother with some bogus bet on a hypothesis? Because if you state one, this stock critique is de-fanged, it is much easier to argue the merits of a given hypothesis than it is the merits of the lack of a hypothesis. "Applicant really needs to add the following control conditions and these extra manipulations..." Yes, yes, the reviewer is out to lunch and you can dispose of these silly observations with a few pithy rejoinders. Don't do this. It puts your advocate in the hole. Just give them something, after all, you don't have to actually do everything you propose!

Deny your detractors grist for their mill.

Should be simple, but isn't. Particularly when the critique is basically a reviewer trying to tell you to conduct the science the way s/he would if they were the PI. (An all to common and inappropriate approach in my view) If someone wants you to cut something minor out, for no apparent reason (like say the marginal cost of doing that particular experiment is low), just do it. Add that extra control condition. Respond to all of their critiques with something, even if it is not exactly what the reviewer is suggesting; again your ultimate audience is the advocate, not the detractor. Don't ignore anything major. This way, they can't say you "didn't respond to critique". They may not like the quality of the response you provide, but arguing about this is tougher in the face of your advocating reviewer.

Give the critical reviewer the excuse s/he needs to shift to advocacy.

Perhaps more of a subcategory of the above two points but worth addressing for one reason. It happens. Frequently. A reviewer that may sound highly critical on the first round may switch to favorable on the next revision; even without the sort of fundamental change that s/he is apparently envisioning. There are many reasons for this and it relates to the overall high quality of grants well down into the 60th percentile and the inevitable competition between grants stacked up in the given reviewer's pile of applications. Not to mention a huge bias for revision ("I'd like to see this one come back"). The point for the current discussion is that number one you don't want to insult the critical reviewer too badly and two, this is why you need to address all concerns as best you can. This way even the critical reviewer can backtrack with little loss of face "Well, the applicant didn't fully satisfy the critique but they did x,y and z which I have to admit is an ideal compromise under the constraints". This happens. Frequently. Otherwise how would the vast majority of grants be funding after revision?

Plan ahead

Although the examples I use are from the grant review process, the application to paper review and job hunts are obvious with a little thought. This brings me to the use of this heuristic in advance to shape your choices. Postdocs, for example, often feel they don't have to think about grant writing because they aren't allowed to at present, may never get that job and if they do they can deal with it later. This is an error. The advocate/detractor heuristic suggests that postdocs make choices to expend some effort in broad range of areas. It suggests that it is a bad idea to gamble on the BIG PAPER approach if this means that you are not going to publish anything else. An advocate on a job search committee can work much more easily with the dearth of Science papers than s/he can a dearth of any pubs whatsoever! The heuristic suggests that going to the effort of teaching just one or two courses can pay off- you never know if you'll be seeking a primarily-teaching job after all. Nor when "some evidence of teaching ability" will be the difference between you and the next applicant for a job. Take on that series of time-depleting undergraduate interns in the lab so that you can later describe your supervisory roles in the laboratory.

7 responses so far

  • msphd says:

    Some interesting points. I particularly like the idea of responding to the advocate rather than the detractor. Shifts the frame of mind towards the positive, and I've never heard it before.
    However, it's obvious that this is a report, since I found a few typos, and the 'plan ahead' bit aimed at postdocs is a bit out of date.
    Some (many?) of us do pay lots of attention to how we will need to write our grants. And as I've written over and over again, some of us even "help" write them now, as part of our 'training'.
    Also, you don't allow for the possibilities of
    (a) stubborn advisor who insists that paper should be BIG even if it takes forever. Why do you assume that the frequency of publication is entirely within our control???
    (b) angry advisor who does not want you taking time away from the bench to teach a class; fellowships that do not allow for less than 100% bench time!
    (c) supervising undergraduates that doesn't really count as teaching experience enough to get a job at a primarily-teaching school (or does it?).

  • DrugMonkey says:

    Hmm, not seeing where the thing about postdocs is "out of date". Still applies from where I sit.
    Also, you don't allow for the possibilities of
    I allow for all possibilities. Never do I suggest that my observations apply to each and every case with high specificity. I make observations which I think generalize quite well but this is not the same as matching every possible career arc closely.
    Why do you assume that the frequency of publication is entirely within our control???
    I do not. It is not entirely out of the hands of postdocs either. Some have an intrinsic BIG PAPER orientation themselves. Some are slow to write. Some PIs are receptive to arguments from postdocs on the careerism front that the times are a-changin' . The fact that YMMV does not mean this is all-or-none advice.
    I will also note that even your perspective falls under my general comments. Sometimes it is worth pointing out as clearly as you can that you are in a lab that only publishes in CNS and that this necessarily decreases publication frequency. I have seen something very much like this occur in discussion of a grant. You might think nobody would question productivity rate when looking at a CV with CNS pubs but I've seen it happen. An advocate for the grant proposal was quite capable of making the obvious argument but it is always best to make the case for her/him, especially if responding to criticism.
    supervising undergraduates that doesn't really count as teaching experience
    Supervising others in the lab goes into the argument about independence, not teaching. You use it to shore up the contention that you can run a research program instead of just being a lone gun geeking away at your bench. As I believe was discussed by PiT somewhere or other, this also goes into your argument in the early days of your faculty appointment that you have training experience. Handy for all kinds of reasons, particularly AREA/R15 and NRSA apps from your group.

  • Alex says:

    angry advisor who does not want you taking time away from the bench to teach a class; fellowships that do not allow for less than 100% bench time!
    First, while every institution and grant has its own rules, there are generally ways to teach. Generally speaking, the paperwork often maintains some fiction that a postdoc is working "full time", i.e. 40 hours/week. For Ph.D. students, the fiction at most schools is that the student is half-time, i.e. 20 hours/week. (And a Ph.D. student who has completed a Masters is eligible to teach part-time at many undergraduate institutions.) So a class taught in the evening is obviously not a violation, right? That's part of how I justified teaching a class at a different (nearby) institution while I was a postdoc.
    Teaching at your own institution is trickier, of course, since it will take you above the limit of 40 hours/week (or 20 hours if you're a student). Also, there are often rules about moonlighting. Sometimes you can get around one or both of those rules by saying that you are teaching as part of your training program. That's part of how I justified my teaching experience as a postdoc. It's less likely to fly with an NIH-funded program, but if we're talking about NSF money, well, NSF has a very strong educational mandate, so you write some statement about "Preparing for a career disseminating science to impact the broader society blah blah" and maybe it will fly.
    If you do manage to teach at your own institution and justify it as part of your training, there's a chance that you won't be able to accept pay for it, but, well, the pay for part-time teaching is crap anyway (of course, I realize that every dollar is precious to a postdoc) and the real benefit is that it increases your employability down the road.
    If you can't teach at your institution, look at nearby institutions. Community colleges are obvious possibilities. Undergraduate colleges are also possibilities. And, believe it or not, even Ph.D.-granting institutions other than your own are possibilities. I contacted all of the local department chairs and asked if they needed any part-time lecturers, and one of them in a Ph.D.-granting department happened to be interested. Surprisingly, they didn't need any lecturers for freshman courses, but he saw my CV and thought I'd be a perfect fit for a graduate course. So that's what I did.
    Of course, commuting between a postdoc and a teaching post is hell on your research productivity. I didn't do it until I had a paper published and another under review. It still shot my productivity to hell, but at least I already had something to show, and it helped me get a job at an undergraduate institution where I am quite happy (assuming the state budget doesn't go to hell and we all lose our jobs).

  • John W. says:

    The "planning ahead" section doesn't sound very realistic because it assumes an ideal world where postdocs in general have the freedom to determine their professional activities. In reality many postdocs have very little say in how they spend their time. Where/when your papers are published and what activities you do besides your own bench research are definitely not up to you. Many PIs in my field treat postdocs as employees who are supposed to do as told and nothing more and to not even dare to have any self interests (otherwise you will be fired and replaced with another postdoc who will toe the party line since there's no shortage of potential employees). If you have time to teach a class then obviously you are not giving as much to your paid position as you could/should have (no matter how many hours beyond 40 you've already put in that week). Ditto about writing grants on your own (though ghostwriting grants for the PI is expected as it is part of your paid job).

  • DrugMonkey says:

    John W., YMMV of course. Every place that I've been around there were people seeking teaching opportunity with at least the acceptance (and often encouragement) of the PI. Papers are a negotiation, of course, and the PI has ultimate say. But nevertheless I have found that there is substantial opportunity for the trainee to make things happen. PIs have a way of softening their position when something like a ready-to-submit manuscript is on their desk, for example.
    As usual, I suggest as gently as I can that if you can only see one possible culture of science (and it isn't working for you) you need to broaden your vision. To see that there are other cultures within biomedical science where life might fit you a little bit better.

  • zayıflama says:

    It is not entirely out of the hands of postdocs either. Some have an intrinsic BIG PAPER orientation themselves.

  • [...] will be revealed for what they are. As I’ve said before, if your grant is to have a chance you are going to have to have won over at least one advocate. Perhaps several. One of these advocates is going to rebut misplaced comments during the course of [...]

Leave a Reply