Peer Review: Advocates and Detractors Redux

A comment on a recent post from Grumble is a bit of key advice for those seeking funding from the NIH.

It's probably impossible to eliminate all Stock Critique bait from an application. But you need to come close, because if you don't, even a reviewer who likes everything else about your application is going to say to herself, "there's no way I can defend this in front of the committee because the other reviewers are going to bring up all these annoying flaws." So she won't even bother trying. She'll hold her fire and go all out to promote/defend the one application that hits on most cylinders and proposes something she's really excited about.

This is something that I present as an "advocates and detractors" heuristic to improving your grant writing, surely, but it applies to paper writing/revising and general career management as well. I first posted comments on Peer Review: Friends and Enemies in 2007 and reposted in 2009.


The heuristic is this. In situations of scientific evaluation, whether this be manuscript peer-review, grant application review, job application or the tenure decision, one is going to have a set of advocates in favor of one's case and detractors who are against. The usual caveats apply to such a strict polarization. Sometimes you will have no advocates, in which case you are sunk anyway so that case isn't worth discussing. The same reviewer can simultaneously express pro and con views but as we'll discuss this is just a special case.

The next bit in my original phrasing is what Grumble is getting at in the referenced comment.

Give your advocates what they need to go to bat for you.

This is the biggie. In all things you have to give the advocate something to work with. It does not have to be overwhelming evidence, just something. Let's face it, how many times are you really in position in science to overwhelm objections with the stupendous power of your argument and data to the point where the most confirmed critic cries "Uncle". Right. Never happens.

The point here is that you need not put together a perfect grant, nor need you "wait" until you have X, Y or Z bit of Preliminary Data lined up. You just have to come up with something that your advocates can work with. As Grumble was pointing out, if you give your advocate a grant filled with StockCritique bait then this advocate realizes it is a sunk cause and abandons it. Why fight with both hands and legs trussed up like a Thanksgiving turkey?

Let's take some stock critiques as examples.

"Productivity". The goal here is not to somehow rush 8 first author papers into press. Not at all. Just give them one or two more papers, that's enough. Sometimes reiterating the difficulty of the model or the longitudinal nature of the study might be enough.

"Independence of untried PI with NonTenureTrackSoundin' title". Yes, you are still in the BigPIs lab, nothing to be done about that. But emphasize your role in supervising whole projects, running aspects of the program, etc. It doesn't have to be meticulously documented, just state it and show some sort of evidence. Like your string of first and second authorships on the papers from that part of the program.

"Not hypothesis driven". Sure, well sometimes we propose methodological experiments, sometimes the outcome is truly a matter of empirical description and sometimes the results will be useful no matter how it comes out so why bother with some bogus bet on a hypothesis? Because if you state one, this stock critique is de-fanged, it is much easier to argue the merits of a given hypothesis than it is the merits of the lack of a hypothesis.

Instead of railing against the dark of StockCriticism, light a tiny candle. I know. As a struggling newb it is really hard to trust the more-senior colleagues who insist that their experiences on various study sections has shown that reviewers often do go to bat for untried investigators. But....they do. Trust me.

There's a closely related reason to brush up your application to avoid as many obvious pitfalls as possible. Because it takes ammunition away from your detractors, which makes the advocates job easier.

Deny your detractors grist for their mill.

Should be simple, but isn't. Particularly when the critique is basically a reviewer trying to tell you to conduct the science the way s/he would if they were the PI. (An all to common and inappropriate approach in my view) If someone wants you to cut something minor out, for no apparent reason (like say the marginal cost of doing that particular experiment is low), just do it. Add that extra control condition. Respond to all of their critiques with something, even if it is not exactly what the reviewer is suggesting; again your ultimate audience is the advocate, not the detractor. Don't ignore anything major. This way, they can't say you "didn't respond to critique". They may not like the quality of the response you provide, but arguing about this is tougher in the face of your advocating reviewer.

This may actually be closest to the core of what Grumble was commenting on.

I made some other comments about the fact that a detractor can be converted to an advocate in the original post. The broader point is that an entire study section can be gradually converted. No joke that with enough applications from you, you can often turn the tide. Either because you have argued enough of them (different reviewers might be assigned over time to your many applications) into seeing science your way or because they just think you should be funded for something already. It happens. There is a "getting to know you" factor that comes into play. Guess what? The more credible apps you send to a study section, the more they get to know you.

Ok, there is a final bit for those of you who aren't even faculty yet. Yes, you. Things you do as a graduate student or as a postdoc will come in handy, or hurt you, when it comes time to apply for grants as faculty. This is why I say everyone needs to start thinking about the grant process early. This is why I say you need to start talking with NIH Program staff as a grad student or postdoc.

Plan ahead

Although the examples I use are from the grant review process, the application to paper review and job hunts are obvious with a little thought. This brings me to the use of this heuristic in advance to shape your choices.

Postdocs, for example, often feel they don't have to think about grant writing because they aren't allowed to at present, may never get that job and if they do they can deal with it later. This is an error. The advocate/detractor heuristic suggests that postdocs make choices to expend some effort in broad range of areas. It suggests that it is a bad idea to gamble on the BIG PAPER approach if this means that you are not going to publish anything else. An advocate on a job search committee can work much more easily with the dearth of Science papers than s/he can a dearth of any pubs whatsoever!

The heuristic suggests that going to the effort of teaching just one or two courses can pay off- you never know if you'll be seeking a primarily-teaching job after all. Nor when "some evidence of teaching ability" will be the difference between you and the next applicant for a job. Take on that series of time-depleting undergraduate interns in the lab so that you can later describe your supervisory roles in the laboratory.

This latter bit falls under the general category of managing your CV and what it will look like for future purposes.

Despite what we would like to be the case, despite what should be the case, despite what is still the case in some cozy corners of a biomedical science career....let us face some facts.

  • The essential currency for determining your worth and status as a scientist is your list of published, peer reviewed contributions to the scientific literature.
  • The argument over your qualities between advocates and detractors in your job search, promotions, grant review, etc is going to boil down to pseudo quantification of your CV at some point
  • Quantification means analyzing your first author / senior author /contributing author pub numbers. Determining the impact factor of the journals in which you publish. Examining the consistency of your output and looking for (bad) trends. Viewing the citation numbers for your papers.
  • You can argue to some extent for extenuating circumstances, the difficulty of the model, the bad PI, etc but it comes down to this: Nobody Cares.

My suggestion is, if you expect to have a career you had better have a good idea of what the standards are. So do the research. Do compare your CV with those of other scientists. What are the minimum criteria for getting a job / grant / promotion / tenure in your area? What are you going to do about it? What can you do about it?

This echos something Odyssey said on the Twitts today:

and

are true for your subfield stage as well as your University stage of performance.

6 responses so far

  • Comradde PhysioProffe says:

    Two thoughts:

    (1) F32s used to be reviewed by standing subject-matter based study sections that review R01s. Your concept of getting to know the applicant makes me think maybe the move to specialized fellowship sections was a mistake. And not to be a dicke, but also the caliber of scientists on fellowship sections is below the level of R01 sections, at least in my field.

    (2) I had a grant reviewed recently that was built around a particular hypothesis. The reviewers were all "this hypothesis is bullshittio, but the experiments are agnostic as to the hypothesis and will yield useful information, so we're good". Not clear whether I'd have been better or worse off had I not couched my proposed studies in terms of any hypothesis at all.

  • drugmonkey says:

    Number 2 is absolutely fascinating. It could be that you are enjoying the senior-investigator privilege ("yeah, this bit of StockCritique bait is overwhelming but dammit the Proffe Lab can make this into pure gold so let's fund it"). Or it could be that you got a bunch of reviewers who are tired of the "hypothesis-testing" orthodoxy themselves.

    I don't know if putting F32s back in the subject matter sections would help or hurt. one thing is that it would certainly reinforce the review of the F32 on the basis of the reputation of the supervising PI, presumably that person would have had a few grants pass by the section over recent years. And it could complicate it when the science of the F32 was too damn similar to the R01 of the PI that had just been awarded a year ago.

  • early stage postdoc says:

    Thanks for all this, this is incredibly helpful. We lurkers are out there taking notes.

  • I and my PI's experience with the same study section years ago was actually the opposite. He was a brand new PI when I got my F32, and it was reviewed before he had submitted any R01s. Then he started trying to get R01s based on similar proposed aims, and this same study section hammered the shitte out of him over and over and over. And the reasons never had anything to do with the fact that they had already reviewed/funded my F32. In fact, he never ever got R01 funding for that line of research, and eventually he abandoned it.

    I consider it likely that I received benefit of the doubt from the study section. What was also interesting was that they explicitly asserted that the fact that I had posed this particular hypothesis was because I am a newcomer to the subfield, and had I been more experienced, I would never have posed it. So presumably their confidence in my experience in the broader field allowed them to tolerate my newbieness with respect to the subfield. And this study section was populated by multiple experts in the subfield, including the person who actually founded the subfield.

  • Grumble says:

    "...I received benefit of the doubt from the study section..."

    This is precisely what happened to me with both the F31 and F32 fellowships I applied for. I proposed absolutely far-out, almost nonsense hypotheses (I'm still embarrassed by one of them) and the reviewers ate it all up - presumably because it was so different from more senior investigators' grants and the reviewers were inclined to some avuncular indulgence ("reminds me of me when I was that age, chuckle chuckle").

    Even back then, I'm pretty sure neither proposal would have gone anywhere as an R01. The question is whether F review panels are still so indulgent. My somewhat limited experience with them so far as a mentor suggests that they are not.

  • F panels nowadays are far from indulgent, precisely because they are ranking only F applications and have to spread their scores. When a study section is reviewing dozens of R01s and just a couple Fs, they can indulge the Fs.

Leave a Reply