Reviewer mindset 

Mar 22 2016 Published by under NIH Careerism, Peer Review

I was just observing that I'd far rather my grants were reviewed by someone who had just received a new grant (or fundable score) than someone who had been denied a few times recently. 

It strikes me that this may not be universal logic.

Thoughts? 

 Is the disgruntled-applicant reviewer going to be sympathetic? Or will he do unto you as he has been done to?

Will the recently-awarded reviewer be in a generous mood? Or will she pull up the ladder? 

25 responses so far

  • The Other Dave says:

    I agree with you, for both scientific and psychological reasons.

    If someone just got a fundable score, they are more likely to be aware of the literature and doing good science. Those are the people who should be reviewing proposals.

    I think it's also natural to compare one's own proposals to those that are being reviewed. Most people like their own proposals (I hope!). And if ours didn't deserve funding...

    But then... I once suggested to a program officer that study section service should be mandatory for all grantees. He said yea that sounds like a good idea until you realize that some people are good scientists but horrible reviewers.

  • BWJones says:

    In total agreement. Elbows are getting sharp as funding gets thinner and it seems... (non-scientific data) that the review process for grants and manuscripts is getting more contentious as well. As least more contentious than it was just a couple of years ago...

    It does not help that we are in the noise on which grants get funded it seems... Friend back from recent study section wanted to go home and drink because everything they triaged was better than his funded grants.

  • Grumble says:

    Everyone who has just received a new grant or fundable score has also just been fucked 10 times in a row before this one success. So it makes no difference.

  • serialmentor says:

    @BWJones Indeed, every time I serve on a panel I go home wondering how I ever got a grant funded.

  • Imager says:

    At this stage its a lottery anyways. One reviewers doesn't like the technique/method you are proposing because he is from the other side of the divide and you are out, even though its a totally viable and accepted method to do things next to the other one. One guy just rises some slight issue and you are out, even if it is an addressable issue or even a non-issue. One guy thinks this is not relevant (but he is not in the field so doesn't really know) - and you are out. I have even written in grants a whole paragraph why certain things are not an issue (based on prior reviews), added statements with letters from colleagues and references from the literature only to hear the same stupid comment in the next review all over again...
    This is not a scientific review any more, its a "how can I kill this grant now" task force. And in that respect I'd rather have someone who just got lucky and doesn't need to worry for the next year or so and who can be generous and realistic than a disgruntled guy (me included).

  • Imager says:

    @ Grumble - so true.

  • BWJones says:

    I would be more comfortable with a lottery, actually. Once a grant makes it through triage... .

  • drugmonkey says:

    Aren't you people supposed to be cheering me up with optimism?

  • jmz4 says:

    @BW you'd probably have to move the triage score down to 25% or so.

  • Philapodia says:

    @DM

    Why would you think that? We all just come here to bitch.

  • Pinko Punko says:

    I just want reviewer to be engaged. If they feel wronged then they can't complain about it if they turn around and enforce stick critiques. That is my philosophy. Do best job I can and never give applicant a nothing burger. Those don't help program either.

  • Ola says:

    For the study section I sit on, having an active grant that was funded by that specific section is kind of the price of entry, even for ad-hocs. Maybe because we're all funded explains why we're constantly getting grief from the SRO for our average scoring being too high?

    Actually there's a new project for data hound - do average scores from study section rounds correlate in any way with the funded/not-funded status of the reviewers that make up the panel? Would need to pull average scores from CSR and cross-ref with panel rosters and NIH RePorter.

  • drugmonkey says:

    This is a project for Richard Nakamura, frankly.

  • eeke says:

    I found myself defending criticisms for things that I had been dinged for and felt that they were wrong. Even though my own grants got sunk, I didn't feel the need to be petty and sink others just because. If I can help someone else get through, it's a good thing. The sad thing is that there are so many excellent grant applications and only a few of them will be funded anyway. I really don't think it matters whether the reviewer recently had funding or not - a great score does not guarantee funding.

  • Emaderton3 says:

    @ Imager

    I hate to say it, but I kind of agree with your statement about finding ways to "kill a grant." I just got my summary statement back for a resubmitted R01 that I really tried to cater to the first review. I know individual reviewer scores do not always correlate with overall impact score, but to get "not discussed" while receiving all 2-4's from the individual reviewers really depresses me. Ugh. I have always had grants discussed with those type of scores. It is getting so ridiculous at this point. I don't think it matters either.

  • Susan says:

    I'll bite, this is me.

    Funding of my grant didn't happen in a vacuum; it happened after multiple submissions. In fact, the very same grant got triaged at one N** but got very funded at another N**.

    Likewise, I am the same person whose efforts have gotten scored, triaged, funded, and chucked. I have plenty of disgruntled left in me. I will speculate that my grant writing and my grant reviewing have improved in parallel. But I haven't become a different, kinder gentler person overnight. The systemic problems with funding (too many applications, not enough money, glamour and Ivy humping) that affect review are still there, and so are the problems with any given grant.

  • baltogirl says:

    Much rather have someone whose grant has been funded. They will be in a better mood! Yesterday I talked to someone whose grant was reviewed by someone whose grant he had severely criticized in a prior review (not sure how he knew, but he did. Personally, I never know who is reviewing my grants because I don't recognize any names on the roster- the people in my field are too busy writing their own grants to serve). Anyway, this person did not feel the review was fair and so when it was his to turn to review my friend's grant, he triaged it.
    Moral of the story may be to stick up for your field or don't serve, because it may come back to bite you...

  • drugmonkey says:

    If this sort of specific payback reviewing occurs...

    well, this is not a good look, scientists. Don't countenance this nonsense.

  • baltogirl says:

    It is indeed quite depressing. But doesn't everyone know people who are afraid to serve because of this? If you have even one negative comment these days, the grant gets torpedoed. How to handle? Perhaps dm has ideas..

  • drugmonkey says:

    I do not have any idea how to handle the payback issue.

    I have heard enough direct ranting from my peers (that should know better) about how so-and-so "killed their grant for unfair reasons" to be utterly convinced that in the yuge entity that is NIH, there are going to be payback issues in grant review now and again.

    I made up my mind pretty early in my first few stints on study section that I was not going to worry about payback from disgruntled applicants or from other members of the study section that I pissed off. I'm just not the kind of person who can manage to shut up about things just because it may cause personal risk to speak out. If I were advising anyone, I'd tell them to be a lot more circumspect than I am.

    I have had one seeming threat make its way back to me. Rather amazing that it is just the one*. It was while I was still on study section and communicated via one of my closest science buddies so I have reason to believe it was real and specifically directed at me. My friend never would divulge who was sounding off about me so I have no idea what it was about.

    Undoubtedly, I have suffered in review from the impact of things that I have done as a reviewer. It would be very weird if I have not.

    Undoubtedly I have suffered in review from things that I have written as DM on this blog or on the twitts. Many of my peers who review my grants know full well who DM is.

    But you just can't go through life in a defensive crouch.

    __
    *I have also had a number of individuals who I like and respect as scientists who have patted me on the back for my mouthiness at study section. This is nice to hear, of course, but even better it heartens me that people understand when someone is working to make review as good as it can be. Even if they don't actually agree with my positions.

  • qaz says:

    The problem with revenge is that it assumes you actually know who is being unfair to you.

    Every time anyone has approached me to complain that they think I "killed their grant/paper for unfair reasons", it hasn't been me. (More often than not, I was never a reviewer on that paper. But in one grant situation, I was on the study section but I was the one who argued FOR the grant and kept it from being triaged because I thought a different reviewer was being unfair - not out of revenge - just because xe didn't think bunny hopping was valid science.) An editor once told me that people who called to complain that the editor should have sent the paper to friend B instead of to enemy A were usually wrong and that it had been sent to both and person A had liked the paper and person B didn't. I long ago gave up holding grudges because I got convinced that I was usually wrong. I simply do my best and I assume my colleagues are also.

    One of the reasons I like anonymous review is that it lets me (as the author) not know who the pain-in-the-a** is who's holding up my paper and I have to assume that the reason they are complaining is because there's something wrong with my writing that I have to fix.

    In my experience, study section is generally extremely fair and generally providing the best reviews under trying circumstances. This is why it is so important for people (let the assistant professors back!) to gain experience being at study section.

    My mantra is that Complaining about bad reviews doesn't help you get funded. It just makes you feel bad. Next time, try to figure out how to communicate to them better.

  • Lurking SRO says:

    In my experience, applicant's concerns about specific reviewers are almost always unfounded. Their perceived "enemies" are often their friends in review or at worst neutral.

  • DiddlySquat says:

    "Every time anyone has approached me to complain that they think I "killed their grant/paper for unfair reasons", it hasn't been me."

    I have heard this too from other people.

  • The Other Dave says:

    I once denied a postdoc entry to a prestigious training opportunity because I honestly didn't feel he was as well qualified as other applicants. Unfortunately, this pissed off that postdoc's PI. I like that the PI was supportive of his postdoc, but I didn't like that he called me up to tell me that her wouldn't be doing me any favors in the future. A couple years later another guy called me up and told me to list that PI as a conflict of interest on grant proposals, because that PI was a reviewer and totally (and unfairly, in this other guy's opinion) destroyed the chances for my grant to be funded. So I started listing that PI as a conflict. I have no idea whether it helped me get funded.

    The reason this other guy called was not to tell me about that other PI's bad behavior. Rather, this guy wanted to thank me for the nice letter that I wrote in support of his tenure case. Apparently his department head told him that I did a nice job for him. He wondered whether I would be willing to write another letter for him. Of course I was happy to.

    I think all this is weird and inappropriate and wrecks the system. But stuff like this doesn't surprise me anymore, either.

  • Curiosity says:

    This head-game that is he/she scored me well, so I'll score them well, or vice versa is bad bad bad. I am congenitally unable to hold my tongue and if experimental plans strike me as poor I will say so. But when I criticize the work of a supportive colleague it pains me deeply, like I have betrayed them. I force myself to stop feeling wretched though because the review process should not be a popularity contest, right?

Leave a Reply