# My philosophy on annoying NIH grant reviews in a nutshell

We've been talking about grants getting spiked by one outlying, jerk of a Reviewer #3 lately. Here and elsewhere. I have one guiding philosophy when it comes to disappointing grant reviews.
Pardon my PhysioProffish.

Fuck the battle, win the motherfucking WAR!

• True dat! Sack the fuck up and git er done. Also, if your lab finances are relying upon one single particular R01 to live or die, your doing it wrong.
I had a whole long discussion with someone the other day who was moaning and groaning endlessly about how her BESTEST PROJECT EVAR was getting frowned at by program at several different 'tutes and that none of them wanted to even accept its assignment. She was complaining that this was totally RUINING EVERYTHING and that it was going to KILL HER SCIENZ.
I told her she needed to sack the fuck up and write the thing up in a different way. If you can't think of at least five different angles from which to propose a research project that are responsive to the stated interests of at least one 'tute, then you have a serious creativity and cleverness problem.

• wh says:

if your lab finances are relying upon one single particular R01 to live or die, your doing it wrong.
Or, more simply, if your lab finances are relying upon one single particular R01 to live or die, your lab will die, it's just a question of how soon. This is why 10% paylines are unsustainable... all those single R01 labs are going to disappear, but the tenured faculty heading those labs won't.

• gummibears says:

You are right, of course, but within your analogy substandard/biased reviews are equivalent to war crimes, and should be treated accordingly.

• Joe says:

Or, more simply, if your lab finances are relying upon one single particular R01 to live or die, your lab will die, it's just a question of how soon. This is why 10% paylines are unsustainable... all those single R01 labs are going to disappear, but the tenured faculty heading those labs won't.
This is an interesting issue. If people started being satisfied with only one R01, paylines would instantly jump to at least 20-30%.
So why aren't people satisfied with one R01? Why do they want more?
1) Overlap ensures funding continuity. You can lose a grant, or have a gap in funding one aspect of your research, but you'll still have some money. Lab funds are fungible. You can always sneak money from one grant to get preliminary data for another. But if you have zero grants, you are SOL.
2) There are a lot of soft-money positions these days, and the suckers who took them find that their first R01 is sucked up mostly to pay themselves. ...to write more proposals 😉
3) Money buys more than what money buys. It's amazing how institutions can free up time, space, and other resources for PIs who are bringing in the big bucks.
4) More money boosts you into the big leagues. With more money, you can hire more people, buy more expensive frontier-pushing technology, and have more time to push your story at seminars and meetings. You'll be seen and heard, and are more likely to publish in higher-profile journals. These are huge dopamine squirts, and like crack addicts, we'll always crave more.
From the taxpayer perspective, of course, none of these are good reasons to increase the NIH budget, or even support it as it is now. #1, if you want to be a stickler, is technically mis-use of federal funds. #2 is questionable subsidies for universities and research institutes. #3 is basically bribery and graft. #4 is simply greed.
So maybe NIH should cap the number of allowable grants? Force people with big needs to be honest about it and write big proposals which will undergo appropriately bigger scrutiny?
I think so. The navy doesn't get to ask for 100 million kevlar vests but use the money to build a destroyer instead. And people's lives actually depend on what the military does. Why do NIH applicants get to routinely lie about what they 'need' money for? When all that's really usually at stake is the applicant's ability to continue a very expensive hobby?

• pinus says:

Whining about substandard reviews is a waste of time.
We get it...You got one, move on. Raging against this particular machine is just a waste of valuable time and energy.

• gummibears says:

"Whining about substandard reviews is a waste of time."
No. It's defining the problem. Sweeping the problem under the carpet is a waste of time and resources. If the machine is broken, it should be either fixed or turned off, but first you need to notice that it is broken. And what are we having here instead? Constant whining about the treatment of new investigators, about "third reviewers", about outliers, about biases, but NOBODY DARES to reduce these facts to the following, logical common denominator: At NIH, the objective truth and the scientific merit of an application play an unacceptably small role.

• whimple says:

At NIH, the objective truth and the scientific merit of an application play an unacceptably small role.
Ok. I'll bite.
1) What makes you think "objective truth" exists with regard to evaluating the quality of science?
2) Do have any non-circular metrics for determining "scientific merit"?
If you're going to whine, at least do so productively. 🙂

• gummibears says:

Whimple,
I am not naive and I know that achieving 100% of the desirable outcome is not possible, for various reasons. I am referring to the other side of the spectrum: the refusal to condemn OBVIOUS cases of reviewers' misconduct.
I may indeed not be able to define standards of objective truth and scientific merit. But in some cases it is evident that a review lacks the concern for such truth - e.g. when a review is self-contradictory, when it disregards elementary facts that can be found in college textbooks, or when a reviewer purposefully misrepresents the contents of the proposal to construct a damaging critique.

• gummibears says:

I do not expect the review process to be PERFECT. But I am not going to give up on certain minimal standard: critiques must be at least free of evident nonsense and bias, and mechanisms for enforcing such standard should exist.

• DrugMonkey says:

gummibears, my understanding is far from complete but I've heard stories of POs complaining to SROs. enough to credit it as probably true.
so the mechanism works like this- you complain to the PO after the fact or to the SRO just after assignment to her section, while there is a chance it hasn't been sent to a reviewer. that SRO, if convinced, takes a gander at the review and decides not to send your app back to that person. maybe, if it was an ad hoc, fails to invite that reviewer back.
if you make your point and the SRO still assigns your revision or subsequent grants to that reviewer you may conclude that a judgment has been made that you are unjustifiably whining.
Now, I admit that it would be *better* if clearly flawed reviews could be "fixed" without revising the app. but I really don't see how you do this without the whole system collapsing into the chaos as the 80% of all submitted apps claim a flawed review.

• Joe says:

I feel your pain, gummibears, but reviewers are human, and they're not perfect. Even well-meaning reviewers (and I believe the vast majority are indeed well-meaning) can screw up.
Your job, in writing the proposal, is to minimize the chances of this happening.
That's why CPP has posted his wonderful guides to application writing. One of the best things about his guides are his explanations of what reviewers typically see, and therefore expect to see, in each section of the proposal. If you give reviewers what they expect to see where they expect to see it, it helps avoid confusion. This is also why it's useful to read other people's (successful) proposals. Follow their lead.
Besides not confusing reviewers with an unconventional proposal, you also have to clearly justify your work and explain what you plan to do. If a reviewer doesn't 'get it', it's because you didn't explain it clearly. If they don't think your work is important, it's because you didn't justify it adequately.
Never forget that you're begging for money when you write a proposal. Teachers in inner-city schools routinely have to buy pencils and paper and books for their students. Some people don't have a place to live. Some people don't have enough to eat. Some troops in far-off places are out of bullets and getting shot at. Some people can't afford the medication they need to live comfortably, or just live, period. You're asking all of these people for money, or at least asking the government to divert money from these people to your research program. You've got to believe you're worth it, and convince them too.
Biomedical science is not necessarily a victimless crime.

• whimple says:

Biomedical science is not necessarily a victimless crime.
If things stay as they are, cancer will kill 1/3 of the study section members. Not funding my translational human subjects cancer project is not necessarily a victimless crime either. 🙂

• gummibears says:

Joe,
There's no pain. The very same stuff that provoked so many amusing cretinisms from the NIH review panels, was later funded without any fuss by another agency (which, by the way, has even lower success rates).

• Anonymous says:

Joe @11
---"Never forget that you're begging for money when you write a proposal......"--- etc etc
Really? Begging? That's how you look at it?
Firstly, none of what you said in that para has anything to do with the price of tea in China....unless each proposal actually diverts more money into research than has been approved and appropriated. You are not trying to convince reviewers of the worthiness of doing biomed research...you are trying to convince reviewers of why your project is more meritorious than the others in the pile.
The issue w.r.t. review is about the allocation of said money amongst a pool of apps....and platitudes like 'sack the fuck up' or 'fuck the battle, win the war' are meaningless when you find yourself playing a game where the goalposts are essentially moving. Hey, you want better healthcare?...just sack the fuck up! You want better justification for trillion dollar wars?....well, sack the fuck up!! There is nothing in the system that cannot be overcome....if, in the interim that may be crucial to your existence, you are unable as an individual to overcome it, then you must have just needed to 'sack the fuck up'!!
This is a fucking panacea!!
---"If a reviewer doesn't 'get it', it's because you didn't explain it clearly"---
As a generalization, that's got to be the truth! I think that should be included as a disclaimer in every summary statement!
Also, BTW, how about this? Let everyone quit 'begging' for money for biomed research. Let the good pharma companies come up with remedies all by themselves, from scratch, for everything.
You know, because 'the market' will drive everything to a satisfactory conclusion eventually with only the private sector, operating only with a view to monetary profit, running things.....a fucking utopia if I ever dreamt one.....

• DK says:

What makes you think "objective truth" exists with regard to evaluating the quality of science?
If you think about it for a second, you will realize that of course it exists. It has to. Else, science does not exist either. Clearly, the difficulties of identifying objective truth say nothing about its existence.
Do [you] have any non-circular metrics for determining "scientific merit"?
Sure. It's easy to do a posteriori. A funding body (the society as a whole in most cases) simply asks: Is this outcome something we (the society as a whole in most cases) wanted to have in the end? There will be a distribution of answers, but the higher is the probability of the positive answer, the more scientific merit there is. (Coming from another end of it, this means that if it's not something someone cares in any way, shape or form, then regardless of how mind-twisting it may be, we have no way of knowing if it is a science or not; hence in practice we'll assume that it's not). The difficult part is predicting the scientific merit.
Now go back to the original statement made ("At NIH, the objective truth and the scientific merit of an application play an unacceptably small role") and I hope that you now see that the only difficult part of evaluating it resides with defining "unacceptably small". My subjective POV of an average the taxpayer schmuck says that NIH's waste is unacceptably high.

• DK says:

That's why CPP has posted his wonderful guides to application writing. One of the best things about his guides are his explanations of what reviewers typically see, and therefore expect to see, in each section of the proposal. If you give reviewers what they expect to see where they expect to see it, it helps avoid confusion. This is also why it's useful to read other people's (successful) proposals. Follow their lead.
Yeah... Everyone's complaining how the system sucks and everyone is performing to the best to keep the system going. To me, the whole thing always looked analogous to various hazing rituals. "They fucked me, now I will fuck others - that's life as we all know it". Explains a large proportion of what's going on. Of course, nobody admits it explicitly. Instead, we get fairy tales of "I have now matured, I understand how the system works and what it takes to make it work - and, on a whole, I see the system working rather well". The "for me" thing in the end is invariably omitted, though. Case in point: CPP's guides to grant writing. His smugness is a force of nature, that's for sure.

• whimple says:

You know, because 'the market' will drive everything to a satisfactory conclusion eventually with only the private sector, operating only with a view to monetary profit, running things.....a fucking utopia if I ever dreamt one.....
Yeah. Thank God academic biomedical science doesn't work in e.x.a.c.t.l.y. t.h.e. s.a.m.e. w.a.y...

• BTW, #14 was my comment--I forgot to sign in--and if it wasn't clear, the latter parts of that comment was pretty heavy on sarcasm.
Anyway, whimple @17, I'm not sure I got the point....how does academic bioscience work exactly like industry/pharma science?

• whimple says:

They are both driven completely by cash. The difference is that with industry the consumer (of the product sold for cash) is the general public, while in academia the consumer (of the grants "sold" for cash) are academic scientists. No cash = out of business, in both arenas.

• Dr. O says:

Your job, in writing the proposal, is to minimize the chances of this happening.
Joe, I couldn't agree with this statement more. I just got some reviews on my grant that were all over the place, and I'm putting the onus on myself for not making things clear enough. But how do you prepare for people who aren't going to take at least a minimal amount of time to understand your proposal? I had 6 different individuals (student, MD, senior research associate, junior prof, and 2 senior profs) read several drafts of my proposal and worked very hard to address even the most minor concerns they had. One of my reviewers even felt there were no major weaknesses and rated the proposal very highly. Another, however, believed my application was too difficult to understand, and rated it incredibly low. I will do everything I can to address this reviewer's concerns (including jumping through hoops of fire), but it's difficult to understand this kind of disparity.

• Namnezia says:

Dr. O said:

One of my reviewers even felt there were no major weaknesses and rated the proposal very highly. Another, however, believed my application was too difficult to understand, and rated it incredibly low. I will do everything I can to address this reviewer's concerns (including jumping through hoops of fire), but it's difficult to understand this kind of disparity.

To me this says that the bad reviewer was a bit lazy. I do totally see why one should make a grant as easy as possible for the reviewer to read (they are reviewing many grants, they are volunteering their time, etc.). But this coddling of reviewers can only go so far. I mean, are they funding what they think is the best science or simply whoever writes the most eloquently and simplistically? Obviously Dr. O's grant (which sounds like it was a K99) was not impossible to understand since it was vetted many times over by experienced people in her department and one of the reviewers gave her excellent reviews. So I think that the second reviewer was just being lazy when he/she came across something that was out of his/her comfort zone. Not much you can do in this case other than trying to make the grant even clearer and point out the discrepancy (nicely) in the introduction to the revised proposal, but still not quite fair.

• His smugness is a force of nature, that's for sure.

Dude, that's not "smugness". It's called "being right".

• Joe says:

Alas, Dr. O, there are no guarantees. Submitting a proposal is always a gamble. Even loaded dice throw craps now and then.
But is it not still an awesome game?
Remember what a biomedical science job offer is. Someone is saying: 'Here's a half million dollars (or more). Have fun. Change the world.
wooooooooot!
I love science. I love submitting proposals. I love telling people all the slick things we can do to figure stuff out and make the world even more incredible than it already is. I love that they read pages and pages of my stuff. I love that they think about it. I love that they tell me what they think about it. I love that sometimes they think my dreams and promises are worth millions of dollars of taxpayer money (or hundreds of thousands of dollars of hard-earned charity donations). And I love doing it over and over and over again until I die, face down on my desk with a half-filled out animal use protocol form stuck to my eyeball.

• gummibears says:

Joe,
Do you consider normal and acceptable that the taxpayer money you mentioned is distributed with the participation of certain "colleagues", who feel free to write utter nonsense in the critiques? Do you consider it beneficial, when the institutional culture at NIH suppresses research on novel methodologies, in favor of the hypothesis-driven "same methodology, another subject"? (the latter is, actually, an allegation an NIH Program Officer made when we are discussing something over the phone, not just my whining). We have a finite amount of money given to us by the taxpayers, and we should, I believe, maximize the output (out of pure gratitude), instead of just treating science as an awesome game.

• Whimple @19,
I disagree. Both are not driven by cash. The pursuit of both may require cash, but that's a big difference. The pursuit of virtually anything on a sustained and organized basis requires cash.
As far as 'driving' forces go, academia and industry are more dissimilar than similar. Academic research is chiefly driven by the desire to discover new facts and thereby expand the body of knowledge. Industry research is driven chiefly by the desire to create new products that will earn revenue---the acquisition of knowledge in this case is focused on what is required to get the product to market and also what will help to keep the company from getting sued in the future.
Even from the standpoint of allocation of funds for projects, you'd be pretty surprised how fucked up it is in industry. If you think projects get promoted and funded based on merit alone, guess again. If you think promising projects do not languish agonizingly because of petty politics, guess again. If you think there is not wastage of money in the course of which projects are promoted versus killed, guess again.
Academia has its problems, as we all discuss, but it is not close to as screwed up as bio/pharma industry is today.

• Namnezia says:

Gummibears - what's wrong with "same methodology, another subject"? Why can't that be considered cutting-edge science if you are uncovering knowledge that was previously unknown? There's no point in developing new methodologies if you don't plan on exploiting them at the fullest. I'm all in favor of new techniques, but I am not in favor of the view that unless a study is pushing the technological envelope then it is bland, "safe" science. This is the philosophy of many glamour magz and "hot labs" who care more about technique than about actually answering a fucking scientific question. Excellent, cutting-edge science can be done just as well with old or new methods, cheap or expensive. It's just a matter of finding the best technique to answer your scientific question, NOT the other way around, and also of being creative with your resources. That being said NIH has plenty of mechanism to support development of new methodologies, for example these "Eureka" or "Pioneer" awards. Or even an R21. NSF and DoD have similar programs. I know several people who have benefitted from these mechanisms, so they are not as difficult to get as they make it seem. Why don't you apply for these?

• Anonymous says:

Academic research is chiefly driven by the desire to discover new facts and thereby expand the body of knowledge.
Not according the Dean of our College of Medicine. What counts are things can be measured numerically: 1) number of patients discharged, 2) number of medical students, 3) extramural (especially NIH) grant dollars.
That's awesome for you if by happy coincidence the voyage of discovery correlates with grant dollars, but if those ever decouple, you damn well better go with the grant dollars. We know exactly how to quantify how good your science is... we just add up all your extramural cash.

• whimple says:

#27 is me, obviously.

• gummibears says:

Namnezia: you are trying to reduce my point ad absurdum. I am not advocating not applying these methodologies (old and new) to particular practical problems. I am just alleging that the pendulum at NIH (this includes also these R21s and Eurekas) swung way too much toward the "same methodology, yet another subject" philosophy, and the institutional culture there suppresses research on novel methodologies.
There are subdisciplines with already adequate methodologies, but there are also whole areas, where methodologies are still immature and under constant development. The above-mentioned NIH program officer, for example, complained about the systematic suppression of systems biology. I have noticed the same with respect to my line of work (protein design).

• Whimple @27,
Your Dean may have her/his metrics, but what do points 1 and 2 have to do with academic research? Also, what do any of the 3 points have to do with allocation of scores (and therefore dollars) in review?
Sure, your position at a med schl or univ or inst may depend on what extramural , and how the motivation for the respective applications (in the philosophy of the proposed research), and how the execution of the actual research, may differ in academia vs industry.

• SR says:

NIH funding of biomedical research has become just another entitlement program of the US government and like many other similar programs should be either scrapped or overhauled. The scientists who are successful at continuously securing their NIH research funding are in many ways like the welfare receipient who should not be on welfare; like Dick Cheney collecting his Social Security check; like Jim Bunning (R-KY) using his Medicare program.

• antipodean says:

SR
The difference is that NIH research helps people live longer and happier lives. Dick Cheney doesn't.
Please provide any example where government money is better (ethically) and more efficiently (cost-benefit) spent than health research funding.

• SR says:

"The difference is that NIH research helps people live longer and happier lives."
Surely, this is one of the myths that allow the NIH entitlement program to continue unabated. How could you eveluate its real impact on people life longevity and happiness without comparing it to life longevity and happiness under another approach of funding biomedical research? Have life longevity and people happiness did not improve or improved at a slower rate before NIH became the main source of dollars for biomedical rsearch? Do you have data that show that NIH research dollars have prolonged our life span more and made us happier than other research money sources did? Have you figured out the cost of prolonging life and increasing happiness with NIH dollar as compared to philantropist-donated dollar? To commercial research dollar? To university alumnus-donated dollar? I'll bet you the NIH dollar is the most expensive of them. If we could persuade all the American alumni idiots who give money to their alma matter sports programs to instead donate to their univeristy research programs, we could do away with the NIH, live longer and be happier. If our health insurance companies would fund biomedical research with the money they spend on lobbying in Washington, DC, we would live longer and happier than most nations.

• whimple says:

Please provide any example where government money is better (ethically) and more efficiently (cost-benefit) spent than health research funding.
Too easy: Medicare.

• antipodean says:

SR- So your answer to my requesting an exmaple is 'no'. I agree with you otherwise. Sadly none of the alternative options are really available in the land downunder. I get the feeling though that vast amounts of private donations would tend to be aggregated in foundations that would still rely on competitive peer review (see Buffet joining the Gates' as the major instance). I wouldn't want to be a basic scientist who just wants to dick around with cell biology in that environment. Private philanthropy seems to be tagged to certain diseases.
Whimple- probably a good call, since you mean US medicare and not Australian fee-for-service medicare.

As an NIH grantee, I do believe the system is flawed. The issue of multiple grants is an important one, and as many have pointed out the main rationale is simply to provide stability (though as a multiple grantee I can say the notion that one borrows from one grant to feed another is uninformed).
Is it wrong to want such stability? Of course not. Is it unethical to possess two grants? No. Is there a better way? I think so, but it would require some rethinking of the grant process. In my opinion, a HHMI-like system might be better than what we have. The structure might work something like this:
1) Researchers with proven productivity have a certain percentage of their effort offset by NIH, contingent on continued productivity - not a huge offset, but enough to offset a dedicated percentage of research effort. To get this, departments provide matching basic dollar for supplies and provide core lab space. This will reduce the pressure for multiple proposals;
2) R series grants adopt a real triage system instead of the useless one that exists now. Shorter applications are one thing, but make the first shot a preproposal where the basic idea is evaluated with regard to innovation and mission "fit" for the particular institute. Have a special part of study section where these proposals are either greenlighted for the full proposal or rejected (who would not prefer to be rejected at this stage rather than after sweating blood for months and blowing through significant dollars?);
If these two things were done I think things could improve over the current system. Those not in the HHMI-like pool would be on the same footing as new investigators, while productive investigators would still be able to write proposals, but these would be directed at more innovative questions with higher risk and more significant impact.
I'm not sure yet what this would do to pay lines, but let's say NIH supported 25% effort for these HHMI-like investigators. That's 4 investigators supported to do science versus one supported assuming 2 standard R series grants. Assuming 100k salary and 5 years of support that's 500k for HHMI-like support, versus about 3 million for the same 5 years (1.5 million total costs for 2 grants). Support for 4 brains instead of just one, talented grantmaker. Maybe this isn't the perfect solution - but wouldn't it be better to fund more investigators, provide some sort of career stability, remove the heartbreak of wasting time and funds chasing projects that are ultimately judged to be "flawed", and increase paylines for innovative projects?

• whimple says:

1) Researchers with proven productivity have a certain percentage of their effort offset by NIH, contingent on continued productivity - not a huge offset, but enough to offset a dedicated percentage of research effort. To get this, departments provide matching basic dollar for supplies and provide core lab space. This will reduce the pressure for multiple proposals;
Unfortunately, this idea is inconsistent with the academic business model for biomedical research positions. In this model, the direct costs for grants are a loss-leader for what the institutions really want, the indirects. They particularly want multiple grants because each grant beyond the first pays indirects money for costs that have already been covered by the first grant, and that therefore represent pure institutional profit.

• crystaldoc says:

Here is a suggestion for a post topic related to the original point of this one, that of "winning the war". Given recent changes in the NIH process (in particular only one chance for resubmission, and less information in the tea leaves of the new summary statements), when does it make sense to resubmit? When does it make more sense to change it up and put in a new submission? Any pragmatic advice or guidelines based on impact scores or percentiles? How often are A1's funded when the original submission was streamlined? Or at 40-50th percentile, or 30-40th percentile? Are these data available anywhere?

• DrugMonkey says:

Does this help?
http://scientopia.org/blogs/drugmonkey/2008/10/more-on-the-new-nih-policy-on-grant-application-revisions
Of course we have no idea how things will be changing as we full adopt the new single-revision limit and see how good they will be about rejecting thinly-veiled A2's coming in as new grants...

• crystaldoc says:

Thanks for the link. It doesn't really answer my questions, but I guess the data aren't there yet.

•  Support level Reader : $5.00 USD - monthly Supporter :$10.00 USD - monthly Sustainer : $25.00 USD - monthly Angel :$1,200.00 USD - yearly