I cannot tell you how comforting it is to know that no matter the depths and pedantry of my grant geekery, there is always a certain person to be found digging away furiously below me.
Archive for the 'Grant Review' category
I noticed something really weird and totally unnecessary.
When you are asked to review grants for the NIH you are frequently sent a Word document review template that has the Five Criteria nicely outlined and a box for you to start writing your bullet points. At the header to each section it sometimes includes some of the wording about how you are supposed to approach each criterion.
A recent template I received says under Investigator that one is to describe how the
..investigator’s experience and qualifications make him particularly well-suited for his roles in the project?
The NIH has recently issued the first round of guidance on inclusion of Sex as a Biological Variable in future NIH research grants. I am completely behind the spirit of the initiative but I have concerns about how well this is going to work in practice. I wrote a post in 2008 that detailed some of the reasons that have brought us to the situation where the Director of the NIH felt he had to coauthor an OpEd on this topic. I believe these issues are still present, will not be magically removed with new instructions to reviewers and need to be faced head-on if the NIH is to make any actual progress on ensuring SABV is considered appropriately going forward.
The post originally appeared December 2, 2008.
The title quote came from one of my early, and highly formative, experiences on study section. In the course of discussing a revised application it emerged that the prior version of the application had included a sex comparison. The PI had chosen to delete that part of the design in the revised application, prompting one of the experienced members of the panel to ask, quite rhetorically, "Why do they always drop the females?"
I was reminded of this when reading over Dr. Isis' excellent post [Update: Original Sb post lost, I think the repost can be found here] on the, shall we say less pernicious, ways that the course of science is slanted toward doing male-based research. Really, go read that post before you continue here, it is a fantastic description.
Thank you. That's the first time I've seen someone address the reasons behind ongoing gender disparities in health research. I still can't say as it thrills me (or you, obviously), but I understand a bit better now.
Did somebody ring?
As I pointed out explicitly at least once ([Update: Original 2007 post]), research funding has a huge role in what science actually gets conducted. Huge. In my book this means that if one feels that an area of science is being systematically overlooked or minimized, one might want to take a close look at the manner by which science is funded and the way by which science careers are sustained as potential avenues for systematic remedy.
There are a couple of ways in which the generalized problems with NIH grant review lead to the rhetorical comment with which I opened the post. One very common StockCritique of NIH grant review is that of an "over ambitious" research plan. As nicely detailed in Isis' post, the inclusion of a sex comparison doubles the groups right off the bat but even more to the point, it requires the inclusion of various hormonal cycling considerations. This can be as simple as requiring female subjects to be assessed at multiple points of an estrous cycle. It can be considerably more complicated, often requiring gonadectomy (at various developmental timepoints) and hormonal replacement (with dose-response designs, please) including all of the appropriate control groups / observations. Novel hormonal antagonists? Whoops, the model is not "well established" and needs to be "compared to the standard gonadectomy models",
Grant reviewers prefer simplicityKeep in mind, if you will, that there is always a more fundamental comparison or question at the root of the project, such as "does this drug compound ameliorate cocaine addiction?" So all the gender comparisons, designs and groups need to be multiplied against the cocaine addiction/treatment conditions. Suppose it is one of those cocaine models that requires a month or more of training per group? Who is going to run all those animals ? How many operant boxes / hours are available? and at what cost? Trust me, the grant proposal is going to take fire for "scope of the project".
Another StockCritique to blame is "feasibility". Two points here really. First is the question of Preliminary Data- of course if you have to run more experimental conditions to establish that you might have a meritorious hypothesis, you are less likely to do it with a fixed amount of pilot/startup/leftover money. Better to work on preliminary data for two or three distinct applications over just one if you have the funds. Second aspect has to do with a given PIs experience with the models in question. More opportunity to say "The PI has no idea what s/he is doing methodologically" if s/he has no prior background with the experimental conditions, which are almost always the female-related ones. As we all know, it matters little that the hormonal assays or gonadectomy or whatever procedures have been published endlessly if you don't have direct evidence that you can do it. Of course, more latitude is extended to the more-experienced investigator....but then s/he is less likely to jump into gender-comparisons in a sustained way in contrast to a newly minted PI.
Then there are the various things under grantspersonship. You have limited space in a given type of grant application. The more groups and comparisons, the more you have to squeeze in with respect to basic designs, methods and the interpretation/alternative approaches part. So of course you leave big windows for critiques of "hasn't fully considered...." and "it is not entirely clear how the PI will do..." and "how the hypothesis will be evaluated has not been sufficiently detailed...".
Although research funding plays a huge role in career success, it is only part of the puzzle. Another critical factor is what we consider to be "great" or "exciting" science in our respective fields.
The little people can fill in the details. This is basically the approach of GlamourMagz science. (This is a paraphrase of something the most successful GlamourMagz PI I know actually says.) Cool, fast and hot is not compatible with the metastasizing of experimental conditions that is an inevitable feature of gender-comparison science. Trouble is, this approach tends to trickle down in various guises. Lower (than GlamourMag) impact factor journals sometimes try to upgrade by becoming more NS-like (Hi, J Neuro!). Meticulous science and exacting experimental designs are only respected (if at all) after the fact. Late(r) in someone's career they start getting props on their grant reviews for this. Early? Well the person hasn't yet shown the necessity and profit for the exhaustive designs and instead they just look...unproductive. Like they haven't really shown anything yet.
As we all know splashy CNS pubs on the CV trump a sustained area of contribution in lower journals six ways to Sunday. This is not to say that nobody will appreciate the meticulous approach, they will. Just to say that high IF journal pubs will trump. Always.
So the smart young PI is going to stay away from those messy sex-differences studies. Everything tells her she should. If he does dip a toe, he's more likely to pay a nasty career price.
This is why NIH efforts to promote sex-comparison studies are necessary. Promoting special funding opportunities are the only way to tip the equation even slightly more favorable to the sex-differences side. The lure of the RFA is enough to persuade the experienced PI to write in the female groups. To convince the new PI that she might just risk it this one time.
My suspicion is that it is not enough. Beyond the simple need to take a stepwise approach to the science as detailed by Isis, the career and funding pressures are irresistible forces.
We spend a fair amount of time talking about grant strategy on this blog. Presumably, this is a reflection of an internal process many of us go through trying to decide how to distribute our grant writing effort so as to maximize our chances of getting funded. After all we have better things to do than to write grants.
So we scrutinize success rates for various ICs, various mechanisms, FOAs, etc as best we are able. We flog RePORTER for evidence of which study sections will be most sympathetic to our proposals and how to cast our applications so as to be attractive. We worry about how to construct our Biosketch and who to include as consultants or collaborators. We obsess over how much preliminary data is enough (and too much*).
This is all well and good and maybe, maybe....perhaps....it helps.
But at some level, you have to follow your gut, too. Even when the odds seem overwhelmingly bad, there are going to be times when dang it, you just feel like this is the right thing to do.
Submitting an R01 on very thin preliminary data because it just doesn't work as an R21 perhaps.
Proposing an R03 scope project even if the relevant study section has only one** of them funded on the RePORTER books.
Submitting your proposal when the PO who will likely be handling it has already told you she hates your Aims***.
Revising that application that has been triaged twice**** and sending it back in as a A2asA0 proposal.
I would just advise that you take a balanced approach. Make your riskier attempts, sure, but balance those with some less risky applications too.
I view it as....experimenting.
*Just got a question about presenting too much preliminary data the other day.
**of course you want to make sure there is not a structural issue at work, such as the section stopped reviewing this mechanism two years ago.
***1-2%ile scores have a way of softening the stony cold heart of a Program Officer. Within-payline skips are very, very rare beasts.
****one of my least strategic behaviors may be in revising grants that have been triaged. Not sure I've ever had one funded after initial triage and yet I persist. Less so now than I used to but.....I have a tendency. Hard headed and stupid, maybe.
It is one of the most perplexing things of my career and I still don't completely understand why this is the case. But it is important for PIs, especially those who have not yet experienced study section, to understand a simple fact of life.
The NIH Program Officers do not completely understand what contributes to the review and scoring of your grant application.
My examples are legion and I have mentioned some of them in prior blog posts over the years.
The advice from a PO that PIs (such as myself) just needed to "write better grants" when I was already through a stint on study section and had read many, many crappy and yet funded grants from more established investigators.
The observation that transitioning investigators "shouldn't take that job" because it was soft money and K grants were figuring heavily in the person's transition/launch plans.
Apparently honest wonder that reviewers do not read their precious Program Announcements and automatically award excellent scores to applications just because they align with the goals of the PA.
Ignorance of the revision queuing that was particularly endemic during the early part of my career (and pretend? ignorance that limiting applications to one revision round made no functional difference in this).
The "sudden discovery" that all of the New Investigator grants during the checkbox era were going to well-established investigators who simply happened not to have NIH funding before, instead of boosting the young / recently appointed investigators.
An almost comically naive belief that study section outcome for grants really is an unbiased reflection of grant merit.
I could go on.
The reason this is so perplexing to me is that this is their job. POs [eta: used to] sit in on study section meetings or listen in on the phone. At least three times a year but probably more often given various special emphasis panels and the assignment of grants that might be reviewed in any of several study sections. They even take notes and are supposed to give feedback to the applicant with respect to the tenor of the discussion. They read any and all summary statements that they care to. They read (or can read) a nearly dizzying array of successful and unsuccessful applications.
And yet they mostly seem so ignorant of dynamics that were apparent to me after one, two or at the most three study section meetings.
It is weird.
The takeaway message for less NIH-experienced applicants is that the PO doesn't know everything. I'm not saying they are never helpful....they are. Occasionally very helpful. Difference between funded and not-funded helpful. So I fully endorse the usual advice to talk to your POs early and often.
Do not take the PO word for gospel, however. Take it under advisement and integrate it with all of your other sources of information to try to decide how to advance your funding strategy.
from this Advice Corner on modular budgeting:
As you design your research proposal, tabulate a rough cost estimate. If you are above but near the $250,000 annual direct cost threshold, consider ways to lessen your expenses. Maybe you have a low-priority Specific Aim that can be dropped or a piece of equipment you could rent rather than buy new.
I have been flailing around, of and on for a few months, trying to write my Biosketch into the new format [Word doc Instructions and Sample].
I am not someone who likes to prance around bragging about "discoveries" and unique contributions and how
my lab's work is I am so awesomely unique because, let's face it, I don't do that kind of work. I am much more of a work-a-day type of scientist who likes to demonstrate stuff that has never been shown before. I like to answer what are seemingly obvious questions for which there should be lots of literature but then it turns out that there is not. I like to work on what interests me about the world and I am mostly uninterested in what some gang of screechy monkey GlamourHumpers think is the latest and greatest.
This is getting in the way of my ability to:
Briefly describe up to five of your most significant contributions to science. For each contribution, indicate the historical background that frames the scientific problem; the central finding(s); the influence of the finding(s) on the progress of science or the application of those finding(s) to health or technology; and your specific role in the described work.
Now interestingly, it was someone who works in a way most unlike the way I do that showed me the light. Actually, he gave me the courage to think about ignoring this supposed charge in the sample / instruction document. This person recommended just writing a brief sentence or two about the area of work without trying to contextualize the importance or significance of the "contribution". I believe I actually saw one of the five permitted subheadings on his version that was more or less "And here's some other stuff we work on that wasn't easily categorized with the rest of it."
I am at least starting from this minimalist standpoint. I don't know if I will have the courage to actually submit it like this, but I'm leaning towards doing so.
I have been hearing from quite a number of you that you are struggling with creating this new version of the NIH Biosketch. So I thought I'd open it up to comment and observation. Anyone have any brilliant solutions / approaches to recommend?
One of the things that has been bothering me most about this is that it takes the focus off of your work that is specific to the particular application in question. In the most recent version of the Biosketch, you selected 15 pubs that were most directly relevant to the topic at hand. These may not be your "most significant contributions" but they are the ones that are most significant for the newly proposed studies.
If one is now to list "your most significant contributions", well, presumably some of these may not have much to do with the current application. And if you take the five sections seriously, it is hard to parse the subset of your work that is relevant to one focal R01 sized project into multiple headings and still show now those particular aspects are a significant contribution.
I still think it is ridiculous that they didn't simply make this an optional way to do the Biosketch so as to accommodate those people that needed to talk about non-published scholarly works.
I just can't understand what is valuable about showing that a 1%ile difference in voted score leads to 2% difference in total citations of papers attributed to that grant award. All discussions of whether NIH peer review is working or broken center on the supposed failure to fund meritorious grants and the alleged funding of non-meritorious grants.
Please show me one PI that is upset that her 4%ile funded grant really deserved a 2%ile and that shows that peer review is horribly broken.
The real issue, how a grant overlooked by the system would fare *were it to be funded* is actually addressed to some extent by the graph on citations to clearly outlying grants funded by exception.
This is cast as Program rescuing those rare exception brilliant proposal. But again, how do we know the ones that Program fails to rescue wouldn't have performed well?
One of the most perplexing thing I have learned about the review of 5 year R01 NIH grant proposals is a species of reviewer that is obsessed with Future Directions.
It was a revelation to me in one of my first few study section meetings that some reviewers really want to see extensive comment on where the project might be heading after the completion of 5 years of work. As in, a whole subheaded paragraph at the end of the Research Plan. This is insane to me.
For the most part, we all recognize that ongoing results in your own lab and in the field at large are going to dictate what is important to pursue five years from now. So speculation about what is coming next is silly.
And especially when I was a relatively inexperienced grant writer who had been getting beat up for "over ambitious" plans contained in a single 5 year plan, well.... I was amazed that people wanted to see even more in a speculative, hand wavey paragraph.
Consequently, I struggle with this. But I have tried to include something about Future Directions in my proposals. Yes, even now that we have only 12 precious pages to describe the actual plans for the current proposal.
I have recently seen a summary statement that describes insufficient attention paid to the Future Directions as the "primary weakness" of the proposal. I cannot even imagine what this reviewer was thinking. How can this be the primary weakness? Unless there is literally nothing else to complain about. And we know that never happens.
Since we're discussing the amount of PI salary that should be rightfully paid by the NIH versus a local University lately, I have a grant review scenario to mention.
It is not uncommon to see R01 proposals come in from PIs who say that they will charge the grant for "three months summer salary". As we know, this is likely a scenario where the Professor in question has a 9 month salary from his or her University and is permitted to supplement that with up to three months of salary from extramural support funds.
Let us assume we're talking a normal research plan for an R01 that involves research effort pretty much around the calendar year. We're not talking about something that requires focal field work for a few summer months and then can subside into a much lower level of activity for the rest of the year.
On first glance the reviewer can only assume that the PI's remaining 9 months are being paid by the University to DO SOMETHING. Despite comment from Neuro-conservative about situations that seem very strange and unique, my experience is that Universities put some expectation of non-research activity on that 9 month of salary*.
Unless the PI has specified an expectation of research in their official job description, the reviewer can only assume that the effort on the grant will only be available during the summer.
Such a proposal should be met with the utmost skepticism since the conduct of the research requires ongoing supervision of the staff**, at the very least. Right?
So the grantsmithing advice part of this post is that if you are in this sort of situation, be sure to make very clear what your University explicitly expects in terms of your nine-month-hard-salary time.
From the perspective of our ongoing discussion, how is this all supposed to work? What true amount of brain-second-cycles are available to the project at any given time throughout the year?
Teaching duties tend to be rather inelastic and research duties tend to be highly elastic. I can always put off working on a paper or data analysis for another day. I can pick and choose when to work on a poster or oral presentation. I can't really put off lecture at 8am just because I have some exciting results in the laboratory that I want to write up right now. Grading may be a teeensy bit more flexible but there are clear deadlines...unlike paper submissions and most unlike designing new research projects and/or collaborations. Also very unlike meeting with your grad students and postdocs about various things.
I would suggest that under the 50/50 time scenario proposed by Neuro-conservative, one of the two task demands is going to receive short-shrift in a large number of cases. This will mostly be determined by what type of University the PI is employed within. Those that lean towards research? Well, we all know about how the tenure stool really only has one leg. Research. Conversely, there are very high teaching load institutions that inevitably push research toward the background during the active instructional school year.
In these situations either the NIH is being fleeced to support undergraduate instruction or the undergraduate instruction support system (State general funds and tuition, the latter includes scholarships and the like btw, another interested party) is being fleeced to pay for the NIH's business.
The only ethical situation is when there is perfect balance between the expectations of the respective sources of financial support and the PIs actual distribution of work.
I do wonder how many NIH PIs that have nine month salary support actually achieve the appropriate balance of brain effort devoted to their respective tasks. I bet not many.
*I would like to hear some specific language from people's job descriptions that specify that their hard money effort is supposed to be devoted X amount to research, btw. I know these do exist. How commonly?
**Naturally these sorts of proposals are often coupled with 12 mo of full time effort from trainees or techs which supports the notion that the project is not limited to the summer months.