I want to get to the point in my career where other people write the majority of the papers & I just revise them & make sure they are right
— Sciencegurl (@sciencegurlz0) September 29, 2016
I want to get to the point in my career where other people write the majority of the papers & I just revise them & make sure they are right
— Sciencegurl (@sciencegurlz0) September 29, 2016
The NIH FOAs come in many flavors of specificity. Some, usually Program Announcements, are very broad and appear to permit a wide range of applications to fit within them. My favorite example of this is NIDA's "Neuroscience Research on Drug Abuse" PA.
They also come in highly specific varieties, generally as RFAs.
The targeted FOA is my topic for the day because they can be frustrating in the extreme. No matter how finely described for the type, these FOA are inevitably too broad to let each and every interested PI know exactly how to craft her application. Or, more importantly, whether to bother. There is always a scientific contact, a Program Officer, listed so the first thing to do is email or call this person. This can also be frustrating. Sometimes one gets great advice, sometimes it is perplexing.
As always, I can only offer up the way I look at these things.
As an applicant PI facing an FOA that seems vaguely of interest to me, I have several variables that are at play. First, despite the fact that Program may have written the FOA in a particular way, this doesn't mean that they really know what they want. The FOA language may be a committee result or it may just not have been thought that a highly specific type of proposal was necessary to satisfy what goals and motivations existed.
Second, even if they do know what they want in Programville, peer review is always the primary driver. If you can't escape triage it is highly unlikely that Program will fund your application, even if it fits their intent to a T. So as the applicant PI, I have to consider how peers are likely to interpret the FOA and how they are likely to apply it to my application. It is not impossible that the advice and perspective given to the prospective PI by the contact PO flies rather severely in the face of that PIs best estimate of what is likely to occur during peer review. This leaves a conundrum.
How to best navigate peer review and also serve up a proposal that is attractive to Program, in case they are looking to reach down out of the order of review for a proposal that matches what they want.
Finally, as I mention now and again there is an advocacy role for the PI when applying for NIH funding. It is part and parcel of the job of the PI to tell Program what they should be funding. By, of course, serving up such a brilliantly argued application that they see that your take on their FOA is the best take. Even if this may not have been what was their intent in the first place. This also, btw, applies to the study section members. Your job is in part to convince them, not to meet whatever their preconceptions or reading of the FOA might be.
Somehow, the PI has to stew all of these considerations together and come up with a plan for the best possible proposal. Unfortunately, you can miss the mark. Not because your application is necessarily weak or your work doesn't fit the FOA in some objective sense. Merely because you have decided to make choices, gambles and interpretations that have led you in a particular direction, which may very well be the "wrong" direction.
Most severely, you might be rejected without review. This can happen. If you do not meet the PO's idea of being within the necessary scope of what they would ever plan to fund, no matter the score, you could have your application prevented from being routed to the study section.
Alternately, you might get triaged by a panel that just doesn't see it your way. That wonders if you, the idiot PI, was reading the same FOA that they are. It happens.
Finally, you might get a good score and Program may decide to skip over it for lack of responsiveness to their intent. Or you may be in the grey zone and fail to get a pickup because other grants scoring below yours are deemed closer to what they want to fund.
My point for today is that I think this is necessary error in the system. It is not evidence of a wholesale problem with the NIH FOA approach if you shoot wide to the left. If you fail to really understand the intent of the FOA as written. Or if you come away from your initial chat with the PO with a misguided understanding. Or even if you run into the buzzsaw of a review panel that rebels against the FOA.
Personally, I think you just have to take your chances. Arrive at your best understanding of what the FOA intends and how the POs are going to interpret various proposals. Sure. And craft your application accordingly. But you have to realize that you may be missing the point entirely. You may fail to convince anyone of your brilliant take on the FOA's stated goals. This doesn't mean the system is broken.
So take your shots. Offer up your best interpretation on how to address the goals. And then bear down and find the next FOA and work on that. In case your first shot sails over the crossbar.
It always fascinates me how fairly wide-flung experiences with NIH funding coalesce around the same issue sometimes. This particular post was motivated by no less than three situations being brought to my attention in the past week. Different ICs, different FOA, different mechanisms and vastly different topics and IC intentions. But to me, the answers are the same.
WOW. This comment from dsks absolutely nails it to the wall.
The NIH is supposed to be taking on a major component of the risk in scientific research by playing the role of investor; instead, it seems to operates more as a consumer, treating projects like products to be purchased only when complete and deemed sufficiently impactful. In addition to implicitly encouraging investigators to flout rules like that above, this shifts most of the risk onto the shoulders of investigator, who must use her existing funds to spin the roulette wheel and hope that the projects her lab is engaged in will be both successful and yield interesting answers. If she strikes it lucky, there’s a chances of recouping the cost from the NIH. However, if the project is unsuccessful, or successful but produces one of the many not-so-pizzazz-wow answers, the PI’s investment is lost, and at a potentially considerable cost to her career if she’s a new investigator.
Of course one might lessen the charge slightly by observing that it is really the University that is somehow investing in the exploratory work that may eventually become of interest to the buyer. Whether the University then shifts the risk onto the lowly PI is a huge concern, but not inevitable. They could continue to provide seed money, salary, etc to a professor who does not manage to write a funded grant application.
Nevertheless, this is absolutely the right way to look at the ever growing obligation for highly specific Preliminary Data to support any successful grant application. Also the way to look at a study section culture that is motivated in large part by perceived "riskiness" (which underlies a large part of the failure to reward untried investigators from unknown Universities compared with established PIs from coastal elite institutions).
NIH isn't investing in risky science. It is purchasing science once it looks like most of the real risk has been avoided.
I have never seen this so clearly, so thanks to dsks for expressing it.
This was originally posted 16 September, 2014.
Throughout the prior two Council rounds of the year, they have to necessarily be a bit conservative. After all, they don't know in the first Round if maybe they will have a whole bunch of stellar scores come in during the third Round. Some one-off funding opportunities are perhaps schedule for consideration only during the final Round. Etc.
Also, the amount of funding requested for each grant varies. So maybe they have a bunch of high scoring proposals that are all very inexpensive? Or maybe they have many in the early rounds of the year that are unusually large?
This means that come September, the ICs are sometimes sitting on unexpended funds and need to start picking up proposals that weren't originally slated to fund. Maybe it is a supplement, maybe it is a small mechanism like a R03 or R21. Maybe they will offer you 2 years of funding of an R01 proposed for 5. Maybe they will offer you half the budget you requested. Maybe they have all of a sudden discovered a brand new funding priority and the quickest way to hit the ground running is to pick something up with end-of-year funds.
Now obviously, you cannot game this out for yourself. There is no way to rush in a proposal at the end of the year (save for certain administrative supplements). There is no way for you to predict what your favorite IC is going to be doing in Sep- maybe they have exquisite prediction and always play it straight up by priority score right to the end, sticking within the lines of the Council rounds. And of course, you cannot assume lobbying some lowly PO for a pickup is going to work out for you.
There is one thing you can do, Dear Reader.
It is pretty simple. You cannot receive one of these end-of-year unexpected grant awards unless you have a proposal on the books and in play. That means, mostly, a score and not a triage outcome. It means, in a practical sense, that you had better have your JIT information all squared away because this can affect things. It means, so I hear, that this is FINALLY the time when your IC will quite explicitly look at overhead rates to see about total costs and screw over those evil bastiges at high overhead Universities that you keep ranting about on the internet. You can make sure you have not just an R01 hanging around but also a smaller mech like an R03 or R21.
It happens*. I know lots and lots of people who have received end-of-the-FY largesse that they were not expecting. Received this type of benefit myself. It happens because you have *tried* earlier in the year to get funding and have managed to get something sitting on the books, just waiting for the spotlight of attention to fall upon you.
So keep that ball in play, my friends. Keep submitting credible apps. Keep your Commons list topped off with scored apps.
*As we move into October, you can peruse SILK and RePORTER to see which proposals have a start date of Sep 30. Those are the end-of-year pickups.
h/t: some Reader who may or may not choose to self-identify 🙂
— Drug Monkey (@drugmonkeyblog) September 20, 2016
The R21 Mechanism is called the Exploratory/Developmental mechanism. Says so right in the title.
NIH Exploratory/Developmental Research Grant Program ( Parent R21)
In the real world of NIH grant review, however, the "Developmental" part is entirely ignored in most cases. If you want a more accurate title, it should be:
NIH High Risk / High Reward Research Grant Program ( Parent R21)
This is what reviwers favor in my experiences sitting on panels and occasionally submitting an R21 app. Mine are usually more along the lines of developing a new line of research that I think is important rather than being truly "high risk/high reward".
And, as we all know, the R01 application (5 years, full modular at $250K per annum direct costs if you please) absolutely requires a ton of highly specific Preliminary Data.
So how are you supposed to Develop an idea into this highly specific Preliminary Data? Well, there's the R21, right? Says right in the title that it is Developmental.
But....it doesn't work in practice.
So the R01 is an alternative. After all it is the most flexible mechanism. You could submit an R01 for $25K direct costs for one year. You'd be nuts, but you could. Actually you could submit an R03 or R21 for one $25K module too, but with the R01 you would then have the option to put in a competitive renewal to continue the project along.
The only thing stopping this from being a thing is the study section culture that won't accept it. Me, I see a lot of advantages to using shorter (and likely smaller) R01 proposals to develop a new line of work. It is less risky than a 5 year R01, for those that focus on risk/$. It has an obvious path of continuation as a genuinely Developmental attempt. It is more flexible in scope and timing- perhaps what you really need is $100K per year for 3 years (like the old R21) for your particular type of research or job type. It doesn't come laden with quite the same "high risk, high reward" approach to R21 review that biases for flash over solid workmanlike substance.
The only way I see this working is to try it. Repeatedly. Settle in for the long haul. Craft your Specific Aims opening to explain why you are taking this approach. Take the Future Directions blurb and make it really sparkle. Think about using milestones and decision points to convince the reviewers you will cut this off at the end if it isn't turning out to be that productive. Show why your particular science, job category, institute or resources match up to this idea.
Or you could always just shout aimlessly into the ether of social media.
As I've noted on these pages before, my sole detectable talent for this career is the ability to take a punch.
There are a lot of punches in academic science. A lot of rejection and the congratulations for a job well done are few and far between. Nobody ever tells you that you are doing enough.
"Looking good, Assistant Professor! Just keep this up, maybe even chill a little now and then, and tenure will be no problem!" - said no Chair ever.
My concern is that resilience in the face of constant rejection, belittling and unkind comparisons of your science to the true rock stars in a Lake Wobegon approach can have a selection effect. Only certain personality types can stand this.
I happen to have one of these personality types but it is not something of any particular credit. I was born and/or made this way by my upbringing. I cannot say anyone helped to train me in this way as an academic scientist*.
So I am at a complete loss as to how to help my trainees with this.
Have you any insights Dear Reader? From your own development as a scientist or as a supervisor of other scientists?
Related Reading: Tales of postdocs past: what did I learn?
*well maybe indirectly. And not in a way I care to extend to any trainee of mine thankyewveerymuch.
Commenter jmz4 made a fascinating comment on a prior post:
It is not the journals responsibility to mete out retractions as a form of punishment(&). Only someone that buys into papers as career accolades would accept that. The journal is there to disseminate accurate scientific information. If the journal has evidence that, despite the complaint, this information is accurate,(%) then it *absolutely* should take that into account when deciding to keep a paper out there.
(&) Otherwise we would retract papers from leches and embezzlers. We don't.
That prior post was focused on data fraud, but this set of comments suggest something a little broader.
I.e., that fact are facts and it doesn't matter how we have obtained them.
This, of course, brings up the little nagging matter of the treatment of research subjects. As you are mostly aware, Dear Readers, the conduct of biomedical experimentation that involves human or nonhuman animal subjects requires an approval process. Boards of people external to the immediate interests of the laboratory in question must review research protocols in advance and approve the use of human (Institutional Review Board; IRB) or nonhuman animal (Institutional Animal Care and Use Committee; IACUC) subjects.
The vast majority (ok, all) journals of my acquaintance require authors to assert that they have indeed conducted their research under approvals provided by IRB or IACUC as appropriate.
So what happens when and if it is determined that experiments have been conducted outside of IRB or IACUC approval?
The position expressed by jmz4 is that it shouldn't matter. The facts are as they are, the data have been collected so too bad, nothing to be done here. We may tut-tut quietly but the papers should not be retracted.
I say this is outrageous and nonsense. Of course we should apply punitive sanctions, including retracting the paper in question, if anyone is caught trying to publish research that was not collected under proper ethical approvals and procedures.
In making this decision, the evidence for whether the conclusions are likely to be correct or incorrect plays no role. The journal should retract the paper to remove the rewards and motivations for operating outside of the rules. Absolutely. Publishers are an integral part of the integrity of science.
The idea that journals are just there to report the facts as they become known is dangerous and wrong.
Via the usual relentless trolling of YHN from Comrade PhysioProffe, a note on a fraud investigation from the editors of Cell.
We, the editors of Cell, published an Editorial Expression of Concern (http://dx.doi.org/10.1016/j.cell.2016.03.038) earlier this year regarding issues raised about Figures 2F, 2H, and 3G of the above article.
two labs have now completed their experiments, and their data largely confirm the central conclusions drawn from the original figures. Although this does not resolve the conflicting claims, based on the information available to us at this time, we will take no further action. We would like to thank the independent labs who invested significant time and effort in ensuring the accuracy of the scientific record.
Bad Cell. BAD!
We see this all the time, although usually it is the original authors aided and abetted by the journal Editors, rather than the journal itself, making this claim. No matter if it is a claim to replace an "erroneous placeholder figure", or a full on retraction by the "good" authors for fraud perpetrated by some [nonWestern] postdoc who cannot be located anymore, we see an attempt to maintain the priority claim. "Several labs have replicated and extended our work", is how it goes if the paper is an old one. "We've replicated the bad [nonWestern, can't be located] postdoc's work" if the paper is newer.
I say "aided and abetted" because the Editors have to approve the language of the authors' erratum, corrigendum or retraction notice. They permit this. Why? Well obviously because just as the authors need to protect their reputation, so does the journal.
So everyone plays this game that somehow proving the original claims were correct, reliable or true means that the original offense is lesser. And that the remaining "good" authors and the journal should get credited for publishing it.
I say this is wrong. If the data were faked, the finding was not supported. Or not supported to the degree that it would have been accepted for publication in that particular journal. And therefore there should be no credit for the work.
We all know that there is a priority and Impact Factor chase in certain types of science. Anything published in Cell quite obviously qualifies for the most cutthroat aspects of this particular game. Authors and editors alike are complicit.
If something is perceived to be hott stuff, both parties are motivated to get the finding published. First. Before those other guys. So...corners are occasionally cut. Authors and Editors both do this.
Rewarding the high risk behavior that leads to such retractions and frauds is not a good thing. While I think punishing proven fraudsters is important, it does not by any means go far enough.
We need to remove the positive reward environment. Look at it this way. If you intentionally fake data, or more likely subsets of the data, to get past that final review hurdle into a Cell acceptance, you are probably not very likely to get caught. If you are detected, it will often take years for this to come to light, particularly when it comes to a proven-beyond-doubt standard. In the mean time, you have enjoyed all the career benefits of that Glamour paper. Job offers for the postdocs. Grant awards for the PIs. Promotions. High $$ recruitment or retention packages. And generated even more Glam studies. So in the somewhat unlikely case of being busted for the original fake many of the beneficiaries, save the poor sucker nonWestern postdoc (who cannot be located), are able to defend and evade based on stature.
This gentleman's agreement to view faked results that happen to replicate as no-harm, no-foul is part of this process. It encourages faking and fraud. It should be stopped.
One more interesting part of this case. It was actually raised by the self-confessed cheater!
Yao-Yun Liang of the above article informed us, the Cell editors, that he manipulated the experiments to achieve predetermined results in Figures 2F, 2H, and 3G. The corresponding author of the paper, Xin-Hua Feng, has refuted the validity of Liang’s claims, citing concerns about Liang’s motives and credibility. In a continuing process, we have consulted with the authors, the corresponding author’s institution, and the Committee on Publication Ethics (COPE), and we have evaluated the available original data. The Committee on Scientific Integrity at the corresponding author’s institution, Baylor College of Medicine, conducted a preliminary inquiry that was inconclusive and recommended no further action. As the institution’s inquiry was inconclusive and it has been difficult to adjudicate the conflicting claims, we have provided the corresponding author an opportunity to arrange repetition of the experiments in question by independent labs.
Kind of reminiscent of the recent case where the trainee and lab head had counter claims against each other for a bit of fraudulent data, eh? I wonder if Liang was making a similar assertion to that of Dr. Cohn in the Mt. Sinai case, i.e., that the lab head created a culture of fraud or directly requested the fake? In the latter case, it looked like they probably only came down on the PI because of a smoking-gun email and the perceived credibility of the witnesses. Remember that ORI refused to take up the case so there probably was very little hard evidence on which to proceed. I'd bet that an inability to get beyond "he-said/he-said" is probably at the root of Baylor's "inconclusive" preliminary inquiry result for this Liang/Feng dispute.
NPR had a good segment on this today: The Difficulty Of Enforcing Laws Against Driving While High. Definitely well worth a listen.
I had a few reactions in a comment that ended up being post-length, so here you go.
The major discussion of the segment was two-fold and I think illustrates where policy based on the science can be helpful, even if only to point to what we need to know but do not at present.
The first point was that THC hangs around in the body for a very long time post-consumption, particularly in comparison with alcohol. Someone who is a long term chronic user can have blood THC levels that are...appreciable (no matter the particular threshold for presumed impairment, this is relevant). Some of the best data on this are from the laboratory of Marilyn Huestis when she was, gasp, an intramural investigator at NIDA! There are some attempts in the Huestis work to compare THC and metabolite ratios to determine recency of consumption-that's a good direction. IMO.
The second argument was about behavioral tolerance. One of the scientist interviewed was quoted along the lines of saying the relationship between blood levels, repetitive use and actual impairment was more linear for alcohol than for THC. Pretty much. There is some evidence for substantial behavioral tolerance, meaning even when acutely intoxicated, the chronic user may have relatively preserved performance versus the noob. There's a laboratory study here that makes the point fairly succinctly, even if the behavior itself isn't that complex. As a counterpoint, this recent human study fails to confirm behavioral tolerance in an acute dosing study (see Fig 4A for baseline THC by frequency of use, btw). As that NPR piece noted, it would be very valuable to get some rapid field screen for THC/driving - relevant impairment on a tablet.
Five states have recreational marijuana legalization on the ballot this fall, if I heard correctly.
I feel as though we should probably talk about this over the next couple of months.
As most of you are aware, these follow successful recreational legalization initiatives in Washington (2012), Colorado (2012), Oregon (2014), Alaska (2014) and the District of Columbia (2014).