Newly funded NIH PIs

Jun 18 2016 Published by under Ponder

It would be fascinating if NIH did audits to search for who gets funded with their first grant by pedigree and productivity measures.

I'd want to see categorization of number of first and last author pubs. Of course. Some sort of measure of productivity in the time since being appointed to the independent investigator title. Mediated by JIF.

Then pedigree by the grant wealth and productivity of the pre-independence mentors.

I wonder if you can get away with crap productivity of you are tied into the network. And if you can overcome your Outsider status by generating a ton of pubs.

I wonder how likely a newb is to be funded as the years elapse from the time of first appointment without senior author publications.

42 responses so far

  • qaz says:

    It would also be interesting to see how this has changed over the years. There's probably not enough data to measure on a yearly basis, but I bet one could calculate these parameters on a decade or semi-decade basis.

  • I would bet that this is highly study section dependent. My study section takes great pains to try to take insider/outsider/pedigree-type factors out of the equation when reviewing grants, and we have a pretty explicit culture of guiding new reviewers away from the "he's from a great lab" or "he's fantastic, I know he'll do great things" type of nonsense.

  • drugmonkey says:

    See? It would be fascinating to know if claims like PPs that '"we" (IC, subfield, Study section, etc) are awesome and unbiased unlike those guys over there' are valid.

  • old grantee says:

    Yes, indeed!.

  • Comradde PhysioProffe says:

    Note that I didn't claim we are unbiased. I claimed that we are conscious of this particular bias, and we try to limit its impact on scoring. But way to twist words for inflammatory effect, as per the usual for DM.

  • qaz says:

    But maybe if it turns out that CPP's study section is less biased, then other study sections could start doing what they do. It would almost be like a science experiment...

  • old grantee says:

    "But way to twist words for inflammatory effect, as per the usual for DM."

    PP, your "somehow" hypersensitivity does little but supports DM's questioning. IMHO

  • Pinko Punko says:

    I suspect there could be some of this, but hard to control for. I know of examples of top pedigree types that are very anemic on productivity but science is high quality, and grants are works of art. This is not arguing that they should get a pass, but I think sometimes there will be window dressing there if the proposal is really good, and there is a stereotype low publishing perfectionist that also writes extremely well.

  • damit says:

    Anecdotal, and only the people in my sphere...
    Getting funded right out of the gate may not be the best thing.
    Those of us who are forced to get our stuff right are blessed sometimes.
    Just my experience.

  • Luminiferous Aether says:

    Even if not by pedigree, I would be interested in seeing this just as a measure of first and last author pubs. As a brand new PI, I'm starting to plan my first R01 submission and I anticipate not having a last author pub for at least another 18-24 months. In that case, how would my study sections of interest react? As others have said above, I think that this would be very SS dependent, but in general I would like to know of people's experiences as new PIs going in for R01's without any last author pubs.

    P.S: I know that the general recommendation on this blog is to submit an R01 asap to get in the review/resubmit pipeline, but do folks frequently receive their first R01s without a single last author pub?

  • Ola says:

    I took the second route you mention - a shit ton of pubs, to compensate (?) for lack of pedigree. I think probably because of that, I'm more likely to call BS on glam-lab alumnae (GLA) at study section, especially if they've gotten a position and have yet to deliver any publications as senior author.

    Of course, it's always fun to watch GLA struggle when they realize it ain't so easy to survive and get your papers past the editors, out there in the big bad real world.

  • AcademicLurker says:

    @Luminiferous Aether:

    I think "Submit soon, submit often" is still good advice. I put in my first R01 proposal at the end of my first year as a PI with no last author publications. I think part of the reason* it got funded as an A1 is that I had both responded to reviewer criticisms and got some last author papers out.

    *OK, more likely the study section membership changed and I just happened to get a reviewer who really liked what I was doing. Still, I'd like to think that the publications counted for something.

  • Draino says:

    My newly funded story: During my second and third year of independence I tried submitting without any Sr. author pubs. Got mediocre scores that didn't budge between A0 and A1. Criticisms were mostly about the approach and not having previously published on the disease the grant was focused on (mistake: grant was too disease specific for my profile). During year four I focused on publications and finally got my first two papers in year five, in respectable journals but feeling long overdue after 12-18 months of revise/resubmit. Ugh. But I timed that with submitting a new R01 jumping off findings in one of those pubs. Different SS than before, with a few people who know me. Got it funded in the first cycle this year.

    I think it definitely helped to get those publications. At least as important was changing SS and improved grant writing. Now I've gone back to the SS that rejected my first applications, with a new grant on the original disease in question. I hope it helps that one of my pubs last year was on the disease. But it feels like a crap shoot. I'm already considering another SS that could review a retooled version, in case those guys still don't want to accept me in the club.

  • jmz4 says:

    "Of course, it's always fun to watch GLA struggle when they realize it ain't so easy to survive and get your papers past the editors, out there in the big bad real world."
    That does make me nervous. 1) I don't know how publications work at non-CNS journals, which, I admit, is ridiculous. 2) I hope that when I start my lab and put out normal papers, the SS won't perceive that as my work going down in quality, or evidence that my first papers weren't "worthy".
    A separate question is what to do with all the data left over that hasn't been deemed worthy of publication by my boss? Hold on to it until I start my own lab?

  • Another Assistant Prof says:

    I think that one of my biggest mistakes as a new assistant professor was to publish immediately whenever we made a cool discovery. The study section always comments effusively on my outstanding productivity, but then says that my aims are incremental because we already have publications on the topic. I have enough pubs on my CV to look productive for a while (and to get tenure), so I'm not publishing any new data until I get a grant on the topic. This strategy is awful for the advancement of my field, and for the careers of my trainees, but I'm learning that this is how the game is played.

  • Luminiferous Aether says:

    @AL - I suppose that would be an ideal scenario for me seeing as that I too plan to have my first R01 proposal out just before the first year mark. I should aim to publish a senior author pub in the spring of 2018.

  • Dave says:

    Anecdotally amongst some peers of mine chatting at conferences and through NIH Reporter stalking, it does seem like the first last-author pub makes life easier in review, especially if it's a big..ish one. Lots of examples of paper, and then....boom....grant. But of course, correlation does not equal causality here. They may have had time to refine aims, improve preliminary data, become a better grant writer. It will be hard to tease out the variables here.

    The advice I always get from people just a few years ahead of me in my field submitting to the same SS is to WAIT until your grant is strong before going in. Sort of goes against a lot of GENERAL advice here, but it's worked for them I suppose.

  • Dave says:

    I think that one of my biggest mistakes as a new assistant professor was to publish immediately whenever we made a cool discovery

    Time for a bit of saucisson de lyon slicing, no???

  • Dave says:

    I don't know how publications work at non-CNS journals, which, I admit, is ridiculous

    TROLLLLLL

    I hope that when I start my lab and put out normal papers

    TROLLLLLLLLLLLLLLLLLLLLLLLLLLLLL

  • Luminiferous Aether says:

    "The advice I always get from people just a few years ahead of me in my field submitting to the same SS is to WAIT until your grant is strong before going in."

    An important consideration while figuring out how to balance a good proposal with getting into the pipeline asap. Wouldn't want the first proposal to be labeled as a piece of crap and run the risk of being considered "that guy"....

  • drugmonkey says:

    LA-

    There is a grace period. People aren't expecting a paper in the first year. I advise submitting R01 right out of the gate to take advantage of this. But as time elapses the patience of study section members for no-pubs-from-new-lab starts to wane.

  • drugmonkey says:

    AAP- this has to do more with your conservatism in what you propose (no doubt for good, albeit defensive crouch, reasons). No reason you can't propose going big from the basis of your pubs is there?

  • drugmonkey says:

    Dave- on more than one occasion we've had a paper come out right as a grant hits because it has been forged by repeated submission rounds and ongoing "Preliminary Data" studies to address actual and anticipated reviewer concerns.

  • jmz4 says:

    @Dave
    Not really. 1) I think it is well understood by the readers of this blog that CNS papers are not substantially better than non-CNS papers (bigger, denser, or flashier, sure, but not better), so that's not really a brag. 2) Fine, replace "normal" with "non-glam".

    3) Being in a lab where the PI will only work on CNS papers has significant downsides. It means all the papers you started that don't score enough novelty points are just wasted effort or get packed into supplement on the one paper you do get out of your tenure there.

    I got a comment on my K99 application saying something to the effect of "applicant has published well, but that may be due to the labs he has been in." So I've wondered if this is a thing study sections take into account. Ola's comment seemed to indicate it might be.

    DM's line of questioning seems to suggest my pedigree will be beneficial, I just wonder if I have to keep up the trappings of the Glameratti in order to reap the benefits. I'd be worried if that were so, because I don't I think I'll be in a position to become a hot-shot name in my field, and even if I could, I have little interest in replicating my mentors' idolatry of CNS papers.

  • drugmonkey says:

    jmz4-
    Probably the worst thing for someone in your position is to get into a sweet faculty job and fail to produce in the first three years. 1-2 is still honeymoon, after that it looks like you can't get it done. CNS? That would be a home run shot. Great. But if you miss this can look very bad. It is less important whether you've hit demi-glam instead of glam. Your advocates can work with pubs and argue how great the science is. They cannot work with an 0-fer, no matter how much they might like your research proposals.

  • Dave says:

    Especially with a K99. Stakes are high innit.

  • clueless noob says:

    It seems like this would be pretty field- and SS-specific. For instance, I was recently awarded my first R01, and at the time the resubmission was reviewed I had maybe 6 first and 1 last author publication (and 35 others, but still), all in society-level journals (JIF 2-5 or so). I was also an ESI about 3.5 years out of grad school, which probably helped a lot.

  • Philapodia says:

    "I had maybe 6 first and 1 last author publication (and 35 others, but still)"

    Having 42 publications less than 4 years out of grad school is probably on the far right side of the publication distribution curve. Many people doing this gig 10-15 years out from grad school don't have that many publications. This probably played a very big role in your funding, since it makes you look like a superstar. In a lot of fields getting 2-3 pubs a year is doing pretty good.

  • "I think that one of my biggest mistakes as a new assistant professor was to publish immediately whenever we made a cool discovery."

    I don't have a lot of experience with grants, so this could be a silly question, but why would it be a bad idea to publish your cool results and then propose other cool studies in your grant? I'm having a hard time seeing why prior publications on a topic mean that there's no room for new, exciting (and different) work on that same topic. Did your reviewers think your prior publications told us everything we needed to know, and that topic is now closed to science forever? Or is the issue that the new and exciting studies would be too far afield from what you want to do in terms of method or theoretical grounding or what have you?

  • jmz4 says:

    "I'm having a hard time seeing why prior publications on a topic mean that there's no room for new, exciting (and different) work on that same topic."
    -There seems to be a requirement for large amounts of preliminary (unpublished) data in applications. Not sure why or when that happened, or if it is evenly distributed among fields. It is also nonsensical, since it begs the question from whence the funds came to generate said data.

  • Luminiferous æther says:

    @jmz4 - preliminary data need not be unpublished. In fact, a great example of a published piece of preliminary data would be, say, a prior study demonstrating IP of the proposed protein of interest using a particular proven antibody and technique.

    That said, all of it should not be published.

  • Another Assistant Prof says:

    "I'm having a hard time seeing why prior publications on a topic mean that there's no room for new, exciting (and different) work on that same topic."
    -Because when a study section is realistically only going to fund a handful of grants, the ones with an unpublished new molecule or pathway seem more exciting than "that molecule she already published on last year."

  • Assistant Prof says:

    " People aren't expecting a paper in the first year."

    One would THINK this was the case. I know two junior PIs that were critiqued in NIH review for not having a paper out in their first year on the job.

    You'd be shocked at the sporadic complaints sent the way of us new PIs. I put out my first senior author pub in year 2. Put my first R01 forward in year 3, at which point I had 4 senior author pubs. I got complaints in the first round that my publication record was inadequate. I then published 6 new senior author papers in society-level journals (IF 3-9) during the year between my original submission and the revision. The reviewers commented on my upswing and then noted that my publication record was mediocre because the journals were "only mid-tier". Mid-tier! How dare I!

    Chip on my shoulder? Absolutely. My program officer went so far as to chastise me via email for not taking SS critiques on my productivity seriously enough, after she scanned the summary statement. Then she went to my pubmed record, tabbed down to start a new paragraph in her email, and wrote "Huh. Well, I can see you've published quite a few articles recently. Make sure you find a way to fix your reputation."

    That experience still strikes me as a bizarre and *incredibly* unfair way to treat new PIs. I have sat on panels and watched as mediocre publication records are completely ignored when the new PI has a grad or postdoc pub in CNS/comes from a glam lab. My limited experience suggests a glam pub, even during a training period, makes the eyes glaze over for the rest of the CV.

  • Luminiferous Aether says:

    @Assistant Prof - did you experience this at different study sections and/or different ICs, or was it just at one particular SS? If not the former, I would reckon that this is another anecdotal datapoint as to how SS-specific things can be. And also sub-field specific (?).

  • Dave says:

    @Assistant Prof - that sounds bizarre. What field/IC?

  • drugmonkey says:

    Remember folks: just b/c some jackhole reviewer says something like this, it doesn't mean it decided grant outcome. It could be the SS all agrees- was it in the resume of discussion? More likely, if it had come to discussion there would have been an advocate shutting that down.

  • lurker says:

    There is zero accountability for reviewers who act like jackholes, especially if the reviewer is a BSD ripping on a noob PI's first grant. I got reamed for not proposing experiments to study this mechanism that another lab scooped us on a paper that came out in September when the SS met in October. BSD jackhole reviewer wanted me to have ESP when I was writing and submitting my grant app in June. Of course I asked the PO, who only listened on the meeting, but did not recall this being an issue, nor had anything helpful to say.

    Like the other AssProf, I have a giant chip on my shoulder too that I will wear the rest of my life...

    More unfortunately, I have seen this type of crap review now in multiple SS's, as the competition gets more vicious, the churn more relentless. A fine Lord of the Flies situation we all are in because the BSDs rule the arena...

  • Dave says:

    Remember folks: just b/c some jackhole reviewer says something like this, it doesn't mean it decided grant outcome

    POs should help in this regard though, right? Especially for us noobs. They seem to fan the flames at times, rather than helping the candidate understand what's going on.

  • drugmonkey says:

    POs have many remarkable blind spots when it comes to what is really going on with grant review. We've discussed this Dave.

  • Dave says:

    I know we have, but it's annoying.

  • drugmonkey says:

    lurker- struggle with that chip. Lord knows that I do. Golden rule, pecking order, ladder pulling or....?

  • Assistant Prof says:

    "just b/c some jackhole reviewer says something like this, it doesn't mean it decided grant outcome. "

    Yes, but it taints the review process and points to a bigger problem. The "review criteria" are generally a guise for the gut feeling a reviewer gets when they read the application. I'd actually be fine with gut feeling, but let's not pretend that my publication record *actually* mattered. And if my publication record didn't matter, and you have no other complaints about my qualifications, then give me a damn 1 in that category and ding my grant where you actually think it has problems.

    Luminiferous , only one SS, and it wasn't a great fit. Their real beef was that they didn't find my work particularly innovative for their purposes, and I get that. But I wasted 1 year in revision working on publications and responding to technical critiques. They responded that I addressed every single critique (literally, that was how it was phrased), gave me that lovely slam on my "mediocre" record, and then decided that the problem I was trying to tackle was not the one they wanted to see me working on; they made several suggestions for a new direction for the grant and ultimately gave it an unfundable score.

    I totally get why they didn't like my grant. They weren't excited about the impact of the problem I wanted to address and that's fair. But I'm ticked that the guise of productivity and specific concerns over approach (which were all about wanting additional repeats on preliminary data, specific experiments, etc - everything that I could and did address) distracted me. If I understood that they never would have been excited about the grant no matter what, I would have immediately turned to a new SS, and I'd be that much further along the process. That's what's so damaging about this.

    I saw a grant I thought was absolutely atrocious. It was from a spectacular PI. It was highly innovative. It addressed a significant problem. The environment was superb. But the scientific premise was a joke; there was no plausible way they would achieve their goals other than "hey we've done cool stuff before and this will be awesome". I gave them top scores in every category except approach, because approach was where the problem was and nowhere else. I and the other reviewers gave atrocious preliminary impact scores and of *course* the grant was triaged; I wasn't out of line in my assessment and my message to the PI was quite clear. Why can't review work like that? Is it so scary to be accurate in your scoring of the sub categories? Just because you give me a 1 or 2 as an investigator, appropriate to my career stage, doesn't mean you have to average that into my impact score; give me an impact score that represents your overall feeling and then use the individual criteria - in this case, innovation and significance - to actually communicate something. When all 5 criteria score are in the 3-4 range and some of the critiques don't even make sense, the system fails. I had a different program officer tell me to ignore multiple, specific critiques because "Reviewers need to find a way to lower your individual criteria scores; sometimes you have to ignore it". If that's not a broken review system, I don't know what is.

    Dave, engineering, preclinical therapeutics

    "POs should help in this regard though, right? Especially for us noobs. They seem to fan the flames at times, rather than helping the candidate understand what's going on."

    Yeah. That one PO did a lot of damage. I can be angry about it now; at the time, I was absolutely crushed.

    I certainly understand that this is a one-off example, but, at the same time, n=1 matters, and there are a lot of n=1s. While I don't know anyone who has had their publication record attacked in that particularly weird manner, all of the junior PIs I know have some equivalent kind of "WTF" story. It's disheartening. Reviewers should not be allowed to pass along that kind of crap; isn't that the job of the CSO? "Willingness to ignore unfairness" should not be the criteria by which scientists stay in this career path.

Leave a Reply