NIH encourages pre-prints

In March of 2017 the NIH issued a notice on Reporting Preprints and Other Interim Research Products (NOT-OD-17-050): "The NIH encourages investigators to use interim research products, such as preprints, to speed the dissemination and enhance the rigor of their work.".

The key bits:

Interim Research Products are complete, public research products that are not final.

A common form is the preprint, which is a complete and public draft of a scientific document. Preprints are typically unreviewed manuscripts written in the style of a peer-reviewed journal article. Scientists issue preprints to speed dissemination, establish priority, obtain feedback, and offset publication bias.

Another common type of interim product is a preregistered protocol, where a scientist publicly declares key elements of their research protocol in advance. Preregistration can help scientists enhance the rigor of their work.

I am still not happy about the reason this happened (i.e., Glam hounds trying to assert scientific priority in the face of the Glam Chase disaster they themselves created) but this is now totally beside the point.

The NIH policy (see OpenMike blog entry for more) has several implications for grant seekers and grant holders which are what form the critical information for your consideration, Dear Reader.

I will limit myself here to materials that are related to standard paper publishing. There are also implications for materials that would never be published (computer code?) but that is beyond the scope for today's discussion.

At this point I will direct you to bioRxiv and PsyRxiv if you are unfamiliar with some of the more popular approaches for pre-print publication of research manuscripts.

The advantages to depositing your manuscripts in a pre-print form are all about priority and productivity, in my totally not humble opinion. The former is why the Glamour folks are all a-lather but priority and scooping affect all of us a little differently. As most of you know, scooping and priority is not a huge part of my professional life but all things equal, it's better to get your priority on record. In some areas of science it is career making/breaking and grant getting/rejecting to establish scientific priority. So if this is a thing for your life, this new policy allows and encourages you to take advantage.

I'm more focused on productivity. First, this is an advantage for trainees. We've discussed the tendency of new scientists to list manuscripts "in preparation" on their CV or Biosketch (for fellowship applications, say, despite it being technically illegal). This designation is hard to evaluate. A nearing-defense grad student who has three "in prep" manuscripts listed on the CV can appear to be bullshitting you. I always caution people that if they list such things they had better be prepared to send a prospective post-doc supervisor a mostly-complete draft. Well, now the pre-print allows anyone to post "in preparation" drafts so that anyone can verify the status. Very helpful for graduate students who have a short timeline versus the all too typical cycle of submission/rejection/resubmission/revision, etc. More importantly, the NIH previously frowned on listing "in preparation" or "in review" items on the Biosketch. This was never going to result in an application being returned unreviewed but it could sour the reviewers. And of course any rule followers out there would simply not list any such items, even if there was a minor revision being considered. With pre-print deposition and the ability to list on a NIH biosketch and cite in the Research Plan there is no longer any vaporware type of situation. The reviewer can look at the pre-print and judge the science for herself.

This applies to junior PIs as well. Most likely, junior PIs will have fewer publications, particularly from their brand new startup labs. The ability of the PI to generate data from her new independent lab can be a key issue in grant review. As with the trainee, the cycle of manuscript review and acceptance is lengthy compared with the typical tenure clock. And of course many junior PIs are trying to balance JIF/Glam against this evidence of independent productivity. So pre-print deposition helps here.

A very similar situation can apply to us not-so-junior PIs who are proposing research in a new direction. Sure, there is room for preliminary data in a grant application but the ability to submit data in manuscript format to the bioRxiv or some such is unlimited! Awesome, right?

15 responses so far

  • Draino says:

    In a recent faculty meeting, a junior PI brought up the topic of pre-prints. Senior PIs in my department scoffed at giving away your data -- our foundation's data, actually -- before proper peer-review. They thought it was a bad idea, very high risk but low chance that it would be taken as a seal of priority or a confidence builder with conservative old review panels. I'm inclined to agree.

    And welcome back, Monkey! 2017 was dreary and I spent way too much time on PubPeer.

  • assistant prof says:

    Preprints are wonderful. They're not "giving away your data"--they establish priority. People do see them. One of my preprints got cited by an eLife paper even before we got our first reviews back. As we know, peer-review can cause delays that are truly unprofessional. I still think peer review is great and all, but it's flawed and especially risky (for the reasons you cite) for early career researchers who need to balance productivity with impact.

    Maybe the NIH will inspire my dean, who is rumored to have said that preprints won't count toward tenure, since we have no way to assess their quality.... I'm at a top R1.

  • DrugMonkey says:

    I see the NIH policy as a very significant game changer - eventually even for Deans and P&T committees.

  • Anon says:

    "Well, now the pre-print allows anyone to post "in preparation" drafts so that anyone can verify the status."

    I'm formerly from physics and now biophysics. I would never upload a draft to arXiv; neither would most of my colleagues. When you submit your paper to a journal for review, that is, IMHO, the earliest at which it should be uploaded to a pre-print server. (I have many colleagues who would argue that one should wait until it has been peer-reviewed and accepted for publication by a journal, but I am not as strict.) I wouldn't waste my time reading someone's half-baked draft and would quickly come to ignore their content if I suspected that's what they were uploading.

    TL;DR: a pre-print server is not for "in preparation" drafts.

  • drugmonkey says:

    So shocked a fizzycyst thinks they know everything about how biomedical research scientists will behave.

  • Anon says:

    Hey genius, I'm in a biomed dept. now. The colleagues I spoke of -- they are biomedical scientists. But yeah, you know everything, of course. Maybe you have time to read people's junk that they just throw up there to make themselves look more productive than they are. You're welcome to it!

    And BTW, there's a reason arXiv has been so successful. Talk to me when bioRxiv is even in the same league.

  • drugmonkey says:

    Talk to me when bioRxiv is even in the same league.

    Typical fizzycyst, thinking this is some sort of competition over what is the MostAwesomest of Academic disciplines that DoThingsRight.

    I really don't care how a preprint server works for fizziks and maths nerds, and I don't need preprint servers in my discipline to be in any "league" with anything.

    What interests me is how these practices, newish to us, will serve our science and our careers.

  • Anon says:

    "What interests me is how these practices, newish to us, will serve our science and our careers."

    Oh sure ... and God forbid you should try to learn anything from other fields with a lot more experience with these practices.

    Why do have such a huge chip on your shoulder re: physicists? It's hilarious and really sad at the same time.

  • qaz says:

    I have three concerns about preprints that I have never had answered well.

    1. In my experience, reviews are helpful. They often change the outcomes and consequences of a paper. Sometimes they have even changed our results 180 degrees. I don't feel comfortable putting out half-cooked food. In my observation half-cooked food will put you sick.

    How do we prevent people from misusing preprints as "valid results"? If you think the rigor and reproducibility is a problem now, just wait until everyone is treating half-cooked results as real.

    2. Related to that, how do we monitor the interpretation of preprints? Does our hypothetical preprint server have a way of preventing anti-evolutionists or flat-earthers or tobacco-company-shills or whatever from posting something and getting "credit for a scientific discovery"? Do you really think the news media is going to understand the difference between a preprint server and a scientific journal? (Hint: if you are explaining, you are losing.)

    3. Once Deans and P&T committees start counting preprints as papers, how do we prevent spamming or stuffing where people submit crap to the preprints that gets counted in the list? When I was a grad student or postdoc I celebrated when I submitted a paper because it meant I had gotten through the gauntlet of my advisor. Now that I am a PI, I can't celebrate submission because there is nothing preventing me from submitting random strings as my "paper". Counting preprints as publications sounds like a recipe for bad actors gaming the system.

  • ThamizhKudimagan says:

    I had a minor query on bioRxiv that I hope someone may know the answer to:
    If your preprint had a few citations and then the paper gets published, do those citations of your preprint 'transfer' to your publication? If not, wouldn't this affect, say, the h-index?

  • drugmonkey says:

    afaik the answer is no....and you have the h-index q backwards.

  • ThamizhKudimagan - Google Scholar pools the citations - it treats preprints as another version of your paper, provided the authors/title are sufficiently similar. This is an extension of an older problem with citations going to the "early online versions" of published papers etc. Not all databases (ie Web of Science) index preprints, though. And Google Scholar doesn't always do this properly (https://academia.stackexchange.com/questions/95152/do-all-preprint-servers-have-the-non-updating-issue-in-google-scholar). But the fact that Crossref now covers preprints (https://www.crossref.org/categories/preprints/) should provide infrastructure for better solutions to this problem.

    qaz - I'll take a stab:
    1. Peer review is not guaranteed to fully cook your food. Readers should be skeptical of all work, especially work in preprints, but that doesn't mean it doesn't have value. We are ok with work that has not been peer reviewed appearing in talks and posters at conferences; preprints are narrative equivalents.
    2. I like the disclaimers that appear on bioRxiv and PeerJ Preprints: "This article is a preprint and has not been peer-reviewed" etc. Again, this doesn't solve the problem of people being overly credulous in general. I'm optimistic that widespread use of annotations (eg https://climatefeedback.org/) could help general readers better interpret news reports.
    3. P&T committees will have to look at the papers. Obviously reviewer burden is a serious problem, especially for grants, but I'd argue that simply "counting" papers is terrible practice anyway.

  • qaz says:

    Replies to Jessica:

    1. Most talks and conferences have entrance limitations, either through membership (like SFN) or actual peer review of the abstracts. I'd be more OK with a preprint server that was limited either through membership or limited peer review, but that's not a preprint server, that's a journal.

    Also, most talks and conferences used to be transient and would quickly fade into oblivion if the data didn't pan out. Preprint servers are really not. I wonder if preprint servers are going to end up being like legal opinions, where some random judge's opinion becomes citeable precedent even if the entire rest of the court disagreed.

    3. I'm sorry, but there is absolutely no way that P&T committees can "read the papers". A P&T committee covers dozens of fields and may cover the entire university. You can't expect a systems neuroscientist to understand cancer biology or nanoengineering or astrophysics at a level that they can judge whether a paper is worth tenuring someone over. Remember, we're not talking about popular science output for other people to get excited over, nor are we talking about the candidate's research statement that needs to explain for a general audience, we're talking about the primary papers they write which are going to influence their community. There has to be some process of determining if someone is being productive that does not entail reading their lab notebooks.

    In practice, the combination of counting papers and Glamour pubs and impact factor are processes that provide some semblance of that measurement. There are lots of problems with all of these measurements, but saying the solution is to read the papers is just never going to work.

  • Grumpy says:

    As a fizzycist (love that) who dabbles in NIH funding, I will take another stab at qaz's issues (mostly echoing Jessica):

    1. Already covered by Jessica. Of course there are reasons to be concerned that people will jump to conclusions and make more mistakes but this already happens and for the most part science still manages to progress. Perhaps clinical data should be treated differently, I dunno.

    2. Not sure about bioarxiv, but the arxiv is moderated and people can complain about articles. This happens frequently in physics so the cold fusion/etc pseudoscience people get banned from arxiv and have to post on other fringe Preprint servers. Science journalists also have reputations to preserve, and IME recognize that preprints should be treated with additional scrutiny.

    3. In my physics dept we can list submitted articles and link to preprints, but they are not treated the same way as publications (not sure if very mathy string theory type fields handle this differently). P&T committee needs to be aware of standards in a field (eg importance of conference papers for CS, arxiv for high energy theory) anyway.

    other worthwhile features of arxiv: they can be updated with fixed versions whenever the authors want and they are open access.

  • drugmonkey says:

    Science journalists also have reputations to preserve,

    I didn’t realize it is comedy hour.

Leave a Reply