Screwing over a junior colleague to make your point about Impact Factor is stupid

I generally like Stephen Curry's position on the Journal Impact Factor. For example, in today's confessional posting, he says:

mostly because of the corrosive effect they have on science and scientists.

In this we agree. He also posted "Sick of Impact Factors" and this bit focused on UK scholarly assessment. I enjoy his description of the arguments for why the Journal Impact Factor is leading to incorrect inferences and why it has a detrimental impact on the furthering of scientific knowledge.

But he pulled an academic nose sniffer / theological wackaloon move that I cannot support.

I was asked by a well-known university in North America to help assess the promotion application of one of their junior faculty. This was someone whose work I knew — and thought well of — so I was happy to agree. However, when the paperwork arrived I was disappointed to read the following statement the description of their evaluation procedures:

“Some faculty prefer to publish less frequently and publish in higher impact journals. For this reason, the Adjudicating Committee will consider the quality of the journals in which the Candidate has published and give greater weight to papers published in first rate journals.”

He then, admirably, tried to get them to waver on their JIF criterion....but to no avail

The reply was curt — they respected my decision for declining. And that was it.
I feel bad that I was unable to participate. I certainly wouldn’t want my actions to harm the career opportunities of another but could no longer bring myself to play the game. Others may feel differently.

So by refusing to play, he has removed himself as a guaranteed advocate for change. By drawing a hard, nose-sniffing line in the sand that he refuses to play if the game doesn't change.

I prefer a more practical approach to all of this. I think I've alluded to this in the past.

I certainly agree to review manuscripts for journals where they are overtly concerned with "impact and importance" and the maintenance of their Journal Impact Factor. Certainly. And no, I do not ignore their obvious goals. I try to give the editor in question some indication of where I see the impact and importance and whether it deserves acceptance at their high falutin' journal.

But I use my standards. I do not just roll over for what I see as the more corrosive aspects of Glamour Chasing. I rarely demand more experiments, I do not throw up ridiculous chaff about "mechanism" and other completely subjective bullshit and I do not demand optogenetics as the threshold for being interesting.

Stephen Curry could have very well done the same for this tenure review. He could have emphasized his own judgement of the impact and importance of the science and left the JIF bean counting to other reviewers. He could have struck a blow in support of the full and comprehensive review of the actual meat of this poor young faculty members' contributions. Instead, he simply left the field, after sending up an impotent protest flag.

I think that is sacrificing actual progress on ones goals for the fine feeling of chest thumping purity. And that is a mistake.

46 responses so far

  • What a delusional self-absorbed fuckebagge. What he did was not to "throw up a protest flag". Rather, he affirmatively hurt this poor fucker's chances of getting promoted by declining to write a letter. Because T&P committees take into account as one factor the number of letter writers who decline the invitation to write.

  • The Other Dave says:

    For sure, impact factors are relied on only by intellectually lazy reviewers and administrators.

    But yea, I totally agree with you DM. I don't even understand why he got so whipped out of shape. Based on the parts you quoted, I didn't even see any suggestion that he rely on 'impact factor'. My reading suggests merely that they wanted him to consider paper quality/visibility as well as quantity.

    And what was he expecting them to do when he declined to help? Were they supposed to send him candy and beg?

  • eeke says:

    Bruce Alberts has a very nice editorial about this topic in Science:

    http://www.sciencemag.org/content/340/6134/787.full.pdf

  • Here's the link to The San Francisco Declaration on Research Assessment (DORA) http://am.ascb.org/dora/ the Science article talks about.

    This could get interesting, if it takes off.

    Didn't know the Rebel Alliance was based in SF; I thought that was Starfleet. 🙂

  • Jim Woodgett says:

    The quality of papers is the defining factor (for quality of science rather than JIF and/or quantity) and the university is right in highlighting this (their error is in suggesting JIF is a surrogate for quality). My institute asks faculty undergoing review (we don't have any tenured faculty) to identity their most important papers. We provide citation data for these but expect reviewers to read the papers. We totally depend on peer experts and recognise this is work and an unrehearsed burden. That said, response rates are high (>70%) and as has been suggested, if several people decline, it's a worrying sign. Not the place for protest. Hope Stephen changes his mind on this in future.

  • The Other Dave says:

    DORA cracks me up. How much coffee and time did it take them to figure out that the best way to evaluate research is to actually read it, understand it, and think about it?

  • Grumble says:

    I don't understand why this Curry fellow didn't just write the letter and include a paragraph explaining why his letter did not address impact factor. All he has to do is say, hey, I'm an expert in the field and this guy's papers have moved the field forward in X, Y and Z ways, despite the fact that they were all in the International Journal of Druggy Primatology.

  • Interesting DrugMonkey, but your contention that I have screwed someone over is predicated on the assumption that I would have been a linchpin reviewer in this particular case; that almost no-one else could have knowledgeably assessed the candidate. I very much doubt that is the case. The field in which we have a common interest is a large one and I am of the view that there would be a large pool of knowledgeable reviewers to draw upon. That view helped to inform my decision.

    Nonetheless, it wasn't an easy decision and not one that I rushed into. I am not certain that I got it right — hence my decision to publicise it on my blog as a point of discussion, acknowledging that 'others may feel differently'.

    The one thing I regret is not chasing the institution more insistently for a response, a dialogue on the matter of what I see to be their flawed process. I can have another go now - particularly in light of the announcement of the SF declaration.

    You may well be right that I should have participated and opted to emphasise in my assessment that I wasn't going to play their impact factor game, focusing instead focused on the candidate's science. I suppose I gambled that an initial refusal would have more impact with the institution but then misplayed my hand by not following up.

    It's very late here now - won't be able to respond further till tomorrow.

  • Drugmonkey says:

    SC- but of those other colleagues are JIF fans, you've done double harm by leaving it up to them to reinforce the University's procedures.

  • Bill Hooker says:

    I use my standards. I do not just roll over for what I see as the more corrosive aspects of Glamour Chasing. I rarely demand more experiments, I do not throw up ridiculous chaff about "mechanism" and other completely subjective bullshit

    None of which is worth a small squirt of cold goat shit, because you've already grabbed your ankles when you agreed to review for them. When you do that you are ipso facto perpetuating the corrosive bullshit view that "impact" is a meaningful term for something that can be predicted and that some journals can thereby have higher standards than others. There's no downside to telling Glam Mags to fuck off and die.

    Which is a different situation from the one Stephen found himself in, where I mostly agree with you. So long as he took care not to rub his subversive views in their noses (which would only lead the Sniffy Club to disregard his reference, or see it as a negative), he could have done a young colleague a solid (deserved, by Stephen's own account) without significantly propping up the IF nonsense. The committe was including IF as one aspect of review -- a mistake, yes, but not the cornerstone of their existence or goals. Glam Mags absolutely and completely depend on it: when IF dies, so do they. So a sunshine policy is useful with committees, but harmful with Glam Mags.

  • Drugmonkey says:

    You do recognize your dream of universal rejection of Science, Nature and co-conspirators is never goin to happen, right bill?

  • Bill Hooker says:

    We'll probably never be rid of douchenozzles, either -- does that mean you should vote GOP?

  • The Other Dave says:

    @Stephen Curry:

    I get why you hate impact factors, and agree.

    What I don't get is why you assumed they were asking for you to use impact factors in your assessment. Based on the stuff you posted, and DM quoted, I don't see that they were asking for any use of impact factor. They just wanted you to evaluate journal quality as well as paper quantity.

    Promotion & tenure committees are often made up of a mix of people from different fields, and they often have no clue what top journals in a candidate's field are. To them, 10 papers in the XinShi University Journal of Applied Scientific Methodology Archives might indeed look twice as good as 5 papers in Cell. Do you know which publishers are better for people in the humanities? No, neither do I. But I do know that some publishers are indeed considered more prestigious for my colleagues who are poets and historians. That's why if I were on a P&T committee I'd ask the candidates colleagues to comment on that sort of thing. It doesn't mean I am hung up on it. It just means I want advice. That's your job as an external reviewer: Advise.

    As DM said, you wasted an opportunity to help a colleague and explain the true quality of his work.

  • GAATTC says:

    Sounds like Dr. Curry is having a tough time with philosophy meeting reality. I'm currently having a similar conundrum in that I think we should limit the number of grad students we accept, but will gladly take one if they join my lab. Hard to play both sides of the fence when one is passionate. I don't think the tenure committee will count this declining letter against the junior faculty member since it is clear that other factors are at play... In contrast to not writing a support letter because the person does not do good work or is a nut job.

  • @The Other Dave - "They just wanted you to evaluate journal quality as well as paper quantity."

    But that is the core of the problem. Use of the journal name as a signifier of quality is a dangerous practice and one that is built in to the assessment procedures of this institution. That's why I wrote to them seeking an assurance that they would discount any consideration of journal quality in their assessment before I participated. I hoped to provoke a conversation. Maybe I have, but I have no way of knowing because I didn't pursue the matter beyond their reply. That was a mistake.

    In retrospect it probably would have been better if I had posted my conundrum before I had to make a decision on how to proceed. I think it probably is feasible to register a profound objection but still provide an evaluation of the candidate - as DM argues.

    I hope also that the SF declaration will prompt universities to revisit their assessment guidelines with a view to removing criteria that depend explicitly or implicitly on impact factors.

  • The Other Dave says:

    @Stephen Curry:

    Dude, this is how you write stuff like that: "Dr. Z's work on X was mind-blowingly awesome. It changed the field in fundamental and productive ways. Not surprisingly, it was published in one of the fields most respected journals..."

    Or: "Dr. Z's work has been highly focused on the important problem of X, and consistent with that he has had a string of papers in journals that specialize in that area."

    Or whatever. You talk about his work. Mention the journals and why they're appropriate. No need to have a hissy fit.

    Others here can probably come up with other better sentences. I have to go poop.

  • mikka says:

    @SC
    I think that journal quality is a valid criterion when evaluating a CV. You shouldn't have asked them to waive this. You should have asked them for assurances that, as you said, the name of the journal would not be used as a proxy for quality. Two different things.

    (The part of the exchange with the institution that you posted doesn't say that they do the latter, but maybe they refer to this more explicitly elsewhere)

    Could you contact them saying that you reconsidered?

  • The Other Dave says:

    As for the SF declaration: The problem arose because people charged with review of people and science don't know jack squat about what's important and what's not, and people inside the field have obviously been too lazy to explain it.

    You, Stephen, are contributing to the problem by being too lazy (or whatever) to explain stuff to people asking for advice. Lack of informed advice is what makes people create and use bullshit like impact factors.

  • @mikka - "I think that journal quality is a valid criterion when evaluating a CV."

    Well, I think it's dangerous — promoting the kind of laziness that has led to the widespread mis-use of impact factors. Hence the motivation to challenge the status quo.

    However, I agree with the thrust of DM's argument that there are other ways to take a stand. The deadline for the original request has not yet passed so I have this morning written to the Chair of the relevant committee to point out my blogpost and DM's reply — hoping to pursue the dialogue on improvements to their assessment procedures. I have also offered my services as a reviewer.

  • drugmonkey says:

    One thing that we tend to lose track of in this type of discussion is the very real problem that using allegedly "objective" measures like Journal Impact Factor is supposed to address. That is, the effect of arbitrary reviewer bias.

    The flip side of encouraging someone like SC to just write an opinion of the science un-tethered to external measures is that there is room for sandbagging. Unjustified sandbagging. If there were indeed a trustworthy journal ranking and there were a good correlation between that ranking and the individual articles in the journal then the relatively naive viewer could calibrate the comments of a within-field reviewer.

    I think we should keep this in mind as we storm the castle of JIF.

  • Surely objectivity in such things is a bit of a myth? Who can measure the worth of a scientific paper? Though of course, if you know the field, it's not hard to arrive at a value judgement. I don't think there's any real escape from subjectivity, but it can be mitigated by taking multiple viewpoints.

  • The Other Dave says:

    Oh, DM, you crazy goof. I have been reading your blog almost from the beginning, back when you had Bikemonkey on board and CPP was co-author. Remember what you used to write back then? Remember what the premise of your blog originally was? You said over and over that the secret to being a successful scientist was understanding that science was a human endeavor, full of quirks and imperfections. And fighting that reality is foolish and impossible, like wishing the tide to reverse. Ok, you didn't really use those words. But I think it's what you meant. And you were right.

    So now we have one of those quirks: Journal Impact Factor. We can be all gooey and naive and try to wish these things away. Or we can accept that they exist, try to understand why, and work within that reality.

    So let's think about what JIFs really are and what are problem with them is...

    1) Does anyone here feel really qualified to judge work outside their own narrow subfield? (And if you do, watch this: http://www.youtube.com/watch?feature=player_embedded&v=7rMDr4P9BOw)

    2) Does anyone really read all the journals cover to cover enough to feel confidant that they can compare the average quality of papers in different journals?

    So how do you evaluate someone's CV? Do you study their field for years until you are an expert, and then read all of their papers and make a judgement? What if you are chair of the selection committee and have a stack of 200 CVs?

    You need some sort of proxy.

    That proxy could be someone else's opinion. Basically, you ask the opinion of someone else who you think is an expert and is willing to read all the papers. You shift the burden.

    But how do you compare opinions? You need something hierarchical. Numbers are good.

    So you start counting shit. Number of papers. Dollars of grant money. Whatever. But that's no good. You need some measure of quality, not just quantity. How do you translate quality into a number?

    THE WORLD NEEDS TO TRANSLATE QUALITY INTO A NUMBER.

    They desperately need some sort of hierarchical proxy for quality. Anything that promises to be those things will be embraced. JIFs, grant dollars, whatever. Face it. People will count shit. They need to, because they need to make decisions and don't have infinite time.

    So, what's our problem with JIF's? Well, personally my problem with JIFs is they are predicated on the assumption that journal quality is hierarchical. I think journals are different, but mainly because they are designed for different audiences. Just like Maxim is different from Archie Comics is different from Model Railroader Monthly is different from Rolling Stone. Which one is best? I don't know. They're different. Science magazines are the same.

    So the problem with JIFs is that they impose a hierarchical system on something that is not hierarchical. It is like saying that T-shirts are better than Blue Jeans are better than hats, or that rabbits are better than oak trees are better than clouds. The proxy is inherently illogical.

    But you can't just take away JIFs, because I sure as hell am not going to read every paper on every CV that crosses my desk. Plus, c'mon, some papers ARE better than others.

    So what can we use as a proxy for quality of science?

    Grant dollars, assuming that grants are awarded by experts after careful thought to people who have a history of good science? But that's circular, because grant dollars are necessary for good science.

    Letters of recommendation? But these are not really hierarchical, unless maybe we compute the ratio of positive:neutral words and phrases.

    Likes on facebook?

    What?

    What happens when we storm the castle of JIF, burn it to the ground, and then find ourselves with no shelter?

  • drugmonkey says:

    And fighting that reality is foolish and impossible, like wishing the tide to reverse. Ok, you didn't really use those words. But I think it's what you meant. And you were right.

    You misunderstood me entirely. My point has always been that one needs to understand reality and carefully consider one's fights within the context of what is, not what one wishes should be so.

    I do not typically recommend that we should act to reinforce those realities we find abhorrent but rather that we should oppose them in the way that has the best chance of having effect.

  • The Other Dave says:

    You misunderstood me entirely. My point has always been that one needs to understand reality and carefully consider one's fights within the context of what is, not what one wishes should be so.

    Huh? I thought that's what I said.

    Why are you suddenly writing like Edgar Allen Poe?

  • It would be an improvement of DoucheMonkey's incomprehensible discursive gibberish if he started writing like Poe.

  • BikeMonkey says:

    It was a US jr scientist right? Hey, just more grant money for the rest of us if some random Brit torpedoes the career, am. I. Right?

  • whimple says:

    What BM said. Seems like easy prey for the cull. If the C/N/S papers aren't there to speak for themselves, why bother engaging the letter writers to make an effort anyway?

  • BikeMonkey says:

    Cull!!!!!!

  • miko says:

    All hail The Cull.

  • Bill Hooker says:

    Ph'nglui mglw'nafh Cull Bethesda wgah'nagl fhtagn! Iä! Iä!

  • miko says:

    I was picturing Kali, but Cthulhu will do as well.

    Now: prepare the offerings.

  • whimple says:

    I thought SC already offered up the offering?

  • Eli Rabett says:

    There are some REALLY shitty journals out there and at some point somebunny has to say something to the College and University Committees that look at the paperwork.

  • CD0 says:

    I think that there is a difference between accessing to publish in C/N/S and other journals, at least in terms of effort and, in general, novelty of the concept. However, Science is ultimately validated through peer-review.

    I want to think that people aim to publish there because articles become much more visible and reach a broad audience. For the same reason, publishing an article/letter in these journals that do not reach, let’s say, 20 citations (as most Nature articles do), should be hold against the PI. If you are granted the privilege of publishing in a journal weekly read by millions of people and nobody cares, that should be stated in your evaluation.

  • miko says:

    This baffles me. If the place you work cares about C/N/S, they don't need someone outside to say whether you publish there or not.

    For the rest of us, there are real journals and fake journals, and that difference is easy as well. We could charitably interpret the instructions to mean this rather than JIF or glam-ness. Judging from the snippets of correspondence, it is SC who brought up JIF, not the university P&T Committee, who only said "quality of journals." I mean, maybe the candidate is publishing in the sasquatch genome journal or listing posts on friends' blogs as "research articles" (I wish I were kidding, though to be fair this was in the humanities).

    That said, the only rational purpose to having outside reviewers is to assess the science. Asking them to comment on the publication venue is just cowardly ass-covering in case you need "objective" ammo against the person.

    My position is that SC should have assessed the person but refused to discuss pub venue and to say, in detail, why. A dean wouldn't even know what you meant if you said s/he was statistically innumerate, but it might *shame* a P&T committee in a science faculty into rethinking their solicitation letter.

    On the other hand, if the point of this is to please The Kull, then by all means, forget everything you know about stats, pretend JIF matters for assessing papers and authors, and fuck everyone. But beware, The Kull, in its infinite indifference, cares not if you serve The Kull.

  • Eli Rabett says:

    No, you just write about the science and ignore the JIF stuff. If you really want to be mean, you write extensively about the impact on the field the candidate has had and will have.

  • DrugMonkey says:

    So, CD0, perhaps a system of quantification that represents the paper's citations as a z-score for the Journal in question?

  • The Other Dave says:

    I want to think that people aim to publish there because articles become much more visible and reach a broad audience.

    Back when we had to go to the library and flip through journals every month, this was definitely true.

    But now? If it's indexed in pubmed and available online, it's as visible as anything anywhere. I sometimes don't even notice the journal when I look stuff up.

  • mikka says:

    That could be gamed by systematically undershooting so your papers always rack in more citations than the journal average.
    Perhaps the data jockeys can come up with a way to match by MeSH terms and time since publication, and then rank by number of citations. That way we could tell how well a paper did relative to the field to which it pertains.
    Such a metric would normalize against overheated/high citation fields, allowing the world's foremost expert in the mating habits of the wild cockle to feel better about having a worse h-index than any random postdoc working on stem-cell-nanotech-against-bioterror.

  • The Other Dave says:

    @mikka: Who cares if someone is the foremost expert in a field that no one cares about?

    I happen to be the world's foremost expert in picking my own nose. But I don't think that qualifies me for anything in particular.

  • mikka says:

    TOD: I agree, but we should not equate high citation numbers (stuff people care about) with relevance (stuff that is important). Some fields have much higher citation numbers just because they are the fad du jour. Case in point: GWAS studies, which are the main drivers behind Nature Genetics IF surge.

    And if it turns out that you are in the Department of Picking Your Nose, maybe your tenure committee does care about the fact that you are the world's foremost expert on it. Or on cockle sex, or whatever field of interest with sub 5 IFs.

    Or should we just study the stuff that gets high citations? That way we could just cite each other until we choke on each other's dicks.

  • The Other Dave says:

    @mikka:

    Everyone cites GWAS studies because they are desperate to find health relevance for whatever dumb thing they might be studying. It's bedside-to-bench, which is almost as good as bench-to-bedside.

    I cannot cite my nose-picking studies because, despite an N of several thousand and extensive use of state-of-the-art visual and gustatory analysis, I cannot get my work published. Some of the problem may be that I insist on submitting my manuscripts the old school way, on paper, rather than via a web site. But how else can the editors and reviewers examine the attached samples?

    I have tried very hard to tell certain members of my department, as well as many young female scientists at meetings, that I am the world's foremost expert not just on cocklesex, but on sex in general. But they do not seem to believe me. I have also failed to get this work published, except in Penthouse Letters and YouPorn, but the impact factor of those places is apparently not impressive enough for people. They will not even view the data.

    Anyway, I totally agree with you that citation rates are stupid. My two personal examples above are consistent with that. But if you like, I will send you a booger and some porn if you promise to cite me in your next article.

  • mikka says:

    Send away. But the best I can do is cite it as a Personal Comucosation.

  • GMP says:

    What CPP said above -- declining to write a letter for tenure screws over your junior colleague, period, as T&P committees and the higher-ups look at the number of declinations. There are plenty of ways to discuss the impact of a colleague's work without talking about JIF or even the number of citations. That's why the letters are there and in my experience they carry a tremendous amount of weight.

  • [...] over at Drug Monkey’s blog. He is very critical of my stance and I think may have a point (see his comment thread for details). As a result, while I have not changed my view of the reliance that the selection [...]

  • [...] decision — mentioned above — not to participate in the review of a promotion candidate over at Drug Monkey’s blog. He is very critical of my stance and I think may have a point (see his comment thread for [...]

Leave a Reply