What would you ask Sally Rockey?

(by drugmonkey) Feb 26 2015

Apparently Sally Rockey, NIH Deputy Director in charge of the Office of Extramural Research is on some sort of University tour because I have received several queries lately that go something like this:

Dear Drugmonky:
Sally Rockey will be visiting our University soon and I have the opportunity to ask a question or two if I can get a word in edgewise between all our local BigWig voices. Do you or your Readers have any suggestions for me to add to my list of potential things to ask her?
A. Reader

I have my thoughts and suggestions, of course, but mostly my Readers know what those are.

How about you folks in the commentariat? What would you ask Sally Rockey if you had her in a small room with your peers?

20 responses so far

Sustaining NIH funding then and now: 58% as many Type 2 awards in FY2014

(by drugmonkey) Feb 25 2015

Datahound has a cool new analysis posted on the distribution of competing continuation R01/R37 awards (Type 2 in NIH grant parlance).

There is one thing that I noticed that makes for a nice simple soundbite to go along with your other explanations to the willfully blind old guard about how much harder the NIH grant game is at the moment.

Datahound reports that in FY 1995 there were 2653 Type 2 competing continuation R01/R37 awards funded by the NIH. In FY 2014 there were only 1532 Type 2 competing continuation R01/R37 grants awarded.

I make this out to be 58% of the 1995 total.

This is a huge reduction. I had no idea that this was the case. I mean sure, I predicted that there would be a big decline in Type 2 following the ban of A2 revisions*. And I would have predicted that the post-Doubling, Undoubling, Defunding dismality would have had an impact on Type 2 awards. And I complained for years that the increasing odds of A0 apps being sent into the traffic holding pattern itself put a kibosh on Type 2 because PIs simply couldn't assume a competing continuation would be funded in time to avoid a gap. Consequently PIs were strategically putting in closely related but "new" apps in say Year 3 of the original noncompeting interval.

But I think if I had been asked to speculate I would have estimated a much smaller reduction.

*I can't wait until Datahound brackets this interval so we can see if this was the major effect or if the trend has developed more gradually since 1995.

9 responses so far

Nature is not at all serious about double blind peer review

(by drugmonkey) Feb 25 2015

The announcement for the policy is here.

Before I get into this, it would be a good thing if the review of scientific manuscripts could be entirely blind. Meaning the authors do not know who is editing or reviewing their paper- the latter is almost always true already - and that the editors and reviewers do not know who the authors are.

The reason is simple. Acceptance of a manuscript for publication should be entirely on the basis of what is contained in that manuscript. It should rely in no way on the identity of the people submitting the manuscript. This is not true at present. The reputation and/or perceived power of the authors is hugely influential on what gets published in which journals. Particularly for what are perceived as the best or most elite journals. This is a fact.

The risk is that inferior science gets accepted for publication because of who the authors are and therefore that more meritorious science does not get accepted. Even more worrisome, science that is significantly flawed or wrong may get published because of author reputation when it would have otherwise been sent back for fixing of the flaws.

We should all be most interested in making science publication as excellent as possible.

Blinding of the peer review process is a decent way to minimize biases based on author identity, so it is a good thing.

My problem is that it cannot work, absent significant changes in the way academic publishing operates. Consequently, any attempts to conduct double-blinded review that does not address these significant issues is doomed to fail. And since anyone with half a brain can see the following concerns, if they argue this Nature initiative is a good idea then I submit to you that they are engaged in a highly cynical effort to direct attention away from certain things. Things that we might describe as the real problem.

Here are the issues I see with the proposed Nature experiment.
1) It doesn't blind their editors. Nature uses a professional editorial staff who decide whether to send a manuscript out for peer review or just to summarily reject it. They select reviewers, make interim decisions, decide whether to send subsequent revised versions to review, select new or old reviewers and decide, again, whether to accept the manuscript. These editors, being human, are subject to tremendous biases based on author identity. Their role in the process is so tremendously powerful that blinding the reviewers but not the editors to the author identity is likely to have only minimal effect.

2) This policy is opt-in. HA! This is absurd. The people who are powerful and thus expected to benefit from their identity will not opt in. They'd be insane to do so. The people who are not powerful and are, as it happens, just exactly those people who are calling for blinded review so their work will have a fair chance on its own merits will opt-in but will gain no relative advantage by doing so.

3) The scientific manuscript as we currently know it is chock full of clues as to author identity. Even if you rigorously excluded "we previously reported..." statements and manged to even out the self-citations to a nonsuspicious level (no easy task on either account) there is still the issue of scientific interest. No matter what the topic, there is going to be a betting gradient for how likely different labs are to have produced the manuscript.

4) The Nature policy mentions no back checking on whether their blinding actually works. This is key, see above comment about the betting gradient. It is not sufficient to put formal de-identification in place. It is necessary to check with reviewers over the real practice of the policy to determine the extent to which blinding succeeds or fails. And you cannot simply brandish a less than 100% identification rate either. If the reviewer only thinks that the paper was written by Professor Smith, then the system is already lost. Because that reviewer is being affected by the aforementioned issues of reputation and power even if she is wrong about the authors. That's on the tactical, paper by paper front. In the longer haul, the more reputed labs are generally going to be more actively submitting to a given journal and thus the erroneous assumption will be more likely to accrue to them anyway.

So. We're left with a policy that can be put in place in a formal sense. Nature can claim that they have conducted "double blind" review of manuscripts.

They will not be able to show that review is truly blinded. More critically they will not able to show that author reputational bias has been significantly decoupled from the entire process, given the huge input from their editorial staff.

So anything that they conclude from this will be baseless. And therefore highly counterproductive to the overall mission.

37 responses so far

A lesson on Supplemental Materials

(by drugmonkey) Feb 24 2015

Good! That's my response. It is fantastic if someone can publish a paper on stuff that was essentially hidden in the Supplementary Materials of some other paper. 

This is great news. 

23 responses so far

Brian Williams, Bill O'Reilly and the RealProblemTM with the alleged profession of journalism

(by drugmonkey) Feb 23 2015

Brian Williams' evolving story..

"We". "Our". "in front of us". "all four of our low-flying Chinook took fire"

Bill O'Reilly's alleged war journalism story has been covered by David Corn who details how O'Reilly uses terms like "active war zone", "combat situation" and "I've been there".

What really chaps my hide is not that Brian Williams eventually conflated* all of his reporting in his own mind into it being the helicopter he was riding in that took a hit from a RPG. It is not the fact that eventually, at one point, O'Reilly directly conflates** his presence reporting the Argentine / GB conflict from Buenos Aires with the actual combat operations in the Falklands by saying "a war zone situation, in Argentina, in the Falklands".

What I deduce from all the he said/ she said is that Williams was indeed flying around in a Chinook when one of them in the group got hit by RPG. This appears to have been miles away from the chopper Williams was in and they were all ordered down to the ground for related or unrelated safety issues. It also seems reasonable that perhaps the chopper Williams was in was hit by the odd AK-47 round.

O'Reilly, it seems, was in Buenos Aires and never in the Falklands, over a thousand miles away. He was probably in a street protest. Probably, there were armed authorities, either police or soldiers present at the street protest. It may or may not have been a threatening and frightening situation to each individual journalist but there is no evidence of authorities firing on civilians to any large extent.

With this understanding of the probable facts, go back and look at how Williams AND O'Reilly carefully parse their words. You can see how carefully they select the words they use to describe things, how tenderly they craft their story to generate a false impression without actually lying. They want you to come away from their reporting with a feeling that they were deep in the danger. In O'Reilly's case, he seems mostly to deploy this for the purpose of bolstering his war-time correspondent journalism street cred, long after the primary reporting was done.

No matter.

This speaks to how the professional journalist type views the ethics and acceptable behaviors of their profession.

It is perfectly okay, even desirable, to create an entirely false image in the minds of their audiences just so long as they do not directly tell a clear falsehood. That is what their ethics hinges upon....whether it can be proved they told a lie. Creating a lie in the ear of their audience by using words that are not, strictly speaking, false? That's perfectly okay. Williams and O'Reilly are only being criticized now because they slipped over the line and said something that was directly falsifiable on the face of their words. Not because they carefully selected superficially true statements to create a false narrative in the mind of their audiences.

This is my problem with journalism.

*aha! gotcha.

**aha! gotcha.

27 responses so far

Patricia Arquette, Privilege and the Oppression Olympics

(by drugmonkey) Feb 23 2015

I was watching the Oscars last night when Patricia Arquette busted out some equal-pay feminism in her acceptance speech.

"To every woman who gave birth, to every taxpayer and citizen of this nation, we have fought for everybody else's equal rights," Arquette said, her voice intensifying. "It is our time to have wage equality once and for all and equal rights for women in the United States of America!"


I was hooting and hollering too much, I assume, because I got shushed. Apparently some other people in the room wanted to hear what else was being said or whatever.

So it was with some confusion that I saw backlash later on the Twitts about her. It seemed to be of the intersectionality sort of criticism. Also known as the Oppression Olympics. Not to make light of it but look, we all come with various attributes that confer privileges upon us in this society we inhabit. Most of us have one or two attributes that confer the opposite. Some unlucky folks have a pretty tough menu of biases slanting against them. So yeah, there seemed to be a drumbeat of Twitterage against Patricia Arquette's immense privilege of wealth, whiteness and heteronomativity. I thought at first that this was undeserved, based on what she said from the stage...it's the Oscars for goodness sake, of course they are all white and perfect and immensely rich.

Then today I finally happened upon her expanded backstage comments. From this account:

"The truth is: even though we sort of feel like we have equal rights in America, right under the surface, there are huge issues that are applied that really do affect women," she mused. "And it's time for all the women in America and all the men that love women, and all the gay people, and all the people of color that we've all fought for to fight for us now."

Breathtakingly tone-deaf.

Look, I've spent a lot of time in my life feeling sorry for myself. I get it. It is really, really easy to focus narrowly on that one aspect, attribute, experience, factor or misfortune that leaves the self at apparent disadvantage. And it is correspondingly easy to forget all about all the other factors and attributes that have conveyed immense privileges upon our lives.

This is not solved by the data, of course. Firstly, because we can all pick and choose which truthy stat we want to brandish. Is it equal pay? Very easy to brandish the generally accepted, broad brush stats for men versus women. And very easy to ignore that women of color are even more screwed than woman not of color. Easy to have no idea whatsoever how well minority men are paid relative to women not of color. Or what being gay confers in terms of salary.

And it is incredibly seductive to argue the anecdote. Well, Oprah! And J.Lo. And Eddie Murphy! And FFS Neil Patrick Harris is the Master of Ceremonies for goodness sake! They are sitting right there, so therefore why would anyone think of how their respective skin tones and desired life-partner would have anything to do with equal pay for women, eh?

Academic science is no different my friends. If this highly public case makes the intersectionality issue clearer to you than it has ever been, do try to turn that inwards.

We run across these examples on the blog all the time, of course. Whether we're discussing the struggles of women in science, the Ginther report, outing yourself to search committees or thesis advisors, the Baby Boomer hegemony of NIH Grant funding, postdoctoral pay rates or the evils of PIs with too many grants, the issues are the same.

"Sure, sure, there are these other biases in careers. But what is REALLY important is that I, the speaker, haven't experienced* any of those advantages that adhere to my classes and characteristics. And let me tell you about my specific set of life events that prove that really, I personally have been at huge disadvantage. So it is totally misplaced to talk about the general advantages of my characteristic X because the anecdote of me proves that X is much less important than this totally other thing that I happen to suffer from."

At this point one or the other of you, DearReader, may suspect I am talking about you in particular. Naturally, I am not. This is a common theme. Very common.

It is something that I have suffered from in my life and continue to do so. I have felt immensely sorry for myself a lot over the years.

Like many of you, I can claim one or two disadvantages within a context of immense privileges when it comes to pursuing the career of academic science. Like many of you, I CANNOT HELP BUT IGNORE MY PRIVILEGES AND PITY MYSELF ABOUT MY HARDSHIPS. Like many of you, I feel compelled to speak out about perceived injustices in the world. Like many of you, some of those injustices I speak about happen to be ones that I think affect me. Like many of you, some of those injustices I speak about do not happen to affect me in any direct way.

And, like many of you and Patricia Arquette, I often speak about injustices in a way that appears to ignore the fact that other people have it a lot worse.

Social media has a way of helping us to remember that other people have it even worse. And that trying to recruit others to help you in your fights, without ever appearing** to be that concerned about their fights comes across as selfish and tone deaf.

*of course you have, you just think that this is totally normal and average and deserved, and thus not worthy of inclusion in any discussion.

**For all I know Patricia Arquette is a huge fighter for underrepresented groups, including ethnic minorities and LGBT folks. But her comments certainly didn't convey that.

The tldr; version of this post:

7 responses so far

NIH clumsily tries to .. [something] ... for grant reviewers

(by drugmonkey) Feb 20 2015

I noticed a funny one in the NIH Guide notices today.

NOT-OD-15-035 Reinforcing Service to the Biomedical Research Community

Yes, yes. I see. "Reinforcement" of a behavior like "Service to the Biomedical Research Community" means increasing the strength or probability of the behavior. So yes, that's good. What are they trying to do here?

This Notice gratefully acknowledges, and seeks to reinforce, service to the biomedical research community by recipients of National Institutes of Health (NIH) research funding (see NOT-OD-10-089). Obtaining input from qualified experts across the entire spectrum of the extramural research enterprise furthers diversity of scientific thought, inclusiveness, and breadth of perspectives necessary to evaluate applications in a review process that strives for integrity and fairness. The interdisciplinary, collaborative, and global nature of biomedical research today requires increasingly complex review panels that need both broad and specific expertise in countless topic areas. Thus, the NIH, the biomedical research community, and the general public benefit from the service of NIH-funded investigators and maximize the Nation's investment in biomedical research.

Yes, yes. Very nice. but what are they actually doing to reinforce the behavior?

The NIH expects principal investigators of NIH supported grants and contracts to serve on NIH peer review groups, when asked. Therefore, the NIH expects grantee institutions and R&D contract recipients to encourage their NIH-funded investigators to serve on NIH peer review and advisory groups. These groups include Scientific Review Groups (or “study sections”) in the initial peer review of grant applications and technical evaluation of R&D contract proposals, National Advisory Boards or Councils (NACs) for second-level peer review, NIH Boards of Scientific Counselors (BSCs) for intramural programs, and Program Advisory Committees (PACs) for initiative development and concept review.

emphasis added.

Okay, so any University with a pulse is already encouraging their PIs to serve on study section. Right? They know about how this will help their bottom IDC line, yes? And if they are discouraging any subset of investigators from serving I imagine it is the Assistant Professors...who the NIH / CSR isn't looking to recruit anyway.


I have a suggestion. Two actually. The first one is hey, if you want to reinforce a behavior, why don't you use the delivery of a rewarding stimulus? I mean sure, you give us reviewers a delay in the submission deadlines, that's cool and all. But obviously the NIH thinks they need something more. How about protection from budget reductions? A couple of extra percentile points on newly competing awards?


Okay, that costs you money, I realize. How about something very cheap with some motivational value? Journals often publish a list of their reviewers at the end of the calendar year and thank them for their service. It's nice. But the NIH can do this one better. Set up a website with a list of reviewers and the number of grants they've been assigned to review. Maybe do it by year too and provide permalinks.

Trust me, academics will eat this up. They will check out how many reviews their buddies are/are not doing and give them a little hell for not matching up around the conference coffee table. They will start linking to their entry from their websites and bragging about it in their P&T documentation.

I wonder. Really, NIH. Do you have anyone making policy that understands people even the tiniest little bit? I am about the opposite of a people person and it took me like two tweets to think of this.

31 responses so far

Poll: graduate student applicant standards

(by drugmonkey) Feb 20 2015

What are your program's standards for GRE percentiles and GPA?

That is, what would be the minimum score that would be essentially unremarkable, and require no other compensating attributes, to justify an invitation to interview?

45 responses so far

Thought of the Day

(by drugmonkey) Feb 20 2015

I think I have received at least four NIH grants on the topic of "how in the heck is there very little science done on this entirely obvious (to me) idea?".
Continue Reading »

3 responses so far

If you care about the Emeritus NIH Grant...

(by drugmonkey) Feb 20 2015

...you need to take action on the official form.

Click through and get that done.

3 responses so far

Older posts »