Archive for the 'NIH' category

Undue influence of frequent NIH grant reviewers

Feb 07 2018 Published by under Fixing the NIH, Grant Review, NIH, NIH Careerism, NIH funding

A quotation

Currently 20% of researchers perform 75-90% of reviews, which is an unreasonable and unsustainable burden.

referencing this paper on peer review appeared in a blog post by Gary McDowell. It caught my eye when referenced on the twitts.

The stat is referencing manuscript / journal peer review and not the NIH grant review system but I started thinking about NIH grant review anyway. Part of this is because I recently had to re-explain one of my key beliefs about a major limitation of the NIH grant review system to someone who should know better.

NIH Grant review is an inherently conservative process.

The reason is that the vast majority of reviews of the merit of grant applications are provided by individuals who already have been chosen to serve as Principal Investigators of one or more NIH grant awards. They have had grant proposals selected as meritorious by the prior bunch of reviewers and are now are contributing strongly to the decision about the next set of proposals that will be funded.

The system is biased to select for grant applications written in a way that looks promising to people who have either been selected for writing grants in the same old way or who have been beaten into writing grants that look the same old way.

Like tends to beget like in this system. What is seen as meritorious today is likely to be very similar to what has been viewed as meritorious in the past.

This is further amplified by the social dynamics of a person who is newly asked to review grants. Most of us are very sensitive to being inexperienced, very sensitive to wanting to do a good job and feel almost entirely at sea about the process when first asked to review NIH grants. Even if we have managed to stack up 5 or 10 reviews of our proposals from that exact same study section prior to being asked to serve. This means that new reviewers are shaped even more by the culture, expectations and processes of the existing panel, which is staffed with many experienced reviewers.

So what about those experienced reviewers? And what about the number of grant applications that they review during their assigned term of 4 (3 cycles per year, please) or 6 (2 of 3 cycles per year) years of service? With about 6-10 applications to review per round this could easily be highly influential (read: one of the three primary assigned reviewers) review of 100 applications. The person has additional general influence in the panel as well, both through direct input on grants under discussion and on the general tenor and tone of the panel.

When I was placed on a study section panel for a term of service I thought the SRO told us that empaneled reviewers were not supposed to be asked for extra review duties on SEPs or as ad hoc on other panels by the rest of the SRO pool. My colleagues over the years have disabused me of the idea that this was anything more than aspirational talk from this SRO. So many empaneled reviewers are also contributing to review beyond their home review panel.

My question of the day is whether this is a good idea and whether there are ethical implications for those of us who are asked* to review NIH grants.

We all think we are great evaluators of science proposals, of course. We know best. So of course it is all right, fair and good when we choose to accept a request to review. We are virtuously helping out the system!

At what point are we contributing unduly to the inherent conservativeness of the system? We all have biases. Some are about irrelevant characteristics like the ethnicity** of the PI. Some are considered more acceptable and are about our preferences for certain areas of research, models, approaches, styles, etc. Regardless these biases are influencing our review. Our review. And one of the best ways to counter bias is the competition of competing biases. I.e., let someone else's bias into the mix for a change, eh buddy?

I don't have a real position on this yet. After my term of empaneled service, I accepted or rejected requests to review based on my willingness to do the work and my interest in a topic or mechanism (read: SEPs FTW). I've mostly kept it pretty minimal. However, I recently messed up because I had a cascade of requests last fall that sucked me in- a "normal" panel (ok, ok, I haven't done my duty in a while), followed by a topic SEP (ok, ok I am one of a limited pool of experts I'll do it) and then a RequestThatYouDon'tRefuse. So I've been doing more grant review lately than I have usually done in recent years. And I'm thinking about scope of influence on the grants that get funded.

At some point is it even ethical to keep reviewing so damn much***? Should anyone agree to serve successive 4 or 6 year terms as an empaneled reviewer? Should one say yes to every SRO request that comes along? They are going to keep asking so it is up to us to say no. And maybe to recommend the SRO ask some other person who is not on their radar?

___
*There are factors which enhance the SRO pool picking on the same old reviewers, btw. There's a sort of expectation that if you have review experience you might be okay at it. I don't know how much SROs talk to each other about prospective reviewers and their experience with the same but there must be some chit chat. "Hey, try Dr. Schmoo, she's a great reviewer" versus "Oh, no, do not ever ask Dr. Schnortwax, he's toxic". There are the diversity rules that they have to follow as well- There must be diversity with respect to the geographic distribution, gender, race and ethnicity of the membership. So people that help the SROs diversity stats might be picked more often than some other people who are straight white males from the most densely packed research areas in the country working on the most common research topics using the most usual models and approaches.

**[cough]Ginther[cough, cough]

***No idea what this threshold should be, btw. But I think there is one.

18 responses so far

The past is prologue: Political NIH interference edition

Jan 24 2017 Published by under NIH, NIH funding, Science Politics, Science Vault

From a prestigious general science journal:

"Important elements in both Senate and the House are showing increasing dissatisfaction over Congress's decade-long honeymoon with medical research....critics are dissatisfied...with the NIH's procedures for supervising the use of money by its research grantees....NIH officials..argued, rather, that the most productive method in financing research is to pick good people with good projects and let them carry out their work without encumbering them...its growth has been phenomenal....[NIH director}: nor do we believe that most scientific groups in the country have an asking and a selling price for their product which is research activity...we get a realistic appraisal of what they need to do the job..the supervisory function properly belongs to the universities and other institutions where the research takes place....closing remarks of the report are:...Congress has been overzealous in appropriating money for health research".

D.S. Greenberg, Medical Research Funds: NIH Path Through Congress Has Developed Troublesome Bumps, Science 13 Jul 1962, Vol. 137, Issue 3524, pp. 115-119
DOI: 10.1126/science.137.3524.115 [link]
__
Previously posted.

9 responses so far

Thing I learned on Twitter

Jan 18 2017 Published by under Grant Review, NIH

Valid #NIHgrant review is determined only by the aspects of the application that I excel at and any other influence is unfair bias.

2 responses so far

Ask DrugMonkey: Should I go over my NIH Program Officer's head?

Jan 12 2017 Published by under Ask DrugMonkey, NIH, NIH Careerism

I get this question from grant applicants now and again so I thought maybe it was time to answer it on the blog. The latest version was from the Twitts:

Ok, first of all "escalate" is the wrong way to think about it. So don't do that. But you should absolutely explore the opinions and input of other Program Officers if you are unhappy with the responses (or lack thereof) that you are getting from your assigned PO.

As a brief reminder, many if not most of the NIH IC's have their POs arranged in a hierarchical structure. The smallest unit is typically a Branch, inhabited by ~2-5 POs, one of whom is the Branch Chief. The grant applications assigned to those POs will all share certain scientific properties, depending on how the Branch is designed. The individual POs in the Branch may have primary distinct roles and expertises in terms of their portfolios but there will be substantial overlap. The Branch Chief is responsible for all of the grants and applications in her Branch, obviously. These are small groups of people so, also obviously, they are closely interacting colleagues. They talk to each other a LOT about the business of the Branch. This is one practical reason you don't want to think about "escalating" and you want to approach matters carefully. The whole Branch may actually share your assigned PO's low opinion of your work. The Chief may be totally buddies with your assigned PO and not really appreciate you screaming about how she or he is incompetent, biased and shouldn't be working for the NIH at all.

Branches are collected into Divisions. I'm a little less certain about the universality of how ICs are organized on this but sometimes the Division director also functions, in essence, as a Branch Chief. She just also has the responsibility for overseeing the entire Division of related Branches.

Still with me? Take a stroll on the Organization page of your favorite IC to see what I mean if this is confusing.

Division directors are allowed to talk to God, aka the IC Director. What I mean by this is that when it comes to the hammer and tongs discussion of what is to be funded, what can possibly be picked up with exception funding, etc, it is the Division director level that is making the case. To all the other Division Directors and to the IC Director. I think they are the ones called upon in Council meetings, generally, if a specific question arises.

The point here is that the Division Director needs to know your applications too. They have a direct chain-of-command responsibility for them. And ultimately they have a responsibility for the performance of the entire Division portfolio of funded grants. They are involved.

Another thing to remember. POs get promoted up the ranks. The Branch Chief of today might be the Division Director of tomorrow. Your PO may become Branch Chief. Also, there can be some shuffling of individual POs across Branches (and even ICs as it happens).

This is why I continue to bang on about how it is in your best interest to meet POs, many of them, and to continue your relationship with them when opportunities arise (annual scientific meetings, for example).

So, back to the question. This usually arises because the applicant feels like their assigned PO is just not interested in their work. The PO may never return their calls. The PO may actively criticize their Specific Aims and tell them not to apply. The PO may be giving all sorts of unhelpful advice or just sticks to the mantra (I advise you to revise and resubmit). The PO may be refusing to push for a pickup for a grey zone score.

An obvious thing to do is to appeal. To try to get someone else.

This is a reasonably good idea. You just need to approach it judiciously. POs can be biased or they can just not "get" your work or proposal. They may have applications on their list that are higher priority to them. They may still be bitter about something that happened with your grad student advisor*!

If your PO is not your Branch Chief, that is probably your first stop. As I say above, it is possible that she knows all about your situation but perhaps she does not. So give it a try. It is also not impossible that she knows all about the limitations of PO X under her Branch but can only really act when someone complains.

When you take it up the chain, I always think the best approach is to be in a stance of seeking advice, rather than complaining about your rights.

"I don't understand...there is a lack of [feedback, enthusiasm, explanation]...perhaps my applications are being assigned to the wrong PO, would another one be better?"

That sort of thing. You can take this same approach with the Division Director. If you do this, however, you need to express doubt that the original Branch is the right one and find some key words in the description of another Branch to suggest perhaps that is a better fit.

Ultimately, sure, you can take this straight to the IC Director. Even the NIH Director, I suppose.

Your ability to get them to take your call or pay any attention to your concerns whatever will depend on your status in the world. I've definitely had senior colleagues who are in continual contact with IC directors and would for sure talk to them directly about grant matters. Things as specific as picking up a near-miss grant application for funding. If you happen to know an IC director well, sure, go for it when the situation is really critical. Other people are sure as heck doing it so why shouldn't you?

I'll close by reiterating that you need to be judicious about this. Keep entitled demanding far away from your thoughts. Keep angry complains about the bias and incompetence of the PO that is frustrating you out of your mind. Take the position of seeking information. Strike an attitude of not understanding why your experience is different from the advice you are getting to contact POs.

__
*Kidding?

17 responses so far

Today's ponder

Dec 19 2016 Published by under NIH Careerism, Tribe of Science

Today's version of this was me pointing out that if you are on a "9 month appointment" of Salary X but every workplace expectation is that you will be doing University related work for 12 months, that in fact Salary X is your base 12-mo salary. The "9 month" thing is a dodge the Universities pull to turn your job into a contingency plan like selling cars.

If you sell a grant idea, you get to bonus your Salary X to the tune of those three extra summer months.

I'm sure there are a lot of fancy accounting reasons Universities pull this. There is certainly a whiff of distasteful "sing for your supper" in the underlying expectation that such Profs must acquire extramural funding to pay for themselves that I'm sure is being whisked aside with this dodge.

What I don't understand is why so many of the victims of such schemes are so amped to defend them and call me terrible for pointing this out.

Look, if there is genuinely a situation where your Professor career is a-okay from start to finish if you only work 9 months out of the year than sure. I buy it. This person's 9-month salary is plausibly a 9-month salary. I'm going to raise an eyebrow if they don't cut off your card key access and VPN over the summer but....okay, fine.

But, the second you have a situation where you are expected to work those extra three months on University related business in order to retain your job or to advance normally (see: tenure) then this is a base salary for a 12-month job.

33 responses so far

The Pirate Stronghold Strategy

Dec 16 2016 Published by under NIH Budgets and Economics, NIH Careerism

Being a pirate probably really sucked.

Your odds of profiting from a raid were dodgy. Some of the victims fought back. The authorities might show up. Don't even start with me about the storms.

If you were a pirate captain....whooo. Do you know how hard it is to get good help? How expensive to refit and provision a ship? And where do you store your money so that you don't lose it and can afford to pay crew if the raid didn't go well this time?

Especially when you are constantly on the run?

Wouldn't it be great to have a place to go? Wouldn't have to be fancy. Just some basic support to help you refit the ship, provision it and hire crew for your next raid on the coastal settlements.

This is what NIGMS has been doing with their strategy of getting sustenance grant funding to as many of their people as possible. Keep lots of privateer crews, sorry, labs, alive...but just barely. Then you know they will launch raids on the other ICs to bring in their booty. Which they will spend a lot of back at the pirate stronghold.

24 responses so far

Here's the deal

Dec 14 2016 Published by under Careerism, NIH Careerism

When a lever of power unexpectedly extends into your operant chamber, press it.

That is what they have done to get to where they are. Constantly.

I know it irritates you that the world works this way. It irritates me too. This irritation changes nothing.

Take the opportunity. If needs be, remind yourself of all those times the system screwed you over. Let this make up for that.

Press the damn lever.

14 responses so far

I blew it, here's the version of 21st Century Cures that was passed by the House

Dec 01 2016 Published by under NIH, NIH Budgets and Economics

ooops.

I had the wrong version. Thanks to Jocelyn Kaiser of Science mag for alerting me.

The 21st Century Cures site is here and I think the right version of the bill is here in PDF form.

12 responses so far

Overtime rules

So. A federal judge* managed to put a hold on Obama's move to increase the threshold for overtime exemption. Very likely any challenge to this will fail to succeed before a new Administration takes over the country. Most would bet there will be no backing for Obama's plans under the new regime.

NIH is planning to steam ahead with their NRSA salary guidelines that met the Obama rule. Workplaces are left in a quandary. Many have announced their policies and issued notification of raises to some employees. Now they are not being forced to do so, at the last hour.

My HR department has signaled no recent changes in plans. Postdocs will get raises up to the Obama threshold. There are some other categories affected but I've seen no announcement of any hold on those plans either.

How about you folks? What are your various HR departments going to do in light of the de facto halt on Obama's plans!

__
*activist judge

56 responses so far

Foreign Applicant Institution NIH Grants

Nov 21 2016 Published by under NIH, NIH funding

The NIH allows non-US Universities and other institutions to apply for NIH grants. I don't pay much attention to this issue so I don't know how many.
What I am curious about is whether the PIs who review have noticed anything about these applications. How are they received in study sections you have attended? Is there a high bar for the unique environment or capabilities? 
I believe I was on study section during a transition where the foreign applications were essentially treated like domestic apps to where there was intense skepticism. This was around the mid-naughties, approximately a decade ago. 
I'm curious what you folks are seeing. 

26 responses so far

« Newer posts Older posts »