I love it when the reviewers really GET the paper, what we are trying to do and why it is important!
If asked to pick the top two good things that I discovered about grant review when I first went for a study section stint, I'd have an easy time. The funny thing is that they come from two diametrically opposed directions.
The first amazing good thing about study section is the degree to which three reviewers of differing subdiscipline backgrounds, scientific preferences and orientations agree. Especially in your first few study section meetings there is little that is quite as nerve-wracking as submitting your initial scores and waiting to see if the other two reviewers agreed with you. This is especially the case when you are in the extreme good or bad end of the scoring distribution.
What I usually found was that there was an amazingly good amount of agreement on overall impact / priority score. Even when the apparent sticking points / points of approbation were different across all three reviewers.
I think this is a strong endorsement that the system works.
The second GoodThing I experienced in my initial service on a study section was the fact that anyone could call a grant up off the triage pile for discussion. This seemed to happen very frequently, again in my initial experiences, when there were significantly different scores. In today's scoring parlance, think if one or two reviewers were giving 1s and 2s and the other reviewer was giving a 5. Or vice versa. The point being to consider the cases where some reviewers are voting a triage score and some are voting a "clearly we need to discuss this" score. In the past, these were almost always called up for discussion. Didn't matter if the "good" scores were 2 to 1 or 1 to 2.
Now admittedly I have no CSR-wide statistics. It could very well be that what I experienced was unique to a given study section's culture or was driven by an SRO who really wanted widely disparate scores to be resolved.
My perception is that this no longer happens as often and I think I know why. Naturally, the narrowing paylines may make reviewers simply not care so much. Triage or a 50 score..or even a 40 score. Who cares? Not even close to the payline so let's not waste time, eh? But there is a structural issue of review that has squelched the discussion of disparate preliminary-score proposals.
For some time now, grants have been reviewed in the order of priority score. With the best-scoring ones being take up for discussion first. In prior years, the review order was more randomized with respect to the initial scores. My understanding was the proposals were grouped roughly by the POs who were assigned to them so that the PO visits to the study section could be as efficient as possible.
My thinking is that when an application was to be called up for review in some random review position throughout the 2-day meeting, people were more likely to do so. Now, when you are knowingly saying "gee, let's tack on a 30-40 min discussion to the end of day 2 when everyone is eager to make an earlier flight home to see their kids"...well, I think there is less willingness to resolve scoring disparity.
I'll note that this change came along with the insertion of individual criterion scores into the summary statement. This permitted applicants to better identify when reviewers disagreed in a significant way. I mean sure, you could always infer differences of opinion from the comments without a number attached but this makes it more salient to the applicant.
Ultimately the reasons for the change don't really matter.
I still think it a worsening of the system of NIH grant review if the willingness of review panels to resolve significant differences of opinion has been reduced.
One of the erroneous claims made by Steven McKnight in his latest screed at the ASBMB President's space has to do with the generation of NIH funding priorities. Time will tell whether this is supposed to be a pivot away from his inflammatory comments about the "riff raff" that populate the current peer review study sections or whether this is an expansion of his "it's all rubbish" theme. Here he sets up a top-down / bottom-up scenario that is not entirely consistent with reality.
When science funding used to be driven in a bottom-up direction, one had tremendous confidence that a superior grant application would be funded. Regrettably, this is no longer the case. We instead find ourselves perversely led by our noses via top-down research directives coming from the NIH in the form of requests for proposals and all kinds of other programs that instruct us what to work on instead of asking us what is best.
I find it hard to believe that someone who has been involved with the NIH system as long as McKnight is so clueless about the generation of funding priorities within the NIH.
Or, I suppose, it is not impossible that my understanding is wrong and jumps to conclusions that are unwarranted.
Having watched the RFAs that get issued over the years in areas that are close to my own interests, having read the wording very carefully, thought hard about who does the most closely-related work and seeing afterwards who is awarded funding... it is my belief that in many, many cases there is a dialog between researchers and Program that goes into the issuance of a specific funding announcement.
Since I have been involved directly in beating a funding priority drum (actually several instruments have been played) with the Program staff of a particular IC in the past few years and they finally issued a specific Funding Opportunity Announcements (FOA) which has text that looks suspiciously similar to stuff that I have written, well, I am even further confident of my opinion.
The issuance of many NIH RFAs, PAs and likely RFPs is not merely "top-down". It is not only a bunch of faceless POs sitting in their offices in Bethesda making up funding priorities out of whole cloth.
They are generating these ideas in a dialog with extramural scientists.
That "dialog" has many facets to it. It consists of the published papers and review articles, conference presentations, grant applications submitted (including the ones that don't get funded), progress reports submitted, conversations on the phone or in the halls at scientific meetings. These are all channels by which we, the extramural scientists, are convincing the Program staff of what we think is most important in our respective scientific domains. If our arguments are good enough, or we are joined by enough of our peers and the Program Staff agree there is a need to stimulate applications (PAs) or secure a dedicated pool of funding (RFAs, PASs) then they issue one of their FOA.
Undoubtedly there are other inputs that stimulate FOAs from the NIH ICs. Congressional interest expressed in public or behind the scenes. Agenda from various players within the NIH ICs. Interest groups. Companies. Etc.
No doubt. And some of this may result in FOAs that are really much more consistent with McKnight's charge of "...programs that instruct us what to work".
But to suggest that all of the NIH FOAs are only "top-down" without recognizing the two-way dialog with extramural scientists is flat out wrong.
Datahound showed is that there are something on the order of 5,000 PIs that lose their last bit of NIH funding (as PI) in a given year.
What I want to see from McKnight is some clear identification of the deserving, amazing, super star impactful scientists who are in this situation.
This would go a long way toward is being able to assess his various claims for why these people (and surely he has a substantial list in mind- tens at the least?) are not funded.
Looks like both Oregon and Alaska passed initiatives to legalize the recreational use of marijuana.
(1) A person commits the offense of use of marijuana while driving if the person uses any marijuana while driving a motor vehicle upon a highway.
(2) The offense described in this section, use of marijuana while driving, is a Class B traffic violation.
a related item that I like because it calls for research:
(4) On or before January 1, 2017, the commission shall:
(a)Examine available research, and may conduct or commission new research, to investigate the influence of marijuana on the ability of a person to drive a vehicle and on the concentration of delta-9 tetrahydrocannabinol in a person's blood, in each case taking into account all relevant factors; and
(b) Present the results of the research to the Legislative Assembly and make recommendations to the Legislative Assembly regarding whether any amendments to the Oregon Vehicle Code are appropriate.
(13) "Marijuana extract" means a product obtained by separating resins from marijuana by solvent extraction, using solvents other than vegetable glycerin, such as butane, hexane, isopropyl alcohol, ethanol, and carbon dioxide.
aha, found this part:
SECTION 57. Homemade marijuana extracts prohibited. No person may produce, process, keep, or store homemade marijuana extracts.
so you can't make solvent extractions for home use but you *can* make vegetable glycerine extractions. Weirder. If the idea is to keep people from doing dangerous stuff with explosive solvents, this would be solved short of prohibiting "keep, or store homemade marijuana extracts", no?
In case you are wondering, vegetable glycerine extracts can be used in vape pen / e-cig type devices.
(b) Nothing in this chapter is intended to allow driving under the influence of marijuana or to supersede laws related to driving under the influence of marijuana.
Steven McKnight is simply intoxicated with his first taste of a social media imbroglio.
...inclusion of the volatile word [riff-raff] in the C3 essay prompted widespread attention. For this, I am simply delighted. This was my first brush with social media, and I can clearly see its power.
My next two essays, for the December and January editions of ASBMB Today, will deal with this flaw head-on. Trust me — I will take off the gloves and fight bare-fisted in those two essays.
I'm sympathetic. We all get a little giddy the first time we spark a social media dustup that gets all sorts of people talking about us and our pet opinions. Doesn't make him any more right in his opinions, but whatever.
What really interests me is his choice of a sports-analogy that is more apt than he realizes.
In the state of Texas, tens of thousands of young kids begin competing in organized football during elementary school. The enterprise is highly inclusive and exceedingly diverse. By the time these kids get to high school, they know a lot about the sport and have begun to develop skills. In high school, however, a weeding-out process begins. Not all kids make the junior varsity and varsity teams, and not all kids — even if they make the team — are apportioned equal playing time. As things progress to college, the weeding-out process becomes all the more acute. Playing on Friday nights as a high-school athlete in Texas is lots of fun with broad participation. Playing on Saturdays as a college athlete may be equally fun, but only the most competitive kids are on the field. The final weeding-out step comes when players are drafted by the National Football League — 32 teams sport 53-man rosters, meaning that only 1,696 young men are eligible to suit up for Sunday football. These are the best of the best athletes and are rewarded accordingly....I think of science in this same way.
Naturally this is just a re-hash of the baseball player analogy that Comradde PhysioProffe loves to deploy on these pages and it has a lot of truth in it.
I wrote a post once upon a time that is relevant to this issue. HIGHLY relevant.
Hmm. You know, I once watched a Rose Bowl in which an undersized mediocre looking, but nevertheless competent, quarterback did a decent job of not losing too badly to his opposition. Faint praise right? Well, homie went on to a NFL pro career and made tons of cash while being, well, still kinda mediocre. Back in the 1978 Rose Bowl, however, fans were lucky enough to watch one Warren Moon (Wikipedia) of the UW Huskies whup up on the U. Mich Wolverines (boo!). Of course, even for some third rate collegiate bowl game, the fans were lucky to have him.
He was recruited by a number of colleges, but some wanted to convert Moon to another position as was the norm for many major colleges recruiting black high school quarterbacks. Moon decided to attend West Los Angeles College in 1974-75 where he was a record-setting quarterback. After Moon showed his ability at West L.A., only a handful of four-year colleges showed interest in signing him. Offensive Coordinator Dick Scesniak [University of Washington], however, was eager to sign the rifle-armed Moon.
...oh, for chrissakes! People. This was the 1970s!!! Oh yeah, that's right. I remember those days. Black players can't be quarterback, you see. Don't have the right shoulder structure, it's a genetic thing doncha know. Plus, they aren't as good at all that, you know, quarterbacking stuff....
Come to think of it, I seem to recall some weebag Div I hockey player (who never ended up going anywhere professionally) writing some paper about how black people's hip structure precluded them from skating very well. (Or, skating like gangbusters and then fixing, oh, knees and hips for a living as an orthopedic surgeon)
Sorry. Back to the point. Oh yes. Warren Moon. Back to the Wikipedia:
Throughout his CFL career, Moon amassed 1,369 completions on 2,382 attempts (57.4 completion percentage) for 21,228 yards and 144 touchdown passes. He also led his team to victory in 9 of 10 postseason games. He was inducted into the Canadian Football Hall of Fame in 2001 and the Edmonton Eskimos Wall of Honour. In 2006, he was ranked fifth on a list of the greatest 50 CFL players presented by Canadian sports network TSN.
Not to shabby for a guy thought physically and mentally incapable of playing the quarterback position because of his skin color, right? Pretty decent.
What? What's that you say? There's more? Oh, riiiiigghht. That Warren Moon. The one who next jumped to the NFL and played from 1984-2000 as one of the more exciting quarterbacks to take the field,
Combining his NFL and CFL stats, Moon's numbers are nearly unmatched in professional football annals: 5,357 completions in 9,205 attempts for 70,553 yards and 435 touchdowns. Even if his Canadian League statistics are discounted, Warren Moon's career is still exceptional: 3,988 completions for 49,325 yards, 291 touchdown passes, 1,736 yards rushing, and 22 rushing touchdowns. During his NFL career, Warren Moon was named to nine Pro Bowl games (1988-1995, 1997).
I'm just getting going...
Alright. There's really not much point in going on and on to list Owens and Ashe and Gibson and the Williamses and Ribbses and Woods and Jones and all the other great athletes who thrilled (or continue to thrill) us with their class, competence and courage. Little point in detailing for each case where and when the operating rules of their sports (official and/or de facto) would have (or did..or still do) prevent them from excelling because of their skin color. Not much profit in describing how the overt bigotry of "they can't do it" papered over the fear that someone might be better than the rest of us. Silly to talk about the moral repugnance of categorically closing off the open field of play to some people just to benefit ourselves or those more like ourselves.
Because, you know, we're beyond all that sort of thing now. And...this is a blog that is supposed to focus on science. And the conduct of science. Which is objective. The only goal is the discovery.
To spell it out for Dr. McKnight and his fellow travelers....
Science is not a pure meritocracy. In the recent past when current generations were getting their start academic science was even less of a pure meritocracy. People who didn't look the right way, choose parents in the right way, express external dangly bits in the right way....all sorts of people who might have come to the table with the right brain equipment were systematically excluded. Denied from the competition before it even properly got started.
This still goes on. The "weeding-out" process that McKnight refers to (sports and science alike) is affected by bias. Opportunity is afforded to not the purest demonstrable talent. The pool of talent is chosen by the coaches. If they don't think a black kid can play quarterback, they will do their damndest to convert him to some other position so as to keep his "talent". How many Warren Moons did we never get to see on the field taking snaps?
As the man said, I think of science in this same way.
The coaches are the lab heads. The grayer bearded and bluer of hair. The gatekeepers are supported by their peers in review, in conference program committee and on hiring committees. Just as assuredly as coaches are supported by their owners, boosters, loyal alum, etc who have definite opinions on what a quarterback should be.
Somewhat less categorically, the coaches of sports and the coaches of science have individual biases as well. Show early signs of talent or even just effort.... and the coach gives you more playing time, calls the plays to you and lets you take the crunch-time shots. Sometimes the favored player is clearly inferior but the coach likes them for some reason (often enough because it is their own child) and wants to give them the best shot. Sometimes specialist talents are not used effectively simply because the coach can't see it or doesn't know how to work this talent into the mix so as to benefit the team (and that specialist talent's development).
I see this all the damn time in my now considerable hours spent watching my kids and their teammates play various sports.
And you know what? It is JUST like a dynamic that goes down in larger academic labs.
It is JUST like a dynamic that goes on in scientific sub-fields.
And McKnight's vision of who the "riff-raff" are and who the real scientists are cannot help but be similarly biased. We can't speculate on the nature of McKnight's biases...who knows, he may think that women or African-American or Asian scientists are the bomb and that standard old American white-guys like himself are a played out demographic*. I don't know**.
But what I do know is that whatever his concept of who is in the "riff raff" pool, he is biased and wrong. How can he not be? Most any individual person is going to be biased. That is why we use grant-selection and faculty-selection processes that depend on a committee of people. So as to hedge our bets against the bias of the individual attitude.
So I welcome this discussion McKnight would like to have. I look forward to further "bare-fisted" assertions of his position.
Because you know what? I know guys like this. I know what they are.
And I'm here to tell you. This guy is going to reveal further depths of his indefensible, personal-bias based and just-plain-wrong attitudes about who the best scientists are. In doing so, he will undercut support for anyone who might be nodding along with his truthiness at present. And that will be a good thing.
*I do actually know at least one highly accomplished privileged older white guy scientist who has expressed a sentiment like this and appears to believe it. Just for the record.
**HAHAHAHAHA, of course, I do know. This guy is going to get caught saying some horrible racist and/or sexist thing along the lines of Jim Watson's finest statements. There is no possible other way this can go down. It's a rule of nature.
It looks like Sally Rockey, Deputy Director in charge of NIH's Office of Extramural Research since 2005, is looking to depart the NIH.
The NU Presidential Search Screening & Selection has identified 4 finalists for the next president of the university. They are:
— Univ. of Nebraska (@u_nebraska) November 3, 2014
3. Dr. Sally Rockey, deputy director for extramural research at the National Institutes of Health
— Univ. of Nebraska (@u_nebraska) November 3, 2014
show that she is on the short-list to become the next President of the University (system) of Nebraska. Other shortlist candidates are a state level commissioner of higher education, a State University President and a State University (system) chancellor.
In the US some State University systems (i.e., multiple campuses which act as semi-autonomous Universities) call their campus heads President and the System-wide leader the Chancellor whereas other systems reverse these titles. This job appears to be the system-wide leadership position. This explains why there are two system-level leaders in the hunt.
It also may influence your opinion on the appropriateness of someone who has been a research administrator her whole career being in the running for such a position. Obviously she is being looked at as some sort of Federal grant rainmaker/expert to upgrade the amount of money that enters the University of Nebraska system from the Federal government and possibly other sources. I cannot imagine why else such a person, with no related experience heading a University or University system would be on the shortlist otherwise.
The main point of this news can be summed up in this handy figure from Jimmy Margulies, New Jersey Record, who was commenting on a different topic. The point remains, however.
The NIH is a sinking ship. I suspect that the folks at NIH realize this and the ones who have opportunity to cash in on their authoritah! by finding a nice top level administrative gig at one of the supplicant Universities will do so. The have-not Universities which find themselves in the most difficulty obtaining NIH funding will be desperate to land a rain-maker and even the "have" Universities may see this as a good investment. Especially if you have an IC Deputy Director or better, you can argue that they have significant administrative experience within an organization not entirely unrelated to academics. It should be an easy sell for a search committee to make the argument for NIH insiders to be considered for University President positions, Deans of Research and the like.
Is it a smart move? Well yes, if you think that their will be some benefit to their insider status. If you think that the replacement figures and holdovers will take the calls of these NIH emigres and listen to the concerns of their new University.
UPDATE: This news account explains that an attempt to close a Nebraska open-records law was made when the previous President of the UN system resigned.
As the law stands now, candidates may be kept private until the search for a president is narrowed to a pool of at least four applicants, all of whom must be disclosed. The bill would have allowed search committees to keep confidential presidential, vice presidential and chancellor candidates until they’ve narrowed the pool to one finalist.
Proponents of the bill say a closed search would allow for a better pool of applicants, including those who may otherwise be hesitant to apply and jeopardize their current position by publicly seeking another one. Opponents say the current law allows for students, faculty, the general public and the media to meet, investigate and learn about the candidates.
Hadley introduced the bill on behalf of the University of Nebraska’s Board of Regents after President James Milliken announced last month that he would be leaving Nebraska to become chancellor of the City University of New York.
The best known Klingon proverb
Revenge is a dish best served cold.
is proof that before they were warriors, the Klingons were academicians.
heard this a few days ago on the country station.
Thanks for sending me to look for it on YouTube MyTChondria, the video adds punch.
Rockey's post on PI survival also had a graph on the persistence of PIs in submitting applications.
She noted that the 1989 first-R01-equivalent folks dropped off in their grant submitting persistence around years 3-5 more than the younger cohorts.
A comment by qaz on the prior post of mine triggered a thought.
What about people who otherwise didn't really want or need a NIH R01 grant but it was a requirement for (or strongly supportive of) a tenure case?
If department expectations/preferences (for tenure or in who they hired in the first place) have changed since the late 80s, this could explain the difference in early drop-out, one-and-done rates across cohort.