Grant review via web forum

Jul 07 2015 Published by under Uncategorized

Does anyone actually like ANY part of reviewing grants via one of those threaded web forum things?

I hate them and I really resent the thought that my precious applications are subjected to this fiasco. 

Grrrr.

UPDATE:
One additional twist to consider. In my most recent few experiences with this as a reviewer, I have noticed that the per-reviewer grant load is low. As in 2-3 apps compared with 8-10 for a regular face to face meeting. I suppose when the costs for adding more reviewers are low (honorarium versus honorarium plus travel, plus housing), the SRO figures that bringing in more focused expertise per-application is a plus.

This comes at the cost of score calibration, in my estimation.

22 responses so far

  • Ola says:

    We did one when our SS was called off at the last minute due to the gubmint shutdown a while back. Took CSR 3 weeks to get it scheduled, by which time everyone had forgotten about their assigned proposals.

    My biggest dislike was the sniper attack mentality toward the end of the meeting. Some proposals were looking just fine, good scores all round, no real problems and probably on track for funding. Then some DB with a stick up their ass about a particular proposal or PI would come along and start nit-picking with an hour or so left on the clock, and before you know it the scores were tanking. Everyone was tired and so didn't want to dig into the details, happy to just go along like sheeple and drop their scores a bit before logging off. In a regular meeting, once a proposal is scored, that's all locked down, no going back. In the online system it ain't over 'til it's over, so a last-minute sniper attack can seriously fuck over an otherwise OK proposal that would've been OK at in-person review.

  • qaz says:

    I loved it when I did it. When I did it, we were told that we had to act as if we were really participating in the meeting (so no running off or only checking in once every few hours), and everyone did. We did 5 grants every two hours. At the end of those two hours, we had to score the grants with estimated scores. We still went back at the end to score final scores (which is true of in-person study sections - you can, no one does, but you can). But the scores and discussion were contained. I felt it was very successful and very fair.

    It had three very important consequences.

    1. People had time to consider things before responding. This meant that people had access to the entirety of the world internet during discussion. When two people in the group argued about whether the grant was replicating an old paper, people had time to go look up that old paper, determine what had been done, what was being done, etc. In a person-to-person study section, everything basically gets discussed immediately. It's pretty common for someone to say "I think X is true." and no one can check it. It's very hard to go look stuff up. In the chat version, looking stuff up provided balance.

    2. There was more time for discussion. The chat-based study sections I participated in did 5 grants every two hours. This meant there was time to go back and forth and we spent a lot more time on discussion. A lot more people participated in each discussion.

    3. It is easier for quiet (and non-English speaking) people to participate. They can type (and check their typing) much more easily than they can break into a heated person-to-person discussion.

    I found the chat-based systems much more fair.

  • physioprof says:

    What I dislike about the on-line review is that it makes it much harder for me to deploy my rhetorical skills to sway other reviewers to my perspective. This is for the reasons pointed out above.

  • Drugmonkey says:

    Qaz- I've seen a case or two where despite someone linking to external evidence proving their point, the opponents to that point just stick with "nope, in my opinion this is....".

    As one would predict.

  • drugmonkey says:

    Cost savings of online asynchronous review may be substantial, btw. http://drugmonkey.scientopia.org/2011/06/27/interesting-slides-from-the-csr-on-nih-grant-review/

  • qaz says:

    CPP - Yes, the online process makes it more difficult for grant review to be based on personality, aggression, and presence. I would think this is a good thing.

    DM - Perhaps. But that's a very different place when you've got the data in front of you (in the online chat) than when you don't (in person). At some level, we have to believe that we really are scientists and we really are swayed by data. It's a lot harder to argue "that's just my opinion" (usually expounded by older white male blowhards deploying rhetorical skills) when someone has just shown the data to the entire committee.

  • drugmonkey says:

    It is true that one can only get a sense of movement from those that comment. It could be the case that one reaches the silent peanut gallery types with linked papers and whatall.

  • Jim Woodgett says:

    Yes, there are some advantages such as suppression of dominant-negative personalities (which should be controlled by the chair but often aren't) and the cost savings and (more importantly) time savings are real. There is also the advantage of on-line look ups (but who doesn't do that at a face to face?).

    There are lots of issues as well. These include the loss of face to face scientific socializing (an critical spin-off of such meetings), reduced peer pressure on-line, tendency towards off-the-cuff rebuffs that you wouldn't get away with in person, dependency on participation and focus, tendency to multi-task, reduced training element (of younger investigators), engagement and not textual hints (that we tend to underestimate in our evaluation of whether someone really understands what they are saying), etc.

  • Josh VW says:

    I know of people with young children who are able to participate in the web-based reviewing who would not be able to go to DC because of childcare issues. (or at least would have a hard time - you can only drop the kids off at the friends/grandparents/etc for trips so often)

  • drugmonkey says:

    Broader participation of potential reviewers is a definite plus, JVW. Totally agree. I have for sure agreed to participate in reviews that I would otherwise have had to refuse.

    One additional twist to consider. In my most recent few experiences with this as a reviewer, I have noticed that the per-reviewer grant load is low. As in 2-3 apps compared with 8-10 for a regular face to face meeting. I suppose when the costs for adding more reviewers are low (honorarium versus honorarium plus travel, plus housing), the SRO figures that bringing in more focused expertise per-application is a plus.

    This comes at the cost of score calibration, in my estimation.

  • MoBio says:

    I don't like it and refuse to participate--though I'm delighted to do phone reviews.

    I've seen some pretty awful things happen to grants --intransigent outlier reviewers with absolutely no expertise killing grants--for examples.

    I didn't see much upside.

    I resonate with the upside for those who cannot travel and who may be inhibited from speaking up in a big room.

  • mH says:

    What about teleconferences as a happy medium? While sucky, I've heard it described by a panel chair as infinitely preferable to chat forums.

  • qaz says:

    Lower reviewer load leads to better (more focused, more time to get it right) reviews. Haven't we all been claiming that reviewers are overburdened?

    More diversity in reviewers is a good thing.

    I've done a few of these, and they've all (in my experience) been very positive experiences.

  • drugmonkey says:

    qaz- how do you think relative ranking and score spreading goes when everyone only has two grants? Don't you think more score compression will result?

  • qaz says:

    DM - No more than in a regular study section.

    Actually, less than in a regular study section. One of the main differences is that in the chat session study sections, one of the ways people communicate is that the entire committee is supposed to put in "your temporary score" sometime before the first hour of the two hour process is up. This means that there is a community-wide discussion about scoring. If someone is scoring higher or lower than the rest, than you get discussion of why. It's just another way to communicate your sense of the grant through the discussion.

    In general, my experience has been that there is much much more discussion from a much broader group in the chat-version. This means that you are less likely to be locked into the three primary reviewers. (Remember, with two hours, you have time to actually read a grant that you weren't primary on. Say, for example, someone says, "they propose to do XYZ in Aim 1." You think "that's stupid. Everyone knows XYZ doesn't work." So you take the time to read the full section on Aim 1 to see whether the primary reviewer got it right. You could never do that in an in-person study section when there's 15 minutes of discussion with people talking all around you. And the rest of the committee only has a few minutes to speak up.) In an in-person study section, most of the comments are questions "did they control for X?" "are they doing Y?" "what about Z?" In the chat-room, there was time to read through the grant to answer these basic questions, so the discussion was much better overall.

    An interesting point is that this worked really best when we were a regular study section and once a year we did chat-based. The other times were in-person. So we had a pretty good overall rapport as a study section already.

    The times I did it as an SEP was less positive. But, honestly, it was still more positive than doing an SEP in person, which is very frustrating, because usually there's a very wide range of topics and only one person's the real expert, so you pretty much have to listen to them.

  • qaz says:

    By the way, if it wasn't clear. Everyone can see that "temporary score". People later put in an official score that the rest of the committee can't see. But the entire committee can see the temporary scores, so you can change it as you get convinced by the discussion.

  • Established PI says:

    I have had very mixed experiences with asynchronous review. I found it worked best for shared instrumentation grants (S10), where the criteria are pretty straightforward. It was sort of OK for a special emphasis panel where the grant load (2-3 apps per reviewer) was low and the field tightly focused. It was a disaster, however, in a recent fellowship (F31/F32) review in which the application load (overall and per-reviewer) was high. Some reviewers spent very little time inputting comments and were largely absent from the discussion. Questions were left hanging for many hours while futile comments were being posted. The chair did his/her best to keep things going but, between non-participants and thoughtless comments, it was simply awful. I came away feeling grateful that none of my trainees had a grant in that cycle, since the scoring ended up being so arbitrary (even by NIH standards).

  • Neuro-conservative says:

    I have done asynchronous twice -- both times for SEPs. They were hands-down the worst reviewing experiences I have ever had. Absolutely no discussion -- more like a few disconnected tweets at random intervals from people quoting a line or two from their written critiques. No amount of cost savings would justify the fail. If you want to save money, a teleconference (which I have done many times) works much better -- everyone is focused and thinking together at the same time.

  • drugmonkey says:

    Maybe we are seeing a difference between SEPs and established panels? That would make some sense to me. Sadly, the SEP is more likely to use online review, right?

  • . says:

    Asynchronous chat is a problem for SEPs, many reviewers were AWOL, although the chair tried to contact and move things along. I've only been on chat with SEPs. Like not travelling, but phone review is much more dynamic, although when the phone reviews go for 4-5 hours, it gets looooong.

  • drugmonkey says:

    Everyone can see that "temporary score". People later put in an official score that the rest of the committee can't see. But the entire committee can see the temporary scores, so you can change it as you get convinced by the discussion.

    This is the only thing I like about online review. The fact that you get some clear feedback on whether you have convinced anyone or if you have failed entirely to advance your position.

  • chemicalbilology says:

    I love it! I find it to be very functional, maybe that's because of the particular group of people who are typically on the SEP for which I have done these. They have both face-to-face and online meetings for this SEP, so usually people kind of already know each other in this group form the face-to-faces, and the internet ones end up working out well. The chairs do a good job of guiding the conversation online, prompting people to address each others' rebuttals/additional evidence/etc., and the SRO is well-organized and pays attention to it all along the way as well.

    I certainly like it way better than the damn phone versions--I HATE talking on the phone in any situation, and hate even more trying to argue grant critique stuff on those big conference calls.

    For the panels I've been on, having two of these and one face-to-face per year would work great for reviewers and applicants alike. I'm about to start a regular standing study section appointment this fall, and we'll see whether it seems like that would work as well as for these other SEPs.

Leave a Reply