I'm mired in an effort to respond to a recent post of drdrA's over at Blue Lab Coats on the deceptively simple issue of a manuscript rejection. This post is apparently a rich vein of blog fodder because PhysioProf already responded to the issue of pleading one's case with the editor and I am trying to follow up on BugDoc's request to expand on a comment I posted at Blue Lab Coats. That effort is bogging down so I thought I'd take care of one little nugget.
The part of drdrA's post which made the most steam come out of my ears was the following point advanced by one of the paper reviewers:
"poorly performed gels and western blots which need to be improved."
Now, I am not privy to the details of this manuscript and critique other than what drdrA has offered. So I will have to make a few assumptions based on what I gather from people for whom "gels and western blots" form a significant fraction of their data. I assume that what the reviewer is talking about is that the figures in the paper do not appear "clean" in some way. Perhaps the background is higher than one might like. The bands perhaps not sufficiently defined. Maybe the gel became distorted so that the bands and lanes do not line up in a nice Cartesian orientation. In short, the figures do not look pretty to the eye. I gather from my colleagues that the ability to produce pretty figures is taken as a point of pride in one's "hands". This I can understand. I also understand that the visual quality is taken as an indicator of the scientific quality or veracity and that, as with drdrA's reviewer's comment, this visual quality is relevant to publication decisions. I find this idea idiotic.
I will note that this trend can sometimes be observed in reviewer comments on other types of data so I'll not need to go into a rant as to why a N=1 like a gel constitutes "data". Take a generalized case of a graph which includes an indicator of central tendency of a group of subjects (the mean) with error bars which describe the variability around that central tendency (such as the standard error of the mean or SEM). I have seen cases in which reviewers of such figures make the fundamental error of interpreting the statistical reliability of the effect being described by the much vaunted inferential technique of "eyeballing the error bars". This is an error similar to the request to "improve" the visual appearance of a gel.
Don't get me wrong. In many cases the formal inferential statistical methods will coincide with the impression the experienced reader gets from the aforementioned eyeball technique. So it is a decent proxy and most people use it to some extent- you go straight to the figures first, do you not? The two inferential techniques do not always coincide, however; this is particularly liable to FAIL in a repeated measures design. And of course the standard of one person's opinion about how big the error bars should be relative to the difference between means is an inherently arbitrary (and thus variable) one. Therefore, suggestions that one doubts the statistics because the error bars seem too large (I've seen this more often than I can believe) are wrong. Suggestions to run more subjects merely to decrease the size of the error bars and thus improve the visual appearance of the figure are wrong (not to mention a clear violation of the animal-use dictum to reduce the number of subjects, if it is animal data).
Now, the reviewer request for pretty data is to some extent only a minor annoyance. One can always point out to the editor in a rebuttal that what is important are the statistics or the meaning of the gel itself. Until one thinks about what this means for the conduct of science. I get this nasty feeling that an obsession with how the data look, over what they mean, is an encouragement to fake data. In the case of gels, an encouragement to keep re-running the experiment, tweaking conditions until one gets a perfect figure. Without, of course, any description of how many repetitions it took to get that one figure nor any description of the experimental variability. In the case of group mean data, an encouragement to toss outliers, to tweak inclusion criteria or treatment conditions. The impetus to use a more-than-sufficient number of animal subjects is not exactly fraud but does contravene the "reduction" principle of animal use.
DearReaders, perhaps you can assuage my concerns by explaining why better looking data are also higher quality data?