What is a scientific "observation"?

Reference to this https://t.co/hc9YYH8Myr popped up on the Twitter recently.
So what constitutes an "observation" to you?

To me, I think I'd need the usual minimum group size, say N=8, and at least two conditions or treatments to compare to each other. This could be either a between-groups or within-subject design.

22 responses so far

  • You pose a very important philosophical question, and I think the answer will vary somewhat from one scientific discipline to another. So for example, for a long time in ecology, direct observation without manipulation was not only considered legit, but somewhat of a gold standard. The idea was that any manipulation performed in the natural environment would obscure the true functioning of complex living systems. In recent years this approach seems to have fallen out of favor, albeit a significant contingent continues to endorse it (myself among them). I believe the approach to be legit, but it does require more thought and creativity concerning hypothesis development.

    Group size is an interesting issue as well. Often investigators assume they have sufficient sample size without validation. My personal preference is to apply confidence interval limits, with associated alpha (error) limits, which not only communicate the validity of sample size, but also reveal to what extent your data are quantitative (rather than qualitative). The two treatment (or more) approach you mention is an intriguing question as well. I've known some investigators that insist upon single treatments with an associated control- i.e., they would never conduct multiple treatments within a single experiment. Obviously issues arise if and when ANOVAs are applied, because performing a series of one-way ANOVAs is not as informative as doing a multi-factor ANOVA.

  • drugmonkey says:

    In my parlance a "control" is included in the treatment conditions for the most part (could be a normalizing control, sometimes, and thus not analyzed).

  • The Other Dave says:

    Wow. Your criteria exclude almost every great discovery in history, and rely on a particular school of increasingly out-of-favor thought developed only in the mid twentieth century. You know that, right?

  • potnia theron says:

    arbitrary group size? I don't think so.

  • DNAdrinker says:

    In physics, one measurement is an observation. The caveat is that the measurement must include error estimates, indicated here in parentheses. So, if Usain Bolt runs the 100m in 9.52(0.005) s he doesn't need to do it again to get the world record. Once is enough. We know he ran it in a time between 9.515 and 9.525 seconds.

    Many biomedical experiments are constructed in a way to answer a binary question. Does X effect Y? The answer is either yes or no. Usually, if the answer is "no", it is deemed uninteresting and not published. If the answer is "yes", people want to repeat that experiment a number of times to establish confidence in the result.

    Instead biomedical experiments should rephrase the "Does X effect Y?" question to be "How much does X effect Y?" Then one could say X increases Y by a factor of 1.10(0.001), which provides more information than simply answering "yes".

    Furthermore, if instead the answer is that X increases Y by a factor of 1.00(0.02), consistent with "No", it still leaves room for improvement. Another person might improve the measurement so that they can say that X increases Y by a factor of 1.01(0.005).

    One current problem with biomedical research is that the Yes/No answers get decoupled from error estimates, or some measure of certainty. Once a result gets published some take it as gospel, while others might think it is a borderline result. The physics approach avoids this problem by explicitly stating errors associated with the measurement and respecting new experiments that work to reduce the margin of those errors.

  • bacillus says:

    A major problem in my field, vaccines and adjuvants, is that adjuvant alone controls are almost never run, and probably account for most or all of the observed improvement in efficacy of the adjuvanted vaccine. Saline seems to be an almost universal control in this area. Also, challenging a vaccinated animal with the target pathogen 7 days later tells you nothing about long-term efficacy which is the key to most successful vaccines. I increasingly feel like my field has been taken over by used car salesmen. Though the reviewers too must shoulder their fair share of the blame.

  • drugmonkey says:

    Saline seems to be an almost universal control in this area.

    Seriously? Wouldn't it be an inactivated or somehow scrambled vaccine with the adjuvant?

    Also, challenging a vaccinated animal with the target pathogen 7 days later tells you nothing about long-term efficacy which is the key to most successful vaccines.

    This seems to be more of a failure along the basic->translational continuum, no? There are various versions of "Does it work?" to be answered. We have something like this in the field of drug abuse because there is this tiny cottage industry of anti-drug vaccines.....

  • shrew says:

    See, this was my point. Biomedical science has drifted so far from Fleming noticing some wacky shit in his petri dish that we wouldn't even call that an observation.

    My orphan figures which first came to my mind as "observations" are multi-panel, long term behavioral measures with within-and between-subjects persistent manipulations. But really those are eminently publishable as-is, they just haven't met the right graphs on mechanism to hook up with and settle down in a nice society level journal.

    But a Fleming-type "observation" is the animal in my last cohort that had a progressive ratio breakpoint almost an order of magnitude higher than dozens of other animals. What's up with her?

    Is publishing "Freakishly diligent female mouse loves responding for reinforcement, n=1" useful to anyone?

  • bacillus says:

    @ DM. Re: saline. Yes really. Also in many studies the unadjuvanted vaccine will casue a 2 log10 reduction in pathogen load, and the adjuvanted vaccine an additional log. Most people don't seem to realize that the vaccine alone is killing 99% of the bugs, and the adjuvant boosts this to 99.9%!

    You are correct about time between vaccination and challenge. You want the vaccine to elicit immunity asap, but also to persist for as long as possible. Most people are not interested in the latter because of per diems, but I occassionally look at efficacy in mice I vaccinated 1-2 years earlier since I dont pay cage charges.

  • The Other Dave says:

    @shrew: Fleming's discovery involved a lot of experiments and measurements. The guy's whole research program for a decade before penicillin was explicitly directed at discovery of antibacterials! The identification of penicillin's antibacterial properties was a very logical development from a leading lab trying to do exactly that.

    Also...

    I remember having a related argument with a roommate who was an animal rights activist. He was claiming that many of the greatest biomedical discoveries, like penicillin, didn't involve animal research. I showed him Fleming's original papers about penicillin, where thousands of animals from several different species were slaughtered in ways that would be illegal nowadays.

  • Who do these fuckefaces think they're gonna get to review "observations" one-by-one was they get submitted?

  • dr24hours says:

    This is beyond absurd. An observation is the noting of a phenomenon, n=1, possibly more. "The moon was at X angle of inclination at 9:53:23pm on January 3rd."

    If you have n>1 and a control group, you're describing an experiment, not an observation.

    #SMH

  • qaz says:

    n=8 whats? cells? petri dishes? experiments? replications using different techniques within the same lab? labs?

    That's silly. An observation is an experimental report. "The moon was at X angle of inclination at Y pm" is an observation. "A mean of X and a standard deviation of Y in my set of N animals" is also an observation. The real question that needs to be asked is whether that observation is interesting and whether it is likely to be useful for the scientific community to know. If the X angle of inclination of the moon is 50 degrees when observed from the equator, then that's a pretty important observation to report. If the distribution of your animals is bimodal than mean and standard deviation is a pretty useless observation. Historically, Phineas Gage was a pretty important observation. As was HM. And lots and lots of bunny hopping experiments with n=8 animals have turned out to be pretty damn poor observations.

    Science just ain't that simple.

  • drugmonkey says:

    They do not seem to be bothering about the quality of the observation as a basic threshold. The idea seems to be incremental advance and, presumably triage of importance by post review

  • Banditokat says:

    I have a lot of observations about y'all. A lot.

  • drugmonkey says:

    Yes?

  • David Condon says:

    I would guess they mean in this context informed opinions about phenomena. So you just write about whatever data you have on the subject, and then describe what you think the data represents in as objective a way as possible. So it's basically peer-reviewed blog posts. That's just my guess from looking it over.

  • Lincoln says:

    Are observations about Bundys publishable? 'Cause I got a few.

  • In response to "The Other Dave"-

    "Wow. Your criteria exclude almost every great discovery in history, and rely on a particular school of increasingly out-of-favor thought developed only in the mid twentieth century. You know that, right?"

    I am not sure what you mean by "your criteria," but for the moment shall consider that you elude to my comments concerning observation without manipulation. Certainly in many scientific disciplines the experimental approach is the only approach applied. Having worked in biochemistry for a dozen years, I cannot recall a single experiment I performed that did not manipulate one or more variables, and did so with appropriate positive and negative controls. Much of science operates this way.

    You are correct to assert that direct observation without experimentation has fallen out of favor- not only in ecology, but most other disciplines. The primary issue lies in attempts to translate correlation to cause & effect/ process & mechanism inferences. A great many attempts to do so in ecology have failed over the past 60 or so years. Thus, the majority of ecologists now poo poo the approach.

    I believe there are several issues giving rise to these failures. Likely the most common cause is an insufficient background in deductive reasoning. To infer process from pattern is by no means easy, or even moderately difficult. It is tremendously difficult. Another problem likely involves grant and publication cycles- which nowadays are relatively short. Typically the amount of data required to infer process from pattern is difficult (although not impossible) to acquire within these cycles. Lastly, the analyses required for pattern to process inferences are far more involved than what is usually applied to experimental systems.

    So I concur with your assertion that direct observation without manipulation is considered "antiquated" by most modern scientists. However there remains a significant contingent in ecology performing such work. And to remind everyone, while Mendel's works with "units of inheritance" were experimental in nature, a vast treasure of observational data were applied by Darwin in conjunction with deductive reasoning and inference concerning development of The Theory of Evolution. So just because this kind of science is really really hard. I would caution against abandoning it entirely.

  • The Other Dave says:

    @DSS; Actually, I was referring to DM's blog post, not your reply. Sorry for the confusion.

  • Ahh- got it. You have nonetheless touched upon a really very deep & profound philosophical issue. It is more than worth the argument.

  • The Other Dave says:

    @DSS: Agreed. That's why I like statistics. It's really the 'science' of how we precisely understand and ultimately define our world. Kind of a quantitative philosophy.

Leave a Reply