The Conduct of Science is a Social Science

I often lament* the fact that Sb doesn't have a blog that focuses on the methods and practices of the social sciences. The reasons are many but I think my most general formulation is that the social sciences are the most scientific of the sciences and nobody seems to understand this.


Simmer down, simmer down.
The reason I say this is because I think that the relative successes in many other sciences, in terms of reductionist information gathering and (unbelievably artificially**) controlled experimental preparations, leads to a false impression of what the scientific process is. Repeatability of our experimental preparations, combined with close correspondence of prediction to outcome, leavened with really low variability leads to a confidence that we know the Answer.
We do not. We get to better and better (perhaps) approximations of the Answer. That is all.
The social sciences, in contrast, are almost inevitably adorned with...imprecision. Our ability to predict individual human behavior and indeed group behavior with what we think of as specificity or accuracy is often pretty bad. Our ability to conduct experiments that are highly repeatable across samples can be maddeningly poor. We can describe group tendencies but results are met with a drumbeat of anecdota which are supposed to falsify the general outcome. We can specify our experimental conditions, but then someone in the audience wants to kvetch about those conditions not being representative of his or her favored conditions. Interestingly, you see this less often in the more reductionist sciences- when does a lay person start hammering you about your buffer or crappy antibody or selected inbred rat strain?
How silly this is. I mean, we take highly inbred mouse strains, throw in a genetic modification with about major five breeding steps to get the right tissue-specific, experimentally switchable expression, toss the heterozygotes, end up with a sample of maybe 5 or 10 per experiment...and then we want to generalize to humans? This is somehow better science than, e.g. asking a bunch of treatment seeking cannabis smokers about their withdrawal symptoms? Better than studying choice behavior of political voters and applying that to subsequent political behavior of other voters? Better than applying IQ test results from one group of US schoolchildren to other US schoolchildren who don't share geography or socioeconomic status or race or whatnot?
Don't even get me started about technically inferior studies, trust me I've run across plenty of these in the non-social sciences. Plenty. Ditto picking on one specific study with no appreciation of the background of other studies in the field.
I was motivated in my musings on this topic by a recent comment which expressed the usual false-confidence attack on a social science hypothesis. In addressing a contention that dissection labs in secondary school biology classes are highly valuable, commenter becca observes:

where is the evidence that dissecting animals is "needed to give them a fully adequate biology course"? Citation please?
(note: I'm sure dissections are more useful/interesting for some students, and I'm sure computer programs are more useful/interesting for others; but I'm interested in evidence that virtual dissections per se lead to poorer educational outcomes in groups)

All science starts with observation and description. In this case, I would point anyone questioning along these lines to the comments on-blog and in the DonorsChoose comments which testify to the individual appreciation of people for their educational experiences. We can quickly get past mere anecdote and see that we are talking about enough people to constitute data. You can start whinging on about generality and self-selection if you like. That is a methods critique and I have no patience for those that use such triumphalist language to deny the data that are in front of their eyes. Similarly, you might look at the requests from teachers who want to launch dissection projects through DonorsChoose.
Then there are some rather consistent generalities about education to consider. General principles along the lines of learn by doing, hands-on, joy of discovery and the like. There are studies on some of these topics.
In short, it is probably the best-supported stance to start from the assumption that dissection is valuable and to ask what controlled studies prove pro or con. In the absence of new evidence, we are best to adopt a null hypothesis based on the extant evidence. Skepticism in science does not mean, "everything that doesn't accord with my personal anecdotal viewpoint is bullshit until and unless there is overwhelming evidence with hundreds of flawless studies to show otherwise". This is not skepticism, this is anti-science. It may be a narrowly targeted area of anti-science being expressed by an otherwise entirely-scientific mind, but it is still anti-science.
Skeptical science tests alternate hypotheses, yes. Skeptical science is prepared to shift and nuance viewpoint based on new evidence or even good rationale. These are essentials. But skeptical science is not the knee-jerk rejection of current evidence just because you managed to think up a credible alternative hypothesis.
This example brings us to another point which is that the discussion of interesting topics has a way of getting derailed because of a lack of agreement in advance about outcome. In this case, what is the desired end-result of high school biology dissection labs? Is it to turn someone on to science in general? Prepare a future biologist who will be involved with anatomy for reality? Start screening surgeons early? Or merely to generate sticky memories in those who will never go on to additional biology education or vocation? Your conception of desired outcome is critical to balancing ethical implications of supporting dissection projects for school children.
Not only that, but it impacts the chances of ever getting the overwhelming high-N proof about functional impact of a 1-2 day high school lab. There are not that many surgeons in the population. They are highly selected and highly trained individuals- getting absolute proof about influences on their motivations for becoming surgeons is going to be hard to come by. Anecdotes may be all we get. The degree to which various biology lessons stick with a non-biology person lifelong? Probably easier to come by. Either way, however, a lack of evidence is not proof of an alternative..it is just a lack of evidence.
Science means going with the available evidence for now, while trying to figure out how to get more and better evidence in the future.


I googled up a couple of blogs that look interesting to me, might have a post now and again that is exactly what I am looking for- approachable descriptions of how the social science process works. If you have any favorite blogs that communicate these ideas drop me a line, would you?


__
*because I don't have the chops to address this stuff properly. I just have enough grasp to understand my own ignorance and to appreciate what the social sciences can do. but this is bloggin' so what the heck...
**in some cases.

22 responses so far

  • yolio says:

    Well, this isn't quite what you asked for, but Andrew Gelman writes a very nice blog. He is a well known Bayesian statistician that works primarily on social science questions. He deals sometimes with nuts and bolts methodological stuff. He also deals a lot with the issue of evidence, being a statistician and all. Anyway, he would be able to point you to some good social science blogs.
    http://www.stat.columbia.edu/~gelman/blog/

  • DSKS says:

    I reckon many life scientists would have something to gain in getting their clock cleaned at least once by a statistician from the social sciences.

  • I often lament* the fact that Sb doesn't have a blog that focuses on the methods and practices of the social sciences.

    I lament this, too! If Sb wants to adopt me, I'd be happy, as a social scientist, to focus more on this than I do at present.
    šŸ™‚

    The reasons are many but I think my most general formulation is that the social sciences are the most scientific of the sciences and nobody seems to understand this.

    You make some interesting points about generalizability, but I think you'd have to be more specific about your definition of "scientific."

    Not only that, but it impacts the chances of ever getting the overwhelming high-N proof about functional impact of a 1-2 day high school lab.

    Perhaps, but social scientists can design studies to properly measure and assess the outcome(s) considered relevant, controlling for other variables. It's feasible. We don't need to rely on collections of anecdotes. While my personal sense is that dissection is a worthwhile exercise, it's just that - a personal sense* - and (without having read the preliminary discussion) becca's question sounds to me reasonable.

    In short, it is probably the best-supported stance to start from the assumption that dissection is valuable

    I can't go along with that, necessarily - not when we're talking about time and money being directed towards one program or another. Even if it is valuable, there may be other, more valuable projects in which to invest those. If people are being asked to make choices, they should have some empirical basis for them. And ethical considerations can't be discounted, of course.
    *From an admittedly Kropotkinian pedagogical approach.
    (Gah. Can't believe I'm commenting on TizzyoProf's blog.)

  • DrugMonkey says:

    Perhaps, but social scientists can design studies to properly measure and assess the outcome(s) considered relevant, controlling for other variables. It's feasible.
    Things that are conceptually possible are not always feasible. I stand by my assertion that the chances of getting a decent study of the impact of 10th grade dissection lab on the eventual creation of a surgeon are lower than getting a decent look at the long term retention of dissection lab experiences versus other biology class concepts in the general population.

  • Things that are conceptually possible are not always feasible.

    True.

    I stand by my assertion that the chances of getting a decent study of the impact of 10th grade dissection lab on the eventual creation of a surgeon are lower than getting a decent look at the long term retention of dissection lab experiences versus other biology class concepts in the general population.

    I'm a bit confused by the latter part of this sentence. With regard to the first part, I've only skimmed the thread you linked to (and have zero desire to involve myself in any PETA discussions), but if the positive claim people are making is that we need more kids (interested in) becoming surgeons and physical dissections play an important role in leading to that outcome, then that is a claim that they would have to try to support - and I agree it would be extremely difficult to do, especially when faced with alternative possible means of achieving this same outcome; they would also have to defend their choice of outcome as the central one to consider.
    It does seem like you're trying to have it both ways: defending the social sciences in one breath and then suggesting in the next that they have little or nothing to offer concerning the question under consideration.
    I have to get to work, but will return later.

  • DrugMonkey says:

    My point is that regardless of desired outcome, it is difficult to have a discussion in which everyone is thinking about different desired outcomes. It helps to be specific.
    I am further asserting that some outcomes are going to pose greater difficulties in generating reasonably acceptable data. The numbers of people who did dissection and went on to non-medical, non-science careers is huge compared to those who went into the sciences or medicine. So prospective, retrospective or cross-sectional study is going to be more difficult if the potential pool of subjects is smaller.
    To insist that you want to see a study that you know has little chance of ever being conducted before you will accept conclusions based on the existing, though less perfect info, well that starts down the anti-science dark side...
    I am arguing that even the less-than-perfect social science study tells us something. Just as less-than-perfect studies in all other areas of science tell us something.

  • It does seem like you're trying to have it both ways: defending the social sciences in one breath and then suggesting in the next that they have little or nothing to offer concerning the question under consideration.

    DM is a master.

  • becca says:

    "Similarly, you might look at the requests from teachers who want to launch dissection projects through DonorsChoose."
    I agree. Let's go for some quotes:

    "I believe that if the students are able to complete a dissection, they will gain a better understanding of the internal anatomy of living things. Therefore, helping them on their science graduation test.
    By helping my students to achieve their goals of passing the science graduation test, you quite possibly could be saving a life. By graduating, students are able to pursue things in life that they otherwise would not have the opportunity to do. This may help a student from homelessness, life on the streets, lives of crime, etc. You will be giving these students a chance at success...something many have never felt."

    I'm cool enough with the first part, but the second part leads me to believe the teacher was more interested in tugging on heartstrings (most effectively, might I add) than in presenting even anecdata rather than her personal beliefs. This witness is suspect. Still, a teacher's beliefs are worth considering.

    "I am a disabled teacher (wheelchair) who finds it very difficult to dissect in my honors biology class with 40+ students in each class.
    Due to my disability and lack of a classroom aide, my students often miss out on dissecting multiple specimens during the school year.
    We have the computers, if we we just had this virtual dissection software the students would gain so much more valuable knowledge. "

    Damn. I am totally giving him money. He totally tugged on my heartstrings without insulting my intelligence in his claims.

    "If they could dissect rats, we could learn about several different organs from one dissection. We will also be dissecting owl pellets and studying how rodents fit into the food chain. We'll examine structural adaptations that enable rats to avoid predators.
    It would be an incredibly memorable experience for the students."

    This teacher is making a claim I can believe. I, for example, still remember vividly how disgusting owl pellets are (and how they smelled).
    Even my mother (after umpteen million years) can tell me what she remembers from her high school dissection (how incredibly tough froggyflesh can be)
    And I'll never forget my first day in the diabetic retinopathy lab when I witnessed the harvesting of the retinas from rats after decapitation. I cried too much.

    "At least once a week one of my students will ask, "When will we get to do a dissection?" I respond to them that we will conduct our dissection lab at the end of the school year. However, due to a severe school district budget deficit, my students may not be able to benefit from this crucial exploratory experience. The cost of dissection specimens is the highest line item expense in our high school science supply budget but serves only a small fraction of our student population. In times of severe budget deficits, items that benefit a small percentage of students are usually the first to be cut. The fetal pig dissection lab is one lab experience that differentiates the honors biology curriculum from the level biology curriculum. It would be extremely disappointing to my students if they were forced to forgo this crucial mammalian dissection."

    Wow. I wonder what Janet would think ("It angers me that we'd treat kids in poorer districts as less worthy of the materials needed to give them a fully adequate biology course.") Is it ethical to treat the DUMB kids as less worthy of materials "needed to give them a fully adequate biology course"? Is DonorsChoose really about 'save the best and bury the rest'?
    Still, the worthiness of the project aside, this is excellent evidence some students are excited about doing dissections.
    Item #330,270,298 in the list of demonstrations that two people can look at the same data and come away with totally different conclusions... DM, you'd best get it through your head that this phenomenon is not always "DM = Right!!!11, other person = ZOMGANTISCIENCEevilz"
    Nobody (other than PETA, which for these purposes is a strawman, unless they start commenting) has claimed dissection is worthless, boring, or forgetable for all students. Dissection having benefits doesn't necessarily mean it's worth the costs.
    "Not only that, but it impacts the chances of ever getting the overwhelming high-N proof about functional impact of a 1-2 day high school lab."
    So for the long-term career outcomes you mentioned, this is usually true (research studies spanning decades are pretty much always too expensive to get high-Ns), but remember, you brought up such outcomes.
    It seems to me you can get a measurable difference in student motivation with a properly designed educational intervention lasting 1-2 days.
    Doest thou regularly read social science research discussed on SB? A 15 minute intervention can be meaningful (http://scienceblogs.com/notrocketscience/2009/04/simple_writing_exercise_helps_break_vicious_cycle_that_holds.php)
    "My point is that regardless of desired outcome, it is difficult to have a discussion in which everyone is thinking about different desired outcomes. It helps to be specific."
    This is exactly true, which is why I'm sure everyone is aware that my comment was not addressing career outcomes in students exposed to dissection (although that is an interesting topic in it's own right). I'm gonna chalk that up to "it's not about me"/"bloggers right to use a comment as a jumping off point for another discussion, that just sounds like it might be attempting to refute the comment"*.
    So here goes, DM, let's get specific:
    Set up a crossover design trial- take two roughly equivalent classes in one school, give them a test to measure anatomy knowledge and attitude toward learning anatomy and biology in general. Then give one class the virtual dissection** , and the other the real dissection. Repeat test. Then give each group of students the other method, and repeat test again. Repeat test at a later follow up time (6 months or more?). Ideally, you'd follow up with the students much later to examine career paths, but that would increase the cost of the study
    This experiment could be designed to help address a number of hypotheses:
    H0) Real dissection, but not virtual dissection, increases the number of students achieving a "passing" grade on a biology test (This is Janet's hypothesis, if the biology test is designed to be a benchmark for an "adequate" education; the design of such a test would, of course, open up a whole 'nother can o' worms, but you could at least start with some state standards)
    H1) Real dissection increases performance on a test of anatomical knowledge
    H2) Real dissection increases enthusiasm for studying anatomy, biology, or science as a whole
    H3) Real dissection influences career aspirations toward science, medicine, or health related fields.
    H4) Real dissection causes more distress to students than virtual dissection
    H5) Virtual dissection followed by real dissection is the most effective course of study with respect to anatomical knowledge (I'd just throw that in because if real schooldistricts were looking at the data, they might want to know how much you gain by the repetition in different forms)
    H6) Real dissection results in better retention of anatomical knowledge
    If you could do the study in a bunch of different kinds of classrooms, it would, of course, be immensely more valuable (and expensive).
    "General principles along the lines of learn by doing, hands-on, joy of discovery and the like. There are studies on some of these topics. " (emphasis added)
    Personally, I agree on the joy of discovery angle... but I dispute that dissection is necessarily a "joyous discovery process" for most students.
    The hypothesis I really want to test is:
    H0: People who are uncomfortable with dissections decide (or are told) biology is not for them.***
    If true, I consider the status quo to be nearly as bad as if this is true:
    H00: People who are uncomfortable using numbers decide (or are told) science (but especially chemistry and physics) is not for them.
    TLDR version: I was not suggesting a hypothesis about 10th grade science lab dissections and creation of future surgeons, that would be relatively difficult (or at least, expensive) to study in a controlled fashion. Instead, I think a number of other interesting hypotheses relating to the utility of dissections (including one suggested by Janet's claim) could be tested via a more controlled method than anecdotes
    And as always, DM, it helps also to look at the literature**** (these are somewhat germane.: Hug, Barbara "Re-Examining the Practice of Dissection: What Does It Teach?" Journal of Curriculum Studies, v40 n1 p91-105 Feb 2008; Donaldson, Lynda, Downie, Roger Attitudes to the Uses of Animals in Higher Education: Has Anything Changed? Bioscience Education e-Journal, v10 Article 6 Dec 2007; Kariuki, Patrick, Paulson, Ronda "The Effects of Computer Animated Dissection versus Preserved Animal Dissection on the Student Achievement in a High School Biology Class" Paper presented at the Annual Meeting of the Mid-South Educational Research Association, Nov 14-16 2001 [http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/19/99/f7.pdf])
    *Somebody has got to tell Dr. Isis's commenters about this rule.
    **NB: I envision the virtual dissection condition as spending an equivalent amount of time as an actual dissection would take watching a video of real dissections and using an interactive computer program to do a dissection- this is the sort of opportunity that was available free, on the web, at least ~ a decade ago. I thought it was HHMI virtual frog dissection, but it doesn't appear to be on their website anymore. They do have a neurophysiology virtual lab, anyone who actually does that want to check it out?)
    ***Anyone wanna suggest a good way to examine this?
    ****An aside: most of the articles that come up on ERIC for animal dissection seem to describe dreadfully dull labs. Counting flower petals? That's right up there with rolling cylinders down inclines in college physics 101.
    (I know, I know, I need to GMOMFB)

  • Funky Fresh says:

    SaltyDouche!!!!

  • KBHC says:

    You have a number of anthropology blogs on Sb, and anthropology is considered a social science. Greg Laden's "Falsehood" series is a pretty good example of social science at work.
    Aside from some anthro, I do agree with you that there is a dearth of soc sci blogs here and I appreciated the points you made.

  • DrugMonkey says:

    I really enjoyed Laden's Falsehoods and think that idea would make a good print book...

  • My point is that regardless of desired outcome, it is difficult to have a discussion in which everyone is thinking about different desired outcomes. It helps to be specific.

    OK, I've now read the other thread and comments. I'm not sure where this business about surgeons is coming from (in addition to its not having been established as an outcome of overriding importance, which would require an argument). You quoted becca's comment, which itself quoted JS's post:
    where is the evidence that dissecting animals is "needed to give them a fully adequate biology course"? Citation please?
    The outcome in question was fairly clear, and she then went on to mention statistical "educational outcomes." This would all be completely feasible to study, in a way that compared physical dissections with models and computer/video simulations. I don't see the problem.
    (I watched State of Play the other day, and it contained a similar sort of sentiment: When referring to the "big story," the character who's also a blogger says something like "I think people should have newsprint on their fingers when they read this." Why on earth? ...OK, maybe I'm reaching, but it seemed somewhat analogous...)
    I'm also not seeing this overwhelming number of people coming forward in that thread to attest to the importance of dissection in their scientific formation. A couple of people mentioned that it was educational, others said it wasn't so much for them, and someone who works on computer models spoke up to challenge some of the claims being made. Even if a larger number had spoken about their personal experience, it wouldn't have constituted solid data. Are you saying the social sciences are the most scientific because we accept anecdotes and ignore mere "methodological critiques"? That simply isn't so (doesn't make sense, either), and the response to a simple request for a citation from the literature to support a claim is hardly a demand for "overwhelming evidence with hundreds of flawless studies."

    I am further asserting that some outcomes are going to pose greater difficulties in generating reasonably acceptable data.

    Aside from the fact that you didn't ask becca to elaborate on the data she would consider acceptable, she was speaking of specific outcomes which are amenable to analysis, so the rest of what you say doesn't seem particularly relevant.

    I am arguing that even the less-than-perfect social science study tells us something.

    Which study? I'm joining this late. Have I missed something?

    SaltyDouche!!!!

    Awwwww. Thanks for trying to drive up my traffic. šŸ˜‰

  • Have I missed something?

    Yeah. WHOOSH!

  • and the response to a simple request for a citation from the literature to support a claim
    Better.

    Yeah. WHOOSH!

    Could you be more specific? Which study or studies were people talking about?

  • and the response to a simple request for a citation from the literature to support a claim
    Oh, FFS. No strikethrough?
    and a simple request for a citation from the literature to support a claim
    There.

  • DrugMonkey says:

    I just rescued becca's comment from the filter, go read it folks.
    I'm gonna chalk that up to "it's not about me"/"bloggers right to use a comment as a jumping off point for another discussion
    This is always the best policy when it comes to this particular blogger (Hi Salty!).

  • Could you be more specific? Which study or studies were people talking about?

    AHAHAHAHAHAHHAAHAHAHAHAHAH!!!!!!!!!!!

  • H0) Real dissection, but not virtual dissection, increases the number of students achieving a "passing" grade on a biology test (This is Janet's hypothesis, if the biology test is designed to be a benchmark for an "adequate" education; the design of such a test would, of course, open up a whole 'nother can o' worms, but you could at least start with some state standards)

    I'll confess that the sense of "adequate" I had in mind here was more expansive than just passing the damn state science standards tests. I suspect a lot of people who teach are shooting for something with a bigger impact, of the sort suggested in Becca's other hypotheses.
    Also, the test population of students will be fairly important here, owing to the impact of different learning styles. Especially for learners who get things better from tactile interactions, not just visual ones, we might end up seeing a big difference in how well the learning sticks (plus how inspired the kids are to learn more, etc.).
    Having more concrete data here can only be a good thing. But in the absence of the sorts of studies we're talking about, ought we to trust the impressions of teachers (who know something about the outcomes they're trying to achieve and the experience they've had trying to achieve them with and without various kinds of resources), or the organization whose mission commits it to arguing against a whole class of resources regardless of their actually efficacy or the efficacy of the alternatives in achieving the teachers' pedagogical goals?

  • becca says:

    "ought we to trust the impressions of teachers..."
    Who we, kemosabe?
    You can find a ready guide in some instructor's voice. If you choose not to research, you still have made a choice.*
    As I mentioned, in searching ERIC with an attempt at an open mind (which is why, I presume, your comment on "in the absence of the sorts of studies we're talking about" is a pure hypothetical?), I came across a description of a lesson plan involving flower petal counting.
    Teachers can say that their impression of counting flower petals is that it will lead to success in life for those who never had a chance at it otherwise. They likely have many students who are more excited about counting flower petals than reading about flowers (likely related to learning styles). I am not one iota convinced that counting flower petals is interesting, nor that reading is inferior to counting.
    *with profound apologies to Rush

  • Can we really trust the findings of social science research?
    There's no way one can perform 'lab experiments' on people
    to study human behavior. Moreover, social science is not influenced
    by a few factors like scientific research but may be influenced
    by external and internal factors, some we might not recognize
    or anticipated. What's the point in conducting social science
    research if we can never trust 100% the results of the research?

  • D. C. Sessions says:

    What's the point in conducting social science research if we can never trust 100% the results of the research?

    Please name one field of scientific research where we can trust 100% of the results.
    Know then thyself.
    Presume not God to scan;
    the proper study of mankind is Man.

    and
    If it is not given to you to complete the task,
    neither are you free to refrain from it.

  • Mishal says:

    I would also add, that for the same skeptical reasons, Psychology also gets the same short end of the stick that the other social sciences does.
    Humans are messy, and their thinking is messy and so biased-loaded it's a wonder there's any kind of consensus on anything. But still, you tell a layperson or someone in the "harder" sciences that you're a Psychologist or in that field and you get the same, "So, I bet you can tell me what I'm thinking right now." Hardy-har-har, nudge-nudge crap.
    Strangely enough, Ethology doesn't get that kind of flak.

Leave a Reply