August 17, 2010
WEIRD Sampling Bias Skews Social Science Research
Via a great article from Jonathan Haidt that I'll be commenting more upon later, yet another reason to view the pronouncements of "experts" with a bit of skepticism:
Who are the people studied in behavioral science research? A recent analysis of the top journals in six sub‐disciplines of Psychology from 2003‐2007 revealed that 68% of subjects came from the US, and a full 96% of subjects were from Western industrialized countries, specifically North America, Europe, Australia, and Israel (Arnett 2008). The make‐up of these samples appears to largely reflect the country of residence of the authors, as 73% of first authors were at American universities, and 99% were at universities in Western countries. This means that 96% of psychological samples come from countries with only 12% of the world’s population. Put another way, a randomly selected American is 300 times more likely to be a research participant in a study in one of these journals than is a randomly selected person from outside of the West.
Even within the West, however, the typical sampling method for psychological studies is far from representative. In the Journal of Personality and Social Psychology, the premier journal in social psychology—the sub‐discipline of psychology that should (arguably) be the most attentive to questions about the subjects’ backgrounds—67% of the American samples (and 80% of the samples from other countries) were composed solely of undergraduates in psychology courses (Arnett 2008). Furthermore, this tendency to rely on undergraduates as samples has not decreased over time (Peterson 2001, Wintre et al. 2001). Such studies are thus sampling from a rather limited subpopulation within each country.
Why is this a problem?
Behavioral scientists routinely publish broad claims about human psychology, cognition, and behavior in the world’s top journals based on samples drawn entirely from highly educated segments of Western societies. Researchers—often implicitly—assume that either there is little variation across human populations, or that these “standard subjects” are as representative of the species as any other. Are these assumptions justified? Here, our review of the comparative database from across the behavioral sciences suggests both that there is substantial variability in experimental results across populations and that standard subjects are particularly unusual compared with the rest of the species—frequent outliers. The domains reviewed include visual perception, fairness, cooperation, spatial reasoning, categorization and inferential induction, moral reasoning, reasoning styles, self‐concepts and related motivations, and the heritability of IQ. The comparative findings suggest that members of Western, educated, industrialized, rich, and democratic societies, [Ed. note: WEIRD] including young children, are among the least representative populations one could find for generalizing about humans. ... we need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity.
The road to Hell is paved with unquestioned assumptions.
I know I've mentioned this before, but as an undergrad I tutored graduate level Education and Psych students in probability and statistics. I was routinely appalled at their complete inability to grasp (and disinterest in learning to avoid) the most basic sampling errors.
We heard a lot during the last administration about how government should base sweeping public policy decisions on the latest research from the social sciences. Skeptics who dared to question The Pronouncements of Science were demonized. How dare these superstitious, ignorant, inbred, snake handling freaks challenge their intellectual superiors?
When I read articles like this, I want to slap one of those "Question Authority" bumper stickers on the forehead of the closest expert I can find. A little humility from these folks would be a refreshing change.
Posted by Cassandra at August 17, 2010 12:54 PM
TrackBack URL for this entry:
My completely unscientific and anecdotal study on psych majors also confirms that the overwhelming majority of psych majors exhibit symptoms of mental illness.
aka, they're bat-(dookie)-crazy. And one of my best friends is finishing up his doctorate. He agrees with me.
Posted by: MikeD at August 17, 2010 01:31 PM
Posted by: BatShit Crazy Psych Student at August 17, 2010 01:33 PM
You're assuming that they can read such large words.
And that they'll look close enough to your face to notice the sticker.
We're raising a bunch of subservient ignorant fools.
signed, disgruntled former college math tutor.
Posted by: htom at August 17, 2010 02:29 PM
The term "Social Sciences" is an oxymoron.
An Army shrink lived with us for a month in Vietnam. His report declared that none of us were sane in any accepted sense of the term, but that we were functioning perfectly in our environment.
In other words, we were doing what any sane person would do to survive.
Now -- in what world is declaring someone simultaneously sane *and* insane a valid scientific finding?
Posted by: BillT at August 17, 2010 02:35 PM
In a world where Kinsey is still believed to have been an unbiased researcher, does something like this actually surprise anyone?
Posted by: ECM at August 17, 2010 05:02 PM
Well done; I look forward to your further comments.
In college I did a research study looking at a specific brain lesion in mice. It had been well studied and had predictable effects. In my study, (which I was only able to do for fairly complicated reasons of my own) I looked at two distinct genetic strains of mice and raised half of each under disparate conditions. I found the "predictable" results were only predictable for the breed typically used in the experiments who were raised in typical lab conditions. Either with different genetics or under non-standard conditions, the results varied. Go figure, genetics and early experience matter.
Posted by: ShrinkWrapped at August 17, 2010 08:39 PM
I have a confession to make: I would have made a lousy scientist :p
Science was the only subject I ever struggled with in school. I got good grades but boy did I have to work hard for them.
Math wasn't difficult for me and English, languages and literature were a walk in the park. But science! All I can say is thank God for a good memory and lab partners who were more patient than I!
Posted by: Cassandra at August 17, 2010 10:28 PM
The fundamental error, of course is considering "social science" to be an actual, you know, science.
Posted by: scottkellyfa711 at August 18, 2010 08:37 AM
Social science is more like social engineering. The theoretical aspects are far less important than what "works". If you can destabilize a government and put all the dissidents into furnaces, does it really matter how good your economic theories are?
Posted by: Ymarsakar at August 18, 2010 09:13 AM
Or even the quality of your gender based studies program. None of that matters, for power is accrued from action, not sitting around reading.
Posted by: Ymarsakar at August 18, 2010 02:55 PM
BillT -- sounds like your shrink (re)discovered the premise behind "Catch-22".
Any field of study which is forced to append "Science" to its name, isn't.
Exhibit B: Computer Science.
Posted by: Grumpy Old Ham at August 18, 2010 06:42 PM