« The NFL's First Female Wide Receiver... | Main | Excellent Post on the Marriage Premium »

March 25, 2013

Tests "Measure What They Measure"

...which is not always the same thing as "measuring what we want to measure"

A growing body of evidence shows, however, that we have grossly underestimated both the scope and the scale of animal intelligence. Can an octopus use tools? Do chimpanzees have a sense of fairness? Can birds guess what others know? Do rats feel empathy for their friends? Just a few decades ago we would have answered "no" to all such questions. Now we're not so sure.

Experiments with animals have long been handicapped by our anthropocentric attitude: We often test them in ways that work fine with humans but not so well with other species. Scientists are now finally meeting animals on their own terms instead of treating them like furry (or feathery) humans, and this shift is fundamentally reshaping our understanding.

Elephants are a perfect example. For years, scientists believed them incapable of using tools. At most, an elephant might pick up a stick to scratch its itchy behind. In earlier studies, the pachyderms were offered a long stick while food was placed outside their reach to see if they would use the stick to retrieve it. This setup worked well with primates, but elephants left the stick alone. From this, researchers concluded that the elephants didn't understand the problem. It occurred to no one that perhaps we, the investigators, didn't understand the elephants.

Think about the test from the animal's perspective. Unlike the primate hand, the elephant's grasping organ is also its nose. Elephants use their trunks not only to reach food but also to sniff and touch it. With their unparalleled sense of smell, the animals know exactly what they are going for. Vision is secondary.

But as soon as an elephant picks up a stick, its nasal passages are blocked. Even when the stick is close to the food, it impedes feeling and smelling. It is like sending a blindfolded child on an Easter egg hunt.

What sort of experiment, then, would do justice to the animal's special anatomy and abilities?

On a recent visit to the National Zoo in Washington, I met with Preston Foerder and Diana Reiss of Hunter College, who showed me what Kandula, a young elephant bull, can do if the problem is presented differently. The scientists hung fruit high up above the enclosure, just out of Kandula's reach. The elephant was given several sticks and a sturdy square box.

Kandula ignored the sticks but, after a while, began kicking the box with his foot. He kicked it many times in a straight line until it was right underneath the branch. He then stood on the box with his front legs, which enabled him to reach the food with his trunk. An elephant, it turns out, can use tools—if they are the right ones.

While Kandula munched his reward, the investigators explained how they had varied the setup, making life more difficult for the elephant. They had put the box in a different section of the yard, out of view, so that when Kandula looked up at the tempting food he would need to recall the solution and walk away from his goal to fetch the tool. Apart from a few large-brained species, such as humans, apes and dolphins, not many animals will do this, but Kandula did it without hesitation, fetching the box from great distances.

Another failed experiment with elephants involved the mirror test—a classic evaluation of whether an animal recognizes its own reflection. In the early going, scientists placed a mirror on the ground outside the elephant's cage, but the mirror was (unsurprisingly) much smaller than the largest of land animals. All that the elephant could possibly see was four legs behind two layers of bars (since the mirror doubled them). When the animal received a mark on its body visible only with the assistance of the mirror, it failed to notice or touch the mark. The verdict was that the species lacked self-awareness.

But Joshua Plotnik of the Think Elephant International Foundation modified the test. He gave the elephants access to an 8-by-8-foot mirror and allowed them to feel it, smell it and look behind it. With this larger mirror, they fared much better. One Asian elephant recognized herself. Standing in front of the mirror, she repeatedly rubbed a white cross on her forehead, an action that she could only have performed by connecting her reflected image with her own body.

Same problem, different context:

Somehow we're always designing tests, disliking the results, and arguing that they don't really measure the right thing, or that the tests are OK in their way but are being used for the wrong purpose, though it's not always easy to see what the right purpose would be and how it's different. The Rhode Island students argue, for instance, that the standardized test under discussion for their school district was "explicitly not designed to be used to make decisions about individual students," which certainly would make it an odd test for the school district to have invested public money in.

Sometimes the problem really is that the test doesn't measure what it was designed to measure. This is the elephant example above. But I suspect that more often, we just don't like what the test is telling us (or we're misusing it).

Maybe what we really need to do is eliminate people from the testing process :p

Posted by Cassandra at March 25, 2013 08:25 AM

Trackback Pings

TrackBack URL for this entry:


"Maybe what we really need to do is eliminate...the testing process."

Having just finished three weeks of annual standardized testing, the VES would go along with this statement in a heartbeat.

Posted by: DL Sly at March 25, 2013 12:59 PM

The boys' test scores were all over the map during their growing up years. That's one reason I'm not terribly impressed with test scores - I'm sure they indicate something, but I'm not sure they're as accurate as we want them to be.

Posted by: Cassandra at March 25, 2013 01:33 PM

The VES always has great scores. She also works hard and gets good grades. I see these tests as more of a test of her patience and ability to perform under a more stressful environment than of what she's learned to date from her teachers.

I remember taking achievement tests in school some 30-odd years ago. They were used to determine what each student needed to improve on not as a measure of how much federal money the school did or did not receive -- and therefore, how much more they will need to achieve their *objectives*.

Posted by: DL Sly at March 25, 2013 01:42 PM

I am currently spending a good chuck of my life trying to teach my mentally-disabled middle son enough math and writing so that he can pass the GED. He has been working steadily at it for over three years. This test is serving a different function -- it is pushing him to become as educated as he can be. (For somebody with his disabilities, he does quite well already. He holds down two part-time jobs -- one of which he went out and got for himself.) And truth be told, the test allows me to feel quite useful.

Just another take.

Posted by: levi from queens at March 26, 2013 08:15 AM

I think tests do provide some useful feedback. The problem probably arises when they're used for things they were never intended to be used for, or when we place too much confidence in them.

I did a lot of self-testing before going back to college to find out where my math skills were weak and where I needed more review. I had been out of school for a decade, and didn't want to waste credits on classes I didn't need if I could avoid that by reviewing the material on my own.

Even my sons' standardized test scores (and I have every test they ever took) came in very handy when we moved to California after a year of home schooling.

When the school learned my sons had been in home school, they were ready to put them in remedial classes - that's how much they hate home schooling. I was able to pull out their standardized test scores and show that they belonged in honors and advanced placement courses.

What you're doing with your son is fantastic, Levi! Your son sounds like a fine young man.

I'm a big believer that the brain is far more malleable than we are taught to believe it is. People have accomplished the most amazing things with a little work and focus - your son is lucky to have a father who believes in him and is doing what it takes to help him master the skills he needs to make his way in life.

Posted by: Cassandra at March 26, 2013 08:51 AM

The one sensible comment I found about the Rhode Island embarrassment has since been scrubbed from the article where I found it. If I remember correctly, the NECAP representative said that the test was designed to give a fine-grained picture of the school's success at the upper levels, and therefore was deliberated curved so as to produce a high failure rate. That might make it a suitable test for district-wide policy reform but not a suitable GED-like requirement. Odd, though, that only one article mentioned this issue, in a brief paragraph at the end, and then removed the paragraph within the week. In other words, there may be good reasons why a test is a good or bad one for a particular situation, but there's next to no rational discussion happening on that topic. Instead, the default position is to assume that it's wrong to focus on academic performance in evaluating the success of academic institutions, because that might interfere in either (1) some student getting a credential that might help him get a job, or (2) some teacher or administrator getting a paycheck.

Posted by: Texan99 at March 26, 2013 10:44 AM