Intuitive Test Theory

Many of us have an intuitive understanding of physics that works surprisingly well to guide everyday action, but we would not attempt to send a rocket to the moon with it. Unfortunately, Mr. Braun and Mr. Mislevy argue, our policy makers are not as cautious when it comes to basing our school accountability system on intuitive test theory. ALONG WITH making sure that our bodily needs are met, one of our first tasks upon entering this world is to try to make sense of it. We do so by continuous observation and generalization, as well as by absorbing the norms of the culture in which we find ourselves. Our understandings typically take the form of stories -- narratives, as the psychologist Jerome Bruner has called them. These stories are attempts to identify why people do what they do -- their beliefs, motives, and plans. This mode of developing and retaining understanding carries over to the physical world, whether natural or human-made. We hear thunder and see lightning, see objects being thrown and falling to the ground, observe cars and computers working (or not), and we construct stories about causes, patterns, and linkages. Now, we make up these stories whether or not we truly understand what is going on. Adults are driven, in exactly the same way as 5-year-olds are, to express their understanding of what is happening around them in terms of narratives. As Howard Gardner has pointed out, stories can differ, often substantially. In most domains of knowledge, we develop very powerful theories when we are very young. . . . No one has to tell a kid that heavy objects fall more quickly than light objects. It's totally intuitive. It happens to be wrong. Galileo showed that it was wrong. Newton explained why it was wrong. But, like others with a robust 5-year-old mind, I still believe heavier objects fall more quickly than lighter objects. The only people on whom these engravings change are experts. Experts are people who actually think about the world in more sophisticated and different kinds of ways. . . . In your area of expertise, you don't think about what you do as you would when you were five years of age. But I venture to say that if I get to questioning you about something that you are not an expert in, the answers you give will be the answers you would have given before you had gone to school.1 Richard Feynman's story for what happens when we throw a rock might be based on the principle of the path of least action and admit to a rigorous rendering in differential calculus, whereas little Jimmy's story is that the rock wanted to get back down to the ground where it belongs. The point is that people construct plausible stories for actions and events based on what they've experienced themselves and on what they've picked up, however loosely or informally, from the culture around them. The Gardner quote highlights two other aspects of these narratives. The first is their tendency to persist, even in the face of evidence to the contrary or confrontation with methods of analysis that are much more powerful. Bruner makes the same point with respect to what he calls psychology. He defines folk psychology as a system by which people organize their experience in, knowledge about, and transactions with the social world. We learn our culture's folk psychology along with its language and norms of social behavior. Bruner asserts, Folk psychology changes but is not displaced by scientific psychology.2 It is the persistence of these narratives (say, in physics) that can be so frustrating for teachers. The second aspect of these stories is that expertise is often very narrowly focused. That is, outside one's area of specialized training, it is uncommon to do much better than a 5-year-old. Indeed, the situation may be even more dire. In a now classic study, the psychologists Amos Tversky and Daniel Kahneman questioned a large number of research psychologists on various aspects of probability and statistics (the design of experiments and the interpretation of the results) that would ordinarily be relevant to their work. …