Epistemology is the “investigation of what distinguishes justified belief from opinion.” It addresses the question of how we know something is true. If we are going to do assessment, we are going to have to have evidence. And we’re going to want that evidence to convey truth.But I bet you didn’t come here for an introductory philosophy class, which is good, because I’m not qualified to teach one. I remember a tiny bit from my philosophy classes, though. One of the things that prompts us to even have a branch of philosophy called epistemology is the observation that our senses can deceive us. I’m not sure we’d be so concerned about what counts as knowledge if we could be confident that our experience was “true,” but we can’t always. For example, I bet most of us spent some late night hours in high school or college wondering how we would know that our waking experience wasn’t actually a dream. After all, sometimes when we are dreaming we think we are having an actual experience.

I’m even more concerned with ways in which our memories can be systematically biased– that is, wrong. I used to do a pretty reliable demonstration in my personality psychology classes. I had a set of twenty hand drawn pictures. I had obtained these pictures by asking a number of graduate students to each draw two pictures of a person. One was to depict a person with large ears and large eyes, and the other was to depict a person with an open mouth and open hands reaching out. I labeled each picture with the word “paranoid” or the word “succorant” (seeking affectionate support). But here’s the key: I applied those labels randomly. A picture showing a person with large eyes and ears was absolutely equally likely to be labeled “succorant” as it was to be labeled “paranoid.”

Then I went to class and lied. I told my students that I had pictures that had been drawn by people in psychotherapy. The story went that the therapist had simply asked the client to draw a person, and that this draw-a person-test was a simple projective test that gave insight into the drawer’s personality. I showed each picture for a relatively brief time and asked the students to identify diagnostic signs for the two personality characteristics (paranoid and succorant). After I showed all 20 pictures, I asked students to write down the diagnostic signs.

I bet you won’t be surprised by the result. Students overwhelmingly said that paranoid clients drew pictures with large eyes and ears and clients with a high need for succor drew pictures with open hands and an open mouth. What they had seen provided no support for that conclusion, but you can see where it came from. This is just one small example of a large number of decision errors that flow from the fundamental way our cognitive apparatus works. In general, we rely on heuristics which are rapid, usually accurate, but systematically flawed to make decisions. That’s just the way we are built. A chapter by Tversky and Kahneman gives a good overview.

So what?

What does all of this have to do with anything? Why this excursion into epistemology and the psychology of cognitive heuristics?  UISG held a tailgate party at the IMU last fall, and I was there for nearly all of it. I left convinced that students had really enjoyed it. What convinced me of that? Well, I saw a lot of students clearly having a great time. Hmm.. but suppose I only noticed (or remembered noticing) the ones having a good time. You know, like how my students only noticed the big eyes and ears on the pictures labeled “paranoid.” How confident can I be in my assessment? That’s the problem.

To be useful, our assessment evidence is going to have to be evidence we believe. That means that casual observation, anecdote, and impressions aren’t going to cut it. What we will need is evidence that has been collected and analyzed systematically. There have to be rules about how we will get the evidence, how we will analyze it, and how we will interpret the analysis. That’s basically the outline for a research methods class (in contrast to philosophy, I actually am qualified to teach that course).

That’s where some of the mystification of assessment that I referred to last time comes from. Questionnaire design! Statistics! Null hypothesis!? Grounded theory! All of that stuff is important, but for a pretty simple reason. We have a duty to ourselves, our students, and the folks who allocate resources to ask ourselves, in the best way we can, “Could we be wrong?” For example, what, besides the fact that the program was a success, might make us believe that  a particular program was a success? Are we being fooled by concentrating on only some of the evidence? Could the outcomes we see be due to differences between participants and non-participants that existed before they ever entered the program? Might the evidence we are looking at be systematically flawed by the way we collected it?

I guess what I’m trying to say is that research methods aren’t just fetishes (though some people treat them that way). Illusions (dreams, hallucinations, casual observations) can be very compelling– and wrong. Formal research methods are practical tools for assuring ourselves that our knowledge rests on something more solid than illusion. If we intend to act on that knowledge, it better rest on a solid foundation.