13 June 2008

A Lesson in Survey Design

The Union recently sent out a "District Climate Survey." I might not be a member, but it showed up in my box and I filled out the bubble sheet. This was a bit of a struggle because although the responses were supposed to be scaled, they didn't follow a traditional Likert format (from Strongly Agree to Strongly Disagree). Instead, the answer choices went like this:

A. Strongly Agree
B. Tend to Agree
C. Tend to Disagree
D. Strongly Disagree
E. Not applicable/Don't know/Not sure

Do you see the problem? By placing the fence-rider choice at E instead of C, they have likely skewed their results. Most people are trained to use "A" as the strongest agreement and "E" for the strongest disagreement. How many people bubbled in "C" now and then by accident---thinking that they were ambivalent and instead indicated dissatisfaction? Same thing for people who might have mistakenly marked "E" when they really needed "D" for their unhappiness. Even though I read the directions (and assume others did, too), it was still a bit of a struggle to remember this very different order across four pages of questions.

Interestingly enough, The Union did not view itself as part of the climate in which we work. There were no questions whatsoever concerning teachers' views of Union activity. Even if they have much to learn about survey structure, they may already know that they shouldn't ask questions they really don't want the answers to.


Ryan said...

I think you pretty much nailed it in the last part. I sent out the WEA survey on Terry Bergeson to my members, and I got blasted 5 ways from Sunday in the results.

Ignorance is bliss.

Hugh O'Donnell said...

The curse of being statistically sophisticated!

In advanced social studies methods I learned test construction from a genius.

In social research (after a full year of stats), I learned to construct surveys.

Your poor union hasn't a clue. But I guarantee that they won't want to be confused by the facts. :)

The Science Goddess said...

Ryan---you will have quite the adventure next year as your union president. I have no doubt about that! From all of the things you have written (including posts about your experience at the assembly in Spokane), it sounds like your union is in a healthy place. People are free to have opinions and the goal is focused on kids (unlike here). I am sure that you'll continue that.

Hugh---I may be a novice with survey design, but I'd hope that in the future I'd remember this example so it wasn't repeated!