18 July 2010

From Standards to Assessment, Part I: Item Writing

If you remember the 70's, then you probably remember Schoolhouse Rock. Conjunction Junction, Three Is a Magic Number, and Interplanet Janet were part of my Saturday mornings as a child. But one of these more than any other has stuck out in my mind as an adult: I'm Just a Bill.



There are two main reasons this particular item continues to haunt my existence. One is simply its face value during this bureaucratic phase of my career. My job exists because of a bill that became law. Ditto for the funding used to bring my working group together. Meetings I attended last month were rank with discussions about federal bills and laws. Education is much more politicized now than when I began teaching, and not just because of my change in jobs. At all levels, public policy is having a greater impact on the classroom.

But my second reason for thinking about the journey of Bill is more metaphorical: how a standard becomes a test item. Because it is a process that typically takes place behind closed doors and among disconnected groups of people, there continues to be a lot of bad information out there about how standardized tests get built. With the advent of new national standards and continued debate about student achievement, it seems like now is a good time to lay out all the pieces. This series of posts is not meant to explore the most contentious part of standardized testing (i.e. what the results are used for)---just how a standard becomes an item. Like Bill to Law, it's a helluva journey.

It all begins with standards. It is important to have a clear target. Here is a non-example, from the Washington State science standards:


What's wrong with this standard? Look at the Performance Expectation on the right. Seriously--can you imagine asking a kid to do this? How would you score it? What we'd be asking for is something akin to this this photo from NASA showing the view of Earth from Mars:


And yet, item writers are stuck with the standards they are given. Let's assume you have something decent to work with. For the purposes of this and upcoming posts, I'm going to pull something from the draft Science Framework---not because I'm in love with them (I'm not), but because it's a common point for us to use. I'm picking one of the targets listed under Life Science 1C for grades 6 - 8: Plants use the energy from light to make sugars (food) from carbon dioxide and water. This process transforms light energy from the sun into chemical energy.

Lots of possibilities here for a test item. We could provide a selection of energy transformations (light to heat, light to chemical, chemical to kinetic, kinetic to sound...) and ask the student to select the one represented by a diagram and/or description of a relationship between sun and plant. We could choose a similar style question, but leave it open and ask students to fill in a response. Or maybe we dig deeper and ask students about what is entering the plant and what is produced. At this point, things are not much different than what a teacher might do to develop a classroom test item.

But there are other things hidden within this standard. Can an item writer use the word "photosynthesis"? Nope. In fact, when you go back and look at this strand in the standards, the term photosynthesis isn't used until grades 9 - 12. We also can't ask students where the energy transformation occurs. A classroom teacher would likely have talked about leaves (or other green parts of the plant)---maybe even chlorophyll. An item writer can't go there with this standard. It's simply about the transformation itself.

To end Part I, I'll leave you with a couple of items. Number One is weak so that we have something to chew on in later posts. Number Two will also come back to haunt us, but for different reasons. Feel free to suggest improvements or additional questions in the comments and stay tuned for Part II (and III). We've only just begun...
1. Which energy transformation results in food for a plant?
a. light to heat
b. chemical to kinetic
c. kinetic to sound
d. light to chemical

2. Plants transform light energy into ________.

No comments: