19 July 2010

From Standards to Assessment, Part II: Item Development

When we last saw our heros, Items One and Two, they were newly birthed and in need of some parenting. As a classroom teacher writing a test, this would likely be the end of the road for these items---a little wordsmithing and then it's off to the xerox. But standardized test items have a far different fate.

Let's get back to Item One:
1. Which energy transformation results in food for a plant?
a.  light to heat
b. chemical to kinetic
c. kinetic to sound
d. light to chemical

It needs some guidance. If we're going to leave it in this sort of format, the answer choices have to be reworked. We have one pair that starts with "light," so we need another pair. Why? Because when a student sees two answers that begin the same, there is an automatic assumption that the answer must be one of them. We've taken what should be a knowledge-based question and made it answerable by using a basic test-taking strategy. So, let's rewrite the answer choice currently in position "c," then put the pairs together. The item stem could also be wordsmithed a bit. How about Which energy transformation creates food for a plant? Let's keep it simple.
1. Which energy transformation creates food for a plant?
a. light to heat
b. light to chemical
c. chemical to kinetic
d. chemical to light

Much better. As items go, this one has a low cognitive demand. Most states use Webb's Depth of Knowledge to categorize items---not Bloom's Taxonomy. Why? Because Webb's version is meant to distinguish among tasks. Bloom did not design his system as a hierarchy---it's not meant to be used to look at difficulty and rigor. So, as we look at the Item One, we would categorize it as having a cognitive complexity equivalent to Level One. The kid could get by answering this using basic recall. Could we increase the cognitive demand? Sure. Just flip around the construction---ask Where could the transformation of light to chemical energy occur? and then carefully pick our answer choices. In that case, a student has to know the transformations which could happen for each of the selections as well as what happens in a plant, then put the information together. However, for the purposes of these posts, we'll stick with what we see above.

Let's not forget Item Two:
2. Plants transform light energy into ________.

An item writer has the responsibility to list as many terms as possible that a student might put in the blank which could be correct, such as "sugar" or "food."

At this point, the items enter a deeper development process. Other item writers will provide feedback. Content specialists will fuss with it. The items will be pooled together with other items and go to a committee which will review them again for content and structure. An item can "die" at any point along the way. Perhaps there are already plenty of items for that standard. Maybe some items are better constructed. It could be that the cognitive demand doesn't suit the needs of the test bank. Sometimes, there is a turd that you can't polish, no matter how hard you try. Just like how a Bill has a journey to become a Law, our lowly items have a number of events to navigate.

Assuming that these two items make it through the weeks of revision and review, they go into a provisional test bank and await their fate in a pilot process. More on that in Part III.


doyle said...

Dear The Science Goddess,

I appreciate the hard work and subtle thought that goes into creating the questions, but I want to scream, scream, scream!!!

The standard plays with energy and matter in a way that can only confuse children, never mind their teachers:

Plants use the energy from light to make sugars (food) from carbon dioxide and water. This process transforms light energy from the sun into chemical energy.

Not sure what "use the energy from light" even means; plants use light energy, not light's energy, perhaps too subtle a distinction for the age group, but if it is, then don't teach them anything about it except that the stuff that comes out when they breathe (CO2 and water) gets "smushed" into food. Children get "smushed"--smushing requires energy input.

No way a 4th grader gets chemical energy, but she might get this--the light and heat you get when you light leaves on fire came from the sun.

And to be super-picky, and a pain in the butt, at least 2 or 3 of my brighter students would point out that mechanical and kinetic energy are crucial in-between steps when the proton gradient drives ATP-synthase to make ATP--the third phosphate is literally smushed onto ADP.

(I even share the story of the scientist, a sheep farmer, ostracized for suggesting that ATP was indeed made mechanically--turns out he was right.)

I realize I'm a nitpickin' dawg here, but that's how I want my lambs to think--science gets down to models and language

We start working on that the moment we start speech, actually before that. Maybe even in utero when we hear rhythms.

And then someone will say "get real--the reality is..." and I start screaming again. The reality is that we have a tenuous grasp on reality, and that science class is the avenue to show kids just how tenuous (and wonderful) our perception(s) of the universe are.

I'm rambling. One more question before I go:

Do you believe that private for-profit companies develop questions as carefully as they should?

The Science Goddess said...

I agree with you that the standard could use some wordsmithing---and would suggest that you make comments known to the NRC by August 2.

Keep in mind that the standard I'm pulling from here is for grades 6 - 8. What elementary and high school need to be able to know and do is different---less depth for lower grades and more depth (including ADP, et al.)for high school. So, you're right on target with your observations that the questions and answers would look very different for those grades.

But keep nitpicking. There's a lot to sort out here.

Do I believe private companies develop questions as carefully as they should? I would say that when it comes to the sorts of questions included with curriculum materials: no. When it comes to large-scale testing: yes. Mind you, most of my experience has to do with non-profit ETS. Lots of science teachers and science enthusiasts who have moved into a different phase for their career and are very concerned with being fair to kids.

Dr Pezz said...

When does the state plan to report the 2009-2010 scores for schools and districts?

I always feel like the scores come out so late that we can't really do anything with them.

The Science Goddess said...

High school math, reading, and writing was officially released first week of June. All other data for other tests/grades should be in districts by August 1 at the latest; official release will be in September, but your school district can hand out scores at any time.

State timelines are actually much tighter than they once were. We originally tested in April and got scores in October/November. Now, we test up until late May (for 3 - 8) and get results in 2 months.

I would suggest talking with your DAC if you're not getting your data in a timely manner.

doyle said...

Dear The Science Goddess,

Nit-picking again, but here's the source of my frustration--and I (perhaps too vigorously) broach this with my colleagues (perhaps too frequently).

Our attempt to do the less depth/more depth in ideas like photosynthesis result in a lot of misunderstanding at the lower grade levels.

If I were king of the curriculum world, 4th-6th graders would repeat van Helmont's experiment in class over the school year(s)--grow a sapling for 3 years, and see how much of its weight comes from the soil--essentially none (as you know BUT most adults do not, even adults that can parrot the PS equation).

Kids got to feel science in their bones or else it's just pointless. Do the van Helmont experiment, and their vision changes. Trees are made of air and water--imagine that!

The Science Goddess said...

The "how" in terms of what gets us from Point A to Point B seems to be the source of most education-related blog posts. Some will agree with you that kids need to feel science in their bones...some will tell you that science which is packaged in a box (i.e. science kit) is good enough...others will say that digital content is better. I'd love to see kids repeat van Helmont's work, but I also know that there will be students who need something other than that to make a connection. Instruction is not one of those things where there is going to be a right---or, at least, a one-size-fits-all---answer. This is what I love about teaching: the creativity involved in helping students learn.

doyle said...

Dear The Science Goddess,

I get that there is no single way to teach science, but if a child cannot internalize that trees do not eat soil, then we're not teaching science, we're teaching mythology.

As far as the van Helmont experiment, the beauty of it is that there is nothing to get except that the tree does not consume soil, and that the tree got bigger anyway. If we tell them its because CO2 and water were joined blah blah blah we're back to mythology (at least in elementary school).

I'd rather not tell them anything.

Roger Sweeny said...


As the Helmont experiment proceeds, any reasonably curious child is going to ask, "If the tree doesn't eat the soil, where DOES the wood come from?" At this point, you're under an obligation to tell them.

(When I say "tell," I don't necessarily mean a lecture. "Have we been watering the tree?' "What do you think happens to all that water?" ...)

doyle said...

Dear Roger,

It's the misplaced "obligation to tell them" that ruins science education.

I might ask a child to breathe out on her hand and ask her what comes out ("air," moisture, and heat--essentially the left side of the photosynthesis reaction: CO2 + H2O + energy).

If I just say water, well, that may add to the confusion. The stuff in trees is not water per se but the proton stripped from water.

I would agree that we need to feed the curiosity, though, and that we need to lead her somewhere. I'd rather talk about how things cannot be made from nothing, and work from there, maybe wander over to the concept of gases, maybe compare the weights of inflated vs. deflated basketballs. Hard to know which way we'd go because it depends on how the child responds to the initial ideas.

To tell her "water" circumvents the whole point of the exercise.

Roger Sweeny said...


I think we may be "in violent agreement."

I would not tell students that "wood comes from water."

I would try to get them to realize that plants take in water (maybe some of them have houseplants that droop if they're not watered or see lawns that turn brown after it hasn't rained for a while).

Then, what happens to the water? Yes, the idea that "things cannot be made from nothing." "You breathe; plants kind of do too." Gases. I like your idea of breathing on the hand.

Finally, the "answer" that plants take water and something from the air and chemically combine them to make wood. (Do I want to even use the term "carbon dioxide?" Maybe. Some kids will love to be able to say it and know a little bit about it like they love to be able to say, "That's a tyrannosaurus.") As you say, "Hard to know which way we'd go because it depends on how the child responds to the initial ideas."

Maybe bring in the concept of energy. Maybe run the reaction in reverse and release energy: burn something. Kids like burning things.

doyle said...

Dear Roger,

I suspect you're right...alas, it's more fun to disagree, but sometime truth gets in the way.

My kids love it when I burn things. One of the very cool things about a good flame is that water, indeed, comes out of it. Hold a chilled glass beaker over the flame, and you will see condensation on it.

Thrills me every time I do it!