22 December 2010

Muddying the Bathwaters

John Hattie, the author of Visible Learning, was the keynote speaker at a recent conference I attended. The book represents years of work reviewing educational research (over 800 meta-analyses) and determine effect sizes of various factors which influence student learning. These include things outside the classroom (e.g., television, siblings) and inside the school (e.g., various instructional models, leadership styles). A couple of things stuck out at me during the presentation---and I am most anxious to have a copy of the book to read.

If it's been awhile since you had to do anything with descriptive statistics, an effect size is a number that describes the strength of a relationship between two variables in a population. This is different from the significance that might describe a particular effect. Effect size is scaled from -1 to +1.

At zero, there is no relationship between the variables. So, think about this as something like trying differentiated instruction in the classroom and seeing absolutely no change in student learning. Those things which decrease student learning (more tv) would reach toward the negative end of the scale and other things (reduced class size) would reach toward the upper end. Hattie's argument, however, is that comparing an effect to zero is the wrong comparison. Why?

Because his research shows that the average effect size---all of the ed research out there put together---is .4. So, for the most part doing something...anything...is better than nothing. But more importantly, we should eliminate strategies that are less than .4. Shouldn't we look for things which will at least get better than average results?

Hattie moved through a list of 120 variables, pausing here and there to talk about one in depth. One of these was class size. Does the graphic below surprise you?

What do we see here? Reducing class size does have a positive impact on student learning; however, it is less than average. Does that mean we shouldn't spend money on lowering teacher-pupil ratios? (Primary teachers, I can hear you screaming from here...) The answer is "sorta." As with anything, the numbers don't tell the whole story. The next question is "Under what conditions does reducing class size have the greatest impact?" or "Why does reducing class size not have a greater effect on student learning?" Basically, teachers who have had professional development in how to work with a class of students above (or below) the standard number have a greater positive impact. Others just teach the same way, regardless of the number of students in the room, and very little difference is made. 

This same sort of questioning could apply to lots of things that fall below the magical .4: differentiated instruction, inquiry-based learning, problem-based learning, and so on. There may be conditions which allow them to be greatly effective, but based on what we have in the research right now---nope. I would guess this is why anecdotal evidence drives so many individual teacher decisions. This might be okay---or it could be dangerous.

Here is another example from Hattie:

This graphic compares two broad leadership styles: principals who push a vision...and principals who function as instructional leaders in their schools. (My hunch is that some of you are thinking you have a principal who is "none of the above.") I found this particular comparison interesting, mainly because those who push back the hardest against the reform movement are the ones who believe having vision trumps all. Obviously, there is something positive---you do want leadership that can inspire and bring together a school. But that is not enough to make even an above average difference in student learning. In fact, it's slightly less effective than reducing class size. Schools need administrators who understand and walk the talk of high quality curriculum, instruction, and assessment. Going on and on about your vision isn't really helping kids.

For those of you concerned that Hattie only considered test scores, he didn't. There are lots of ways to look at student learning---his task was to synthesize them. What things make the biggest difference? High quality feedback on student work, strong teacher-student relationships, among others. You would not be surprised at what makes the top 10. The surprises are all below the .4 mark. While I wouldn't advocate for throwing any instructional babies out with the <.4 bathwater, it would seem that these deserve some careful thought before further implementation. Time and resources (both human and material) are so precious. We need to make the best use of these that we can.


hschinske said...

The thing is that it's not economically feasible to reduce class size in a public school far enough that you would see really big effects for the students. If you could reduce class size to twelve or so, as in top private schools, THAT would have a honking big effect, I'll bet you. Lesser reductions likely make things better for teachers, decreasing teacher burnout and so forth, which I'm sure is good for students too, but not enough to show up in the hard stats that much.

And, of course, there are the factors you already mentioned, such as more experienced teachers, perhaps with particular kinds of training, being either better able to wrangle a large class or better able to take advantage of a smaller class size to differentiate.

The Science Goddess said...

And there are other factors at work, too---like the "one kid" factor...that one kid who really tips the difference (for better or worse) by his/her mere presence in the room. (This was a separate item in Hattie's analysis.)

I can think of classes I had that were too small (e.g. 8 kids)---not enough dynamic to keep things moving...and classes that were very large (38 kids) which were actually okay.

There is so much that goes into a classroom mix, I'm not sure that there is such a thing as a Goldilocks "right size." I'm not sure that we (via Hattie) can say that lowering class size doesn't make a difference. It probably does for some situations and some kids---just not all of them.

The Dirt on Soil said...

I wonder what this test would look like if they only studied students in academically struggling and low-socioeconomic areas. I wonder if different things work better for different population types.