If it's been awhile since you had to do anything with descriptive statistics, an effect size is a number that describes the strength of a relationship between two variables in a population. This is different from the significance that might describe a particular effect. Effect size is scaled from -1 to +1.
At zero, there is no relationship between the variables. So, think about this as something like trying differentiated instruction in the classroom and seeing absolutely no change in student learning. Those things which decrease student learning (more tv) would reach toward the negative end of the scale and other things (reduced class size) would reach toward the upper end. Hattie's argument, however, is that comparing an effect to zero is the wrong comparison. Why?
Because his research shows that the average effect size---all of the ed research out there put together---is .4. So, for the most part doing something...anything...is better than nothing. But more importantly, we should eliminate strategies that are less than .4. Shouldn't we look for things which will at least get better than average results?
Hattie moved through a list of 120 variables, pausing here and there to talk about one in depth. One of these was class size. Does the graphic below surprise you?
What do we see here? Reducing class size does have a positive impact on student learning; however, it is less than average. Does that mean we shouldn't spend money on lowering teacher-pupil ratios? (Primary teachers, I can hear you screaming from here...) The answer is "sorta." As with anything, the numbers don't tell the whole story. The next question is "Under what conditions does reducing class size have the greatest impact?" or "Why does reducing class size not have a greater effect on student learning?" Basically, teachers who have had professional development in how to work with a class of students above (or below) the standard number have a greater positive impact. Others just teach the same way, regardless of the number of students in the room, and very little difference is made.
This same sort of questioning could apply to lots of things that fall below the magical .4: differentiated instruction, inquiry-based learning, problem-based learning, and so on. There may be conditions which allow them to be greatly effective, but based on what we have in the research right now---nope. I would guess this is why anecdotal evidence drives so many individual teacher decisions. This might be okay---or it could be dangerous.
Here is another example from Hattie:
This graphic compares two broad leadership styles: principals who push a vision...and principals who function as instructional leaders in their schools. (My hunch is that some of you are thinking you have a principal who is "none of the above.") I found this particular comparison interesting, mainly because those who push back the hardest against the reform movement are the ones who believe having vision trumps all. Obviously, there is something positive---you do want leadership that can inspire and bring together a school. But that is not enough to make even an above average difference in student learning. In fact, it's slightly less effective than reducing class size. Schools need administrators who understand and walk the talk of high quality curriculum, instruction, and assessment. Going on and on about your vision isn't really helping kids.
For those of you concerned that Hattie only considered test scores, he didn't. There are lots of ways to look at student learning---his task was to synthesize them. What things make the biggest difference? High quality feedback on student work, strong teacher-student relationships, among others. You would not be surprised at what makes the top 10. The surprises are all below the .4 mark. While I wouldn't advocate for throwing any instructional babies out with the <.4 bathwater, it would seem that these deserve some careful thought before further implementation. Time and resources (both human and material) are so precious. We need to make the best use of these that we can.