28 November 2008

Taking False Comfort in Numbers

I think that the comfort many teachers find in calculating percentages, averages, category weighting systems, and other mathematical manipulations while grading is a belief that these practices are indisputable. "Your child has a 78% average because..." It's there in black and white. Can't argue with the numbers, right?

I was thinking about this again after the NSTA presentation I did last week. Some of the participants mentioned that they are required by administration to update student scores in an on-line gradebook each week---which automatically calculates an average, etc. How are you supposed to implement best practices when the tools and expectations are "old school"? Excellent questions, with no simple answers. The kind of change they're needing is a systems approach---and I'm more focused on the classroom. There are likely some ways to manipulate the weighting system in those soulless automatons of grading software such that categorizing things as "formative" gives it an insignificant impact. Perhaps there are other workarounds, as well.

The other part of the question that was worth noting was simply the idea of the audience for these automated grades. An on-line grading system bypasses the student for delivering information. I can see certain advantages to that (depending upon the student) and know that other stakeholders (parents, administrators, athletic coaches...) have their own needs for student grades. However, there is no way to guarantee that these outsiders to the classroom understand what the grade represents. I was reminded of two recent articles (one in ASCD and one in the Washington Post) which compared parent tracking of grades to watching the stock market. From the WaPo:
Parents and students in a growing number of Washington area schools can track fluctuations in a grade-point average from the nearest computer in real time, a ritual that can become as addictive as watching political polls or a stock-market index.

The proliferation of online grading systems has transformed relations among teachers, parents and students and changed the rhythm of the school year. Internet-based programs including SchoolMAX and Edulink are pushing midterm progress reports into obsolescence. Prospective failure is no longer a bombshell dropped in a parent-teacher conference. A bad grade on a test can't be concealed by discarding the evidence. A student can log on at school, or a parent at work, to see the immediate impact of a missed assignment on the cumulative grade or to calculate what score on the next quiz might raise an 89.5 to a 90. Report cards hold little surprise...

"You can walk around this building and every kid knows exactly how they're doing," said John Weinshel, a teacher who administers EdLine at the school. "The curtain has been stripped from the wizard. There's no more mystery. The grade book is open."
It's the last paragraph that bothers me. As long as there are zeros for missing assignments, points taken off for late work (or added for bringing boxes of Kleenex and cans of food), averages, and so forth---how does a kid know how s/he's doing? The grade book may be open in terms of scores, but how those scores are derived is still very much a mystery in most schools. I think that schools may be giving themselves a large dose of false comfort in assuming that just because gradebooks are on-line that the grading process is less of a crapshoot to most observers.

Are we afraid of using professional judgment when evaluating students? Why is that? Is it because we, as educators, lack enough training to discern which student work meets the targets...but don't want to admit that? Are there other reasons why we take such false comfort in the numbers representing learning?

9 comments:

Roger Sweeny said...

Are we afraid of using professional judgment when evaluating students?

Perhaps--because if we did, the answer would almost always be, "hasn't learned much." Or at best, "teacher just doesn't know."

This is deliberately obscured by the terms "formative" and "summative" assessments. But "summative" is wrong. Such assessments are actually "tentative." The number of questions that a student can answer correctly at the end of a unit may represent real knowledge. However, we all know that many of those correct answers simply represent memorization. As hours turn into days after the test, more and more of what they knew at that point will simply fade away.

I used to be surprised by how many of my students wait until the last minute for extra help, even though I try to make clear from the very beginning that the course builds on itself and that it is dangerous to fall behind. Recently, I realized that waiting is a rational strategy in many courses. Consider two possibilities. A: for the two weeks of a unit, the student takes a extra half hour each day to memorize that day's work. B: the student takes two extra hours before the "summative" assessment to memorize the unit. Strategy B takes only 2/5 the time, and since memory fades, the student who uses strategy B may actually do better on the assessment since her memories will be fresh.

We know how our students do on tentative assessments but we rarely test for true understanding months down the line. And if we do give midterms or finals without copious "review" beforehand, we find that, gee, not much of that "knowledge" stuck.

So instead, we run a game. These are the rules: do your homework, get things in on time, do as well as you can on the assessments. We'll give you points in such and such a way. Students know what we want going in and they know that by the rules of the game, the score is fair.

It is, in a number of ways, a good preparation for life after school.

The Science Goddess said...

Is there a time limit which defines "learning"? Is a day long enough...or does information need to be retained for a lifetime in order to be?

If we run a game in the classroom, what message are we sending to kids? Is that really how life on the outside will be for them?

Intriguing things (for me) to ponder.

Clix said...

Yep - a lot of the reason for education is about teaching students to identify and internalize the "rules of the game" - both specific and implied - and then manipulate those rules to create the best possible outcome.

Maybe it sounds cynical, but it corresponds really well to life outside school.

Roger Sweeny said...

Is there a time limit which defines "learning"?

I think it depends on the purpose of the "learning." If I have fled Iran for Sweden and learn the Swedish language because I think I am never going back, that knowledge had better be permanent. If I learn the strengths and weaknesses of next week's football opponent, there is no harm in forgetting most of it after the game is over (at least if I am a player; if I am a coach, I may have to remember for weeks or months in order to analyze the game and prepare for future games).

How long learning should last depends on the purpose of the learning. The only way to decide that is to answer the fundamental question, "What is the purpose of schooling?"

There is much talk that it is to acquire knowledge ("book learning") that will be used in life. But all high school teachers know in their heart of hearts that this is largely false. By graduation, most of what was learned in the previous four years will have been forgotten. Students have a pretty good idea, too. By the time they reach high school, they have stopped asking, "when will I ever use this?" because they know they won't get a straight answer. Most have decided it doesn't matter. High school has its rules. Play by them, and play well, and you will get a good score. Don't, and you won't.

It is this sense in which high school is a game. But then so is much of life. The great economist Frank H. Knight used to argue that Homo sapiens (wise man) was a poor scientific name for humans. He much preferred Homo ludens (game playing man). Humans, he said, are constantly engaging in competitions. They have an amazing ability to understand "the rules of the game" and to abide by them. It is this, he argued, that keeps us out of a Hobbesian "war of all against all." We refrain from stealing not because there is a policeman on every corner, but because we accept the rule that we shouldn't do it (which is, to a significant extent, because we know others have accepted the rule in relation to us).

Speaking of Frank H. Knight, I sometimes think of this passage when I think about grading, "The saying often quoted from Lord Kelvin (though the substance, I believe, is much older) that 'where you cannot measure, your knowledge is meagre and unsatisfactory,' as applied in mental and social science, is misleading and pernicious. ...the Kelvin dictum very largely means in practice, 'if you cannot measure, measure anyway.' That is, one either performs some other operation and calls it measurement or measures something else instead of what is ostensibly under discussion ... To call [such things] measurement seems to be merely embezzling a word for its prestige value." (On the History and Method of Economics, U. of Chicago Press, 1956, p. 166)

Roger Sweeny said...

So what is the purpose of high school?

Speaking sociologically, the major purpose is teenage day care. That is not dishonorable.

Look at what we tell parents. "Give us your children for seven (or more) hours. We will make sure they are safe. We will make sure they don't drink or do drugs. We will keep them away from temptations--and dangerous messages--that they could otherwise encounter. More than that, we will make them better people. We will reward them for pro-social and productive behavior and punish them for anti-social and unproductive behavior. We will teach them knowledge and skills that will give them a better chance to be successful in life."

There is a lot of truth to this. Perhaps the least truthful part is the idea that the things they learn (as measured by the grades they get) cause them to be more or less successful in life. Mostly, it works the other way around. The people who will be most successful (hard-working, goal-directed, organized, smart) anyway are the ones who play the high school--and then college--game well. Their grades largely signify what they bring to the school, not what they get from the school.

Much of the purpose of high school is to fill teenagers' time with harmless activities that help them grow into good adults. I think part of that time should be spent learning some things pretty much permanently: to read with understanding and write understandable paragraphs, to work with fractions and simple equations, to understand how American governments work, to know the framework of human (and American) history, to understand how everything is made of atoms in motion, to have some idea how we can live without backbreaking labor and dying at 40.

I know I've left things off and I would love it if students learned much more--understood things so the knowledge stayed with them for a long time. But if I expected that, I would come home every day disappointed and depressed.

Stardiver said...

Roger, do you think that if one is teaching things that must be simply memorized and don't stick with students, perhaps one is teaching (and assessing) the wrong things? It my intent and hope to teach for lasting learning. It is my intent to assess in a way that will show reliably that this type of learning is taking place, or not.

PS I saw this post on Twitter, where I am Stardiverr.

Roger Sweeny said...

Stardiver,

Some things that are simply memorized do stick with students. Try asking them about the songs of their favorite artist or the performance of the players on their favorite team.

School subjects are often things they just have no inherent interest in.

As part of my job, I try to get them interested and sometimes I am successful. But they rarely care enough to really try to understand.

Assessing for lasting learning is incredibly difficult. By the end of a unit, students can often "talk the talk" and appear to understand. But if you try to talk to them about that material two months later, you usually find that they have forgotten.

I'm sorry to sound so negative. I hope you are successful.

Mr. McGuire said...

Although I use Engrade online gradebook and really think it helps students, I understand your concern. I use rubrics to score most of my students' work. At times, a grade could be hard to explain when comparing work between two students. Sometimes putting a grade at all seems unfair. However, in the age of testing and NCLB, data rules and everyone wants to see the numbers.

Mr. McNamar said...

Science Goddess,
It has been a long time since I've stopped by--forgive me. One might think I've forgotten my Pacific Northwest roots.
The reason I am afraid to use professional judgement is simple. The public doesn't trust us, nor do our students. I am reminded of my Pre-College Englis class from my Washington days. The students were riled up because I scored essays "too harshly." They didn't trust me when I told them that the following year would be even harsher.
So I experimented. A local college professor scored an essay which I had given a B-. The professor gave it an F after reading the first of three pages. My students, after seeing the copy of the scored assignment, concluded that the professor was "way off."