On Tuesday the former superintendent of the Atlanta Public schools, Beverly Hall, along with 26 other Georgia educators, turned herself in at the Fulton County Jail to face charges “including racketeering, theft by taking and making false statements” connected to systemic cheating on standardized tests between 2005 and 2009, according to CNN. Hall, according to the indictment, “protected and rewarded those who achieved targets by cheating” and “ignored suspicious” increases in test scores in her district.
The school cheating revelations are a problem for school evaluation advocates. It’s hard to argue that standardized testing is improving education as long as Americans keep hearing that teachers are dishonest. But the problem, in reality, is the how teachers can cheat, not the use of standardized tests itself.
Discoveries of organized cheating in school districts are rare, but they’re coming up more often. Investigators have discovered widespread problems with standardized examinations in Houston, Chicago, New York, and Washington DC.
This leads testing opponents to argue that standardized testing is the problem. As the American Federation of Teachers clucked, cheating represents “the unintended consequences of our test-crazed policies.”
Proponents of assessment, however, argue that the scandals don’t undermine the value of standardized testing. Education reformer RiShawn Biddle writes that Atlanta’s bubble tests are “the most-useful tools honestly and objectively assessing how students are learning” and the real problem is simply “the craven incompetence of men and women who didn’t want to do the hard work of helping kids learn.”
So is there something wrong with the tests, or the teachers who administer them? Probably both.
Part of the problem is that the tests just aren’t very good.
As Mark Kleinman put it on this blog yesterday:
No Child Left Behind (like other approaches relying on high-stakes once-a-year standardized testing) violates every known principle of quality assurance and process improvement: [American statistician William] Deming [who argued that the key is successful business practice is focus on “continual improvement and think of manufacturing as a system, not as bits and pieces”] was right… but no one in the Department of Education seems to be aware of that.
The Atlanta scandal is just the latest illustration of Dukenfield’s Law: “Anything worth winning is worth cheating for.” Whatever the incentive is for actual good performance, the incentive for simulating such performance by fiddling with the tests is exactly the same. (Campbell’s Law enunciates a related principle: “The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” The pithier version, the Goodhart/Strathern Law, holds that “When a measure becomes a target, it ceases to be a good measure.”)
This is a really important point. Evaluation is useful, but there’s such a thing as bad evaluation. I mean, if you’re going to force the principles of manufacturing on education systems, it’s best to at least look at how good businesses measure performance, not just use some half-assed version of assessment and consider the matter over.
If you have bad schools and bad teachers and then decide to fix the system only by instituting high stakes examinations, of course teachers are going to find a way to cheat. They’d be foolish not to. Why take the hard route when it’s so easy to get rewarded for the easy one?
Even Bill Gates, perhaps America’s main proponents of standards-based education reform, seems to think something has gone wrong. As he wrote in the Washington Post
This is one reason there is a backlash against standardized tests — in particular, using student test scores as the primary basis for making decisions about firing, promoting and compensating teachers. I’m all for accountability, but I understand teachers’ concerns and frustrations.
Even in subjects where the assessments have been validated, such as literacy and math, test scores don’t show a teacher areas in which they need to improve.
If policymakers decide to measure education largely by a standardized test (and then give people bonuses due to high performance on that test), some teachers are going to cheat on the test. And, perhaps more importantly, we won’t learn anything useful about how schools are doing.
The main lesson here shouldn’t be “standardized tests don’t measure what matters” or “we have a lot of bad teachers” (though both of those things are probably true) but, rather, that standardized tests should be one of many evaluation systems used by schools and school districts. The problem isn’t that the evil teachers cheated to get some extra cash; the problem is now we can’t really be sure what’s going on with student performance in Atlanta. [Image via]
