Beware the Poll Write-Ups

Oy, Pew:

At this stage in the campaign, Barack Obama is in a strong position compared with past victorious presidential candidates. With an eight-point lead over Mitt Romney among likely voters, Obama holds a bigger September lead than the last three candidates who went on to win in November, including Obama four years ago. In elections since 1988, only Bill Clinton, in 1992 and 1996, entered the fall with a larger advantage.

That Barack Obama was winning by 8 points last week — the poll was taken September 12-16 and released today — is a pretty good reading for him.

But, c’mon. This poll was done very soon after Obama’s convention (September 4-6), when his bounce was still in effect. Now, let’s see about the “last three candidates” Pew is comparing Obama 2012 to. In 2008, they had Obama exactly tied with John McCain…but that one (seems to have been September 9-14) would have been during McCain’s convention bounce; the Republicans went last in 2008, with their convention taking place September 1-4. In 2004, the Republicans met August 30 – September 2. The one in their current table was taken September 11-14 and had Bush up one point…but they also fielded a September 8-10 survey, and that had Bush up a whopping 16 points! Plus a September 17-21 poll had Bush +3. Going back to 2000, the Democrats met in mid-August, well ahead of Pew’s September poll that had Al Gore up by 5, so that one is at least unrelated to a convention bounce.

Basically, they’re overselling Obama’s advantage, because they’re not comparing like with like. The 2004 case is particularly bad, but overall the advantage for Obama here is just a lot less impressive than the write-up would lead us to believe.

Moreover: since all individual polls are subject to random variation, this kind of comparison just multiplies that kind of error. Think about the three September 2004 polls: odds are that Pew was “wrong” high on the first one, low on the second, and about right on the third (even given that Bush’s read lead was probably higher early in the month than later).

Again: any 8 point lead in a poll is good news, and I’ll agree that a September lead is a lot more meaningful than a July lead. The polling averages suggest that Obama’s lead last week was more like 3 or 4 points than 8 (and no, it wouldn’t totally discredit a pollster to include that in the write-up, at least not in my view), but sure, even post-convention a candidate is still going to be happier up by a few than even or down. The problem here is just that the added comparison to 2008, 2004, and 2000 confuses more than it clarifies — and yet it’s highlighted in their release — what I quoted above is the top paragraph of the write-up on the web page.

Pew does great work, as most of the nationally known pollsters do. What this example reminds us, however, is that the write-ups are often just as prone to error as the numbers — and, perhaps, even more.

[Cross-posted at A plain blog about politics]

Jonathan Bernstein

Jonathan Bernstein is a political scientist who writes about American politics, especially the presidency, Congress, parties, and elections.