Polling data is obviously the mother’s milk of campaign reporting, and sometimes of issues reporting. Without it, most discussions of politics would constantly descend into a nether world between the long-term verities of political science and the short-term impressions of events and anecdotes and, well, spin. I don’t have much patience with people who complain about polls as a contaminant of political analysis and public policy, because all polls generate is information–some of it good, some bad, some valuable, some marginal or irrelevant. Sifting that information and putting it in a proper perspective is an analytical task that is often done poorly. But as Bill James often said about baseball “traditionalists” who rejected the use of statistics they’d never heard of, people who reject new information because it is not the answer to every question typically rely on old information that has not been analyzed or recently reviewed.
I say all that by way of introducing Charles Franklin’s essay yesterday on the public refusal of some media figures to use “unreliable” polling data that privately influences their perception of events:
Just possibly you’ve heard of a surge Rick Santorum is enjoying in Michigan. A surge that has now put him ahead of Mitt Romney in Romney’s childhood home state…
But if you tuned in to The Daily Rundown with Chuck Todd this morning (Feb 15), you didn’t see this chart. You instead heard Todd report on an OHIO poll from Quinnipiac showing Santorum leading Romney 36-29. And then you heard Todd say “Ohio, Michigan, there’s a lot of similarity there.” OK. But what of the data represented in the above chart about Michigan, the critical current topic? Todd continued, saying “We haven’t seen some great polling out of Michigan yet that we are willing to quote, that meets NBC News standards, but its clear Santorum is on the move. We are seeing it nationally. We are seeing it there.”
And so there you have it. NBC News standards force Todd to ignore the evidence of multiple polls from Michigan, and instead rely on one poll from a neighboring state. All to avoid saying the dread words: PPP, or ARG, or MRG or Rasmussen or Mitchell Research. Those polls all show Santorum leading Romney by from 3 to 15 points in Michigan.
Without going through Franklin’s whole argument, the point is that there is no bright line between “good” and “bad” polls, or even pollsters (short of the kind of complete house-of-cards revealed in the DailyKos/R2K fiasco of 2010). You can, as Nate Silver has so methodically done, measure the accuracy over time of various polling outfits. You can evaluate different polling methodologies, and take that into account, and be careful to note variables that affect reliability, such as sample size, likely-voter screens, and length and timing of surveys. When a lot of data is available on a single contest or question, you can reduce the risk of distortions by averaging the results. And in polling about issues, as opposed to simple horse-race tests, you can also look closely at how questions are framed and ordered, and of course, how the results are reported (for that reason, I tend to ignore Rasmussen’s issue polls, but not the firm’s horse-race surveys).
You can and should do all these things, but simple, arbitrary rules that neatly separate types of pollsters (say, by party affiliation, if any) or types of polling (robo-pols versus live interviews) often do more harm than good, in that they restrict the flow of information and can over time mislead us more than the careful utilization of “bad” polls.
So when I use polling data in this blog, I will try to do all the things that appropriately hedge against misinformation and contextualize the implications. But please don’t get mad if I use the findings of this or that pollster that you think is a hack or a crook, or fail to ignore poll findings that contradict what we know to be “true” in our “guts” or according to our principles. More information is better than less information, and while it’s easy to misinterpret it, you really can’t get too much of it.