Not a whole lot of people look very closely at political polls; you see the headlines, maybe you read the “topline” results, it can all become a blur. But sometimes a closer look can raise some real questions.
Two Republican presidential polls out of Iowa this week have gotten a lot of attention: one released Tuesday night by Public Policy Polling, and one released yesterday afternoon by CNN/Time/ORC. PPP (the first pollster to report a calamitous drop of support for Gingrich in Iowa) showed Ron Paul maintaining a narrow lead over Mitt Romney, mainly thanks to his very strong support among self-identified independents, with Gingrich continuing to sink, Bachmann in fourth place and Perry and Santorum tied for fifth, close behind. CNN showed Romney narrowly ahead of Paul, but its big news was that Santorum had moved up into third place with 16%. This confirmed what a lot of observers thought might happen down the stretch–you know, Paul losing some support as conservatives paid attention to his foreign policy views, Santorum beginning to consolidate the Christian Right vote. Additionally, CNN poll came out a day later. So naturally a lot of people, myself included, thought of it as a newer poll, probably reflecting the very latest trend.
Turns out if you look closely, PPP’s survey was conducted on December 26-27, while CNN’s was launched before Christmas. PPP’s sample was somewhat larger, and thus its MoE is a bit lower. PPP uses robo-calls, CNN uses live phone surveys.
But the most important difference between the two, as Nate Silver quickly noted, is that CNN’s sample was limited to registered Republicans, while PPP’s was not. That’s very significant because Iowa allows caucus-goers to re-register at the caucus site, and they often do. And because Ron Paul draws a significant share of his support from people currently registered as indies or even as Democrats, CNN may be under-stating his support, and overstating Romney’s (and perhaps Santorum’s).
Differences in polls can, of course, be exaggerated. PPP didn’t show any “Santorum Surge,” but its press release did note his unusually strong favorable/unfavorable ratings and second-place support, and suggested he might break away from the also-rans. And both polls had small samples and significant MoEs. But it’s generally a good idea to look beneath the top lines before reading too much significance into any one poll.
