POLL WATCH….I’ve gotten a few emails from people who are outraged by alleged methodological problems with the Time poll that showed George Bush leading John Kerry 52%-41%. Here are the complaints:
The poll includes “leaners,” people who were asked to state a preference even if they said they were undecided. The previous poll, taken last week, didn’t include leaners.
I’m not sure why SRBI (the polling firm) excluded leaners in last week’s poll, but they’ve included them in all their previous polls. If you compare their August 3 poll with their most recent poll, it switches from a 48%-44% Kerry lead to a 52%-41% Bush lead. That’s a huge change and the methodology is the same both times.
The pollers asked for male residents first, and talked to females only if no male was available. I don’t know why they do this, but it’s apparently a standard part of their methodology and produces about 52% female respondents. What’s more, all modern polls are weighted to a fare-thee-well to ensure that the demographics of the poll sample are close to the demographics of the country as a whole. There doesn’t seem to be any monkey business here.
The poll was taken during the Republican convention, when undecided voters are most likely to be leaning Republican. This is a fair complaint. There’s not much question that calling people while the convention is in progress is going to show Bush in the best possible light.
Bottom line: there doesn’t seem to be anything wrong with this poll except its timing, which clearly shows the Bush bounce at its maximum. In addition, it might just be an outlier, a poll that happened to get a too-heavy Bush sample by random chance.
In other words, there’s no need to panic. Other polls show a much closer race, and a Bush bounce during the convention is hardly a surprise anyway. At the same time, though, there doesn’t seem to be any obvious problem with the poll’s methodology. It is what it is, and it’s best not to invent spurious reasons to discard evidence just because you don’t like it.
UPDATE: Phil Trounstine, Director of the Survey & Policy Research Institute at San Jose State University, emails an explanation for the “youngest male, oldest female” methodology:
When reputable surveyors do random digit dialing (RDD), they are calling a random selection of households (assuming one phone per household). But once they reach a household, they need a system to randomize whom they speak to within the household. If they just interview the first person to answer the phone, their surveys would be overloaded with female respondents, since women answer the phone first in most households.
There are various different methods for randomizing at the household level. Two methods are most popular. One is the “most recent birthday” method, whereby the pollster asks to speak to the person in the household who has had the most recent birthday. While this maintains randomness, it is often confusing to the respondent and can cause a drop in response rate.
The other popular method is “youngest male, oldest female,” whereby the surveyor asks to speak to people in that order. This is not purely random, but is systematic and has been found to introduce no significant bias. The advantage is that it is easier for the respondent to understand and increases the number of male respondents ? and especially younger males ? who are normally underrepresented in household sampling because they’re harder to reach. This means that when the pollster applies weights to obtain a gender balance, he can apply a smaller weight for gender, which improves the reliability of the data.
Whether there are other methodological issues ? party distribution, definition of likely voters, assumptions about race and ethnicity of voters, etc.) ? is another matter. But the “youngest male, oldest female” selection at the household level is a legitimate methodology.