This new poll—discussed here by Peter Baker and Dahlia Sussmann—is already getting attention for what the article doesn’t say: this survey is a “panel-back.” That is, respondents who were previously interviewed in an April 13-17 poll have been interviewed again. You will learn this if you choose to click through to the toplines of the poll (same deal at CBS). You will learn more if you read what Steven Shepard says at National Journal, or what Mark Blumenthal said about panel-backs a while back.
The funny thing here is that, as far as I can tell, the usual complaint about panel-backs doesn’t even apply. The usual complaint is that the respondents who agree to be reinterviewed are different from those who don’t. I don’t see a ton of evidence for that. Compared to the April sample, the May sample is a bit more likely to disapprove of Obama’s job performance but doesn’t show a similar shift on other related items (like party identification). It’s harder to assess another critique of panels—that answers in the earlier poll affected answers in the later poll—but in general I wouldn’t expect this sort of “panel conditioning” to be a big factor.
Instead, what struck me about this poll was what more CBS and the Times could have done with it—that is, how much better the analysis could have been. I am building here off my post on questions to ask regarding the electoral impact of Obama’s same-sex marriage endorsement. I’ll go through key passages from the NY Times article.
Most Americans suspect that President Obama was motivated by politics, not policy, when he declared his support for same-sex marriage, according to a new poll released on Monday, suggesting that the unplanned way it was announced shaped public attitudes. Sixty-seven percent of those surveyed by The New York Times and CBS News since the announcement said they thought that Mr. Obama had made it “mostly for political reasons,” while 24 percent said it was “mostly because he thinks it is right.” Independents were more likely to attribute it to politics, with nearly half of Democrats agreeing. The results reinforce the concerns of White House aides and Democratic strategists who worried that the sequence of events leading up to the announcement last week made it look calculated rather than principled.
Okay, fine. But exploit the panel! Were respondents who opposed Obama in April more likely to say “political reasons”? Undoubtedly yes. Were people who said “political reasons” less likely to support Obama now than they did in April, controlling for other factors? We don’t know.
The survey results made it clear that the president was wading into a divisive area of American life, one that may not top the nation’s priority list but still has the potential to hurt him at the margins in elections in November. About 4 in 10, or 38 percent, of Americans support same-sex marriage, while 24 percent favor civil unions short of formal marriage. Thirty-three percent oppose any form of legal recognition. When civil unions are eliminated as an option, opposition to same-sex marriage rises to 51 percent, compared with 42 percent support.
“At the margins” is the kind of vague formulation I was hoping reporters and commentators would avoid. Hurt him how much? In which states? With which voters? In my simulations, there wasn’t much evidence that attitudes toward same-sex marriage would really hurt or help Obama.
The poll showed that relatively few voters consider same-sex marriage their top issue amid continued economic uncertainty, and more than half said it would make no difference in their choice for president. But among those who said Mr. Obama’s position would influence their vote, more said they would be less likely to vote for him as a result; in a close race, even a small shift in swing states could be costly.
I tweeted this right after Obama’s announcement:
Do not — repeat, do not — believe survey items that simply ask people whether Obama’s #SSM endorsement will affect their votes. (1/2)
And part 2 of that tweet was a link to Lee’s old post on how voters’ stated reasons for their choices are often not really reasons at all, but rationales. Pollsters keep asking versions of this “Does X make you more or less likely to vote for so-and-so?” question. I can’t tell that answers to that question have any meaning.
But more importantly: USE THE PANEL. The NY Times and CBSalready know whether people changed their vote intention between April and May. How many people changed? In what direction? Did people who said that Obama’s endorsement make the less likely to support him actually change their vote? Did any change also depend on their overall attitude toward same-sex marriage?
Now, granted, the April survey did not include a measure of attitudes toward same-sex marriage. So this poses some challenges for causal inference—namely, did attitudes toward same-sex marriage cause people to change their vote intention, or did people’s vote intention (that is, their preexisting support for Obama) affect their attitude toward same-sex marriage? Still, even with necessary caveats, this analysis would be far more enlightening than relying on voters to tell you whether Obama’s announcement matters for their vote. And it will be more enlightening than vague formulations like “in a close race, even a small shift in swing states could be costly.”
Mr. Obama’s team is counting on the notion that whatever he might lose in votes or intensity of support will be offset by increased excitement among young voters and his liberal base.
USE THE PANEL. Were young voters and liberals more likely to support Obama in May than in April? Now, granted, lots of things happened between April and May besides Obama’s endorsement. But still: wouldn’t it be interesting to know the answer?
The new nationwide poll is based on telephone interviews conducted from May 11 through 13 on landlines and cellphones with 615 adults. With less than six months until the election, Mr. Obama remains in a tight race with Mr. Romney. A month ago, a Times/CBS News poll showed the two tied at 46 percent each; the latest survey had the Republican challenger at 46 percent to the president’s 43 percent, an edge that was within the margin of sampling error of plus or minus four percentage points.
No mention of the panel. This was where that mention might have gone. It would also be nice to compare the results in this poll to other polls.
Mr. Obama’s vulnerable standing in the poll came despite rising optimism about the economy. About a third of voters said it was very or fairly good, the most since January 2008. More than a third said it was getting better, compared with a quarter who said it was getting worse. Jobs and the economy remain by far the most dominant issue, with 62 percent naming it their top priority and 19 percent their second highest. By contrast, just 7 percent chose same-sex marriage as the most important issue and 4 percent as the second-most important.
The low salience of same-sex marriage is important, and good for the Times for including this. But I was a little curious about another aspect of these results: the May survey shows economic evaluations were more positive than in the April survey, even as Obama’s net approval rating declined. For example, the percent saying the economy was good or fairly good went up by 5 points. But Obama’s net approval went from +6 to +2. Now, you’ve got a smaller sample in this second poll, larger margins of error, and also a far from perfect relationship between economic evaluations and presidential approval anyway. So perhaps this result isn’t anything. I just wondered about it, and whether the panel, even with the small sample size, would shed light on changes in economic evaluations vs. changes in approval.
Back to same-sex marriage. The Times headline—“Obama’s Switch on Same-Sex Marriage Stirs Skepticism”—signals “Hey, something is happening here.” But the results reported in the article don’t really provide much evidence that anything is happening, and I suspect that taking advantage of the panel would have confirmed that. But even if so, the article would have been relevant and, most importantly to me, would have exploited unusual and interesting data.
[Cross-posted at The Monkey Cage]