Political polling has suffered some embarrassing failures this year.
In Michigan’s Democratic Senate primary, polls predicted a blowout for Abdul El-Sayed; he won by less than a percentage point. In Wisconsin, polls showed Democratic Socialist Francesca Hong enjoying double-digit leads in the gubernatorial primary; she narrowly lost to David Crowley. Think pieces have abounded on what’s gone wrong with political polling.
In truth, polls have always been unreliable.
Longtime Democratic pollster Stefan Hankin, founder and CEO of Lincoln Park Strategies, explains the myriad of challenges currently facing the polling industry, including overhyped results and the lack of professional standards.
This transcript has been edited for length and clarity. The full interview is available at Spotify, YouTube, and iTunes.
Anne Kim: Before we talk specifically about what’s gone wrong with political polling, I’d love to nerd out a little bit on what makes a good poll. What is your platonic ideal of a poll?
Stefan Hankin: I’ll call it in broad terms. You want a sample size that is proportional to the size of the [state] if you’re doing a state-wide election. You’re going to want a bigger sample size in California than you will in Rhode Island, let’s say, just based on the populations of those states. Now, there’s a ceiling where it doesn’t make too much sense to go much bigger on the sample size because it’s diminishing rate of returns in terms of the margin of error.
So for a larger state like California, 1,200 is probably the maximum we’d ever really recommend unless you really want to get into subgroups, like we want to look at African-American women compared to Hispanic women. But in broad terms, for a smaller state, somewhere around 500 respondents for a statewide survey is your gold standard.
If you’re looking at a general election, ideally you want only “likely” voters who are going to be voting that November. In a primary poll, it’s going to be focused on either Democrats or Republicans who are likely to show up. And that “likely voter” piece is where I’m going to argue 90 percent of the problems are in trying to figure out (a) who are the likely voters and then (b) getting those folks on the proverbial phone. So much more of what we do is texting people or sending out links to an online survey versus having the discussion over the phone.
Anne Kim: So if I’m understanding you right, the ideal political poll is one that samples a large enough number of the people whose behavior you want to predict. So let’s get to the question then of likely voters. Why is it so hard to find them?
Stefan Hankin: There are two ways you can go about it, and usually pollsters do a combination of the two. So there’s past behavior, right? If you’ve voted in the last three general elections, let’s say, the likelihood of you voting in this upcoming general election is incredibly high. Now, once you get past the regular voters, that’s where it starts getting trickier. Everyone has their own approach to it.
Just about every pollster is going to ask at the beginning of the survey, “Are you planning on voting?” And then everyone says yes, because that’s the right answer. A lot of pollsters will go with some sort of scale, and as you get closer to the election, that scale sort of tightens over whom you’re letting into your “likely voting” universe.
But at the end of the day, you’re relying a lot on a human being predicting what they’re going to do and giving an honest answer about what they’re going to do, which is not what humans are great at doing.
Anne Kim: Quick clarifying question. Where do you get the names for the people you want to ask? Do have access to voter rolls?
Stefan Hankin: You can go through the state party. Companies also provide this as a service. They work with the states, they pay money at some level to get those voter rolls, and then they resell that data.
Regardless of which mode, we have an account, we log in, and we put in the parameters of who we’re looking for. So if there are six million voters that meet that criteria, and I need 100,000 of those names for my sample, it’ll pick a random sampling of 100,000. Then we download that list and use those phone numbers or email addresses to reach out to voters.
Anne Kim: So if you reach out to 100,000 people, how many of them respond? How many people do you have to reach out to [to poll] 1,200 or 500 “likely” voters, however you’ve defined it?
Stefan Hankin: When I first got into this business in the early 2000s, we were pulling 20 to 25 names for every complete we wanted to get. So if we wanted 1,000 completes, we’re pulling 25,000 names. Now we’re in the 100 to 120 names per complete, just because it’s gotten that much tougher to get people to take surveys and to answer their phones.
All of us have caller ID. Most of us, myself included, don’t pick up a number that we don’t recognize.
Anne Kim: But even if it’s harder to reach folks, if you are still able to reach the correct number of folks, it shouldn’t affect the accuracy of the polls, you would think. So that leads to the question about whether polls are, in fact, getting less reliable or if we’re just seeing a few high-profile failures everyone’s upset about that are casting all political polling generally into doubt.
Stefan Hankin: I think it’s a little bit of both. With the extra names and numbers that we have to reach out to, the number of days it took to collect the data for a 500-sample survey used to be three days. Now it’s more like five or six, and it takes a little more time to work the sample “correctly.” More time equals more money.
What I think you have, especially in the public domain, is a mix of credible pollsters—people who are doing things “correctly”—and then you have others who are working on slimmer budgets, and organizations that are trying to get attention or to get numbers out there to build a narrative.
And then on top of that, you have campaigns reporting on polls. I probably get fifty fundraising emails a day. It’s insane. But it feels like half of them are talking about some poll that they don’t have a link to. They don’t reference who did the poll, or when the poll came out. “We’re within the margin of error!” “We’re losing!” “We’re down!” “We’re up!” It’s not even malpractice—this should be illegal in my book.
If we want to throw all digital fundraisers into the bottom of the ocean, I’d be fine with that. It’s not helping the industry. Some of the bad reputation is based on numbers that I don’t even actually believe exist.
That’s not to excuse anything that happened in Michigan and Wisconsin; I think that’s a different animal. But [this all contributes to the lack of] trust in polls.
Anne Kim: That raises a lot of interesting questions. Number one, do pollsters have a trade association? Are there professional standards? Can anybody set themselves up as a pollster? Is that part of the problem here?
Stefan Hankin: There are organizations. So there’s AAPOR, the American Association of Statistical … I should know the acronym…
Anne Kim: So this is part of the problem, right?
Stefan Hankin: Yeah. So there are standards, but there’s nothing to stop anyone from claiming that they’re a pollster. There’s no enforcement of any of this.
And now you’ve heard some groups are putting out “synthetic” data surveys to show what voters could say. Again, it’s just [somebody thinking,] “How do we get this information or this narrative out there? Can we get some data, whether it’s real or not, to back up what we want to say?”
There’s literally no one checking these things, and so an email goes out that says, “Hey, we’re down by five!” There’s no one looking into that email saying okay, show us the numbers where you pulled that out [of].
Anne Kim: Right. You can’t fact check a poll. But it sounds like you’re also saying that what went wrong in Michigan and Wisconsin is not a question of bad actors per se or even sloppy actors, but something different. What happened in those two states?
Stefan Hankin: This is all theory because I wasn’t doing any of the polling. I don’t have access to any of the polls and the numbers to say this is what definitively went wrong.
The primary electorate is much harder to predict than the general election, and it’s very hard to compare year to year. So for a general election, we know every four years there’s a presidential, and that’s a very predictive marker. Then we have the years in between, and those are different kinds of voters who turn out for those elections versus the presidential elections.
Now you have years like 2016 where people turned out of the woodwork to vote for Donald Trump. That sounds like I’m excusing our entire industry, but it was just incredibly hard to predict that this was going to happen and/or to be able to pick it up in the polling.
Now in some states, you might not have a competitive primary for years, and then all of a sudden, there’s a competitive primary. Let’s use Michigan as an example. I’m going to plead ignorance on the last time there was a really contested Democratic primary for a U.S. Senate seat in Michigan, but it’s been a few years. So trying to get an understanding of which voters are likely to turn out can be pretty difficult.
I also think what happened is that the public pollsters who were putting out numbers overestimated the number of younger voters who were going to turn out. I think their samples were skewed a little bit more towards younger voters who were probably, in general, [more likely to be] supporters of El-Sayed.
I’ve no idea what the campaign was seeing from internal numbers that never see the light of day. They might have seen it as a much closer race.
Anne Kim: Are we expecting too much from polls? And are pollsters honest about the limitations of their polls when they talk about the results? Has your industry built up an expectation that polls equal behavior exactly as predicted?
Stefan Hankin: I’m going to ask all pollsters who are listening to this to tune out for five minutes, so I don’t get disinvited from any of the pollster gatherings.
The polling industry has oversold what the results of a poll mean. And it’s not just pollsters. We see this on Five Thirty Eight and Real Clear Politics, where there are ratings of how “accurate” a polling outfit is. It’s a B.S. measure.
If we do a survey that says Candidate A is at 46 percent, and Candidate B is at 40 percent, that’s within the margin of error. That’s basically saying that if we repeated this survey 100 times, 95 out of 100 times, we would expect that result. Now the margin of error is going to depend on your sample size, but let’s just say it’s four points.
So if we did this survey 100 times, 95 out of 100 times, it should be somewhere between 50 and 42. And then the other candidate, if they were at 40, it’s really saying that they’re somewhere between 44 and 36. So either it’s kind of a tie race, or there could be a bigger lead or somewhere in the middle.
A lot of times, pollsters will walk into the room and say, “Hey, our poll said 40 to 46, and that is the word of God.” But if it’s a four-point margin of error, and you have a tied race somewhere near 50, your margin of error is closer to seven points in either direction.
A lot of stuff goes into trying to play up pollsters as [people who] can perfectly predict the future. We play into the narrative, and then when it goes poorly, it looks really bad.
Anne Kim: Some are beginning to even argue that polls may be obsolete because now we’ve got prediction markets. Who needs polls if we can just go to Kalshi or Polymarket or whatever and see what the market thinks? Or maybe regular people who may not respond to pollsters will bet on a particular outcome. What’s your response to that line of argument? Is there any validity to that?
Stefan Hankin: You should always be looking at as many data points as humanly possible to get an understanding of where a race is.
If this is a state where there’ve been a lot of surveys released and in the last month, ten out of 11 surveys say one candidate’s winning, they’re probably winning. And then you can go look at Kalshi and say, “Are we seeing similar types of bets there? Are we seeing something different?”
I don’t think Kalshi is not predictive of things, but when we compare our own data to what we see on Kalshi, we’re not seeing this huge difference when people are putting their own money on the line. What I think is happening more on Kalshi and Polymarket is you now have people gaming the system. There are bets being made that are not because they really think this is going to happen, but they’re hedging. This is arbitrage. It’s like, “Hey, if I put money over here, I might be able to chase some people to put these bets in, and that’s going to help me on this other bet.” It’s not just pure, “I really think this candidate’s going to win, and I’m putting five dollars on it.” There’s a lot more going on behind the scenes.
But it’s not worthless data. If we are seeing anything different in the prediction markets than in the polling, how do we want to factor that into our thinking of what this really means?
Anne Kim: That’s an interesting point because the point of being on Kalshi is to make money. So that certainly skews the incentives in a particular direction. And the purpose of polling, presumably, is to find out information. So that’s a very different set of motivations that are motivating both institutions.
Stefan Hankin: A hundred percent.
Anne Kim: Given what’s happened so far, how much credence are you going to be putting into the polls going into November?
Stefan Hankin: As much as possible with the caveat of buyer beware. This is the information we have—until we can figure out some perfect way of predicting human behavior.
The other challenge is that polling’s expensive, and there aren’t a ton of organizations that are paying a lot of money to get public numbers out there. So with some high-profile races, there will be more polls, and then you have that ability to look at the bigger landscape. Like obviously, in a presidential swing state, there are a lot of polls getting pushed out.
But then in a year like this, even in some of the more contested states, it’s not like we’re getting six surveys a week coming out of Texas. Everything I’ve seen so far is showing it’s a very tight race. So it’s probably a very tight race.
If the polls over the next two months show that Talarico’s up and then Talarico ends up winning, regardless by how much, polling was great. If they show Talarico is up, and then he ends up losing, polling’s terrible.
People tend to forget the miss when you win. If you were working for a campaign, and you said our candidate was going to win with 59 percent of the vote, and they end up winning with 54, it’s a five-point miss. That’s not great. There shouldn’t be a big party thrown for us on the amazing job we did.
But there’s going to be a very different reaction if we said, “Hey, we’re going to win with 52 percent” and end up losing at 47. It’s still five points, right? But the outcome is different, so the polling [is perceived] as not great.
Anne Kim: What practical advice do you have for voters and concerned citizens about how they should responsibly consume political polling? I will confess that when I see a poll, I just look at the number, and I tend to skip the part on methodology. What’s your advice on how to read the fine print?
Stefan Hankin: The big thing is if there’s the ability to look broadly. Are there multiple polls you can look at? Go on Real Clear Politics or Five Thirty Eight or whatever aggregator as your first stop of getting a general [feel] for what’s going on in a race.
If that doesn’t exist, and there are just a couple of one-off polls here and there, I think the big thing to look at is who paid for the poll. And is there an actual link to a poll? If you are just getting an email talking about how this candidate is up by a little bit, down by a little bit, whatever it is—if there are no actual results being shared, don’t even look at it. It’s probably just marketing.
Who’s releasing the poll is the other big tell. So if we were talking about Wisconsin, Marquette usually puts out a very good poll. They’ve been pretty spot on far more times than they’ve been “wrong.” And there’s no real motivation for Marquette University to put out somewhat dubious numbers just to push a narrative.
Same thing with The New York Times, The Wall Street Journal, and other established newspapers. There’s no reason why they would be putting out bad numbers for a purpose. Now if the numbers are being released by “Americans for a Free Society” or some group you’ve never heard of, and they don’t have a website, it’s being released for a reason.
It’s the same thing when a campaign releases. Not to say that the campaign made up numbers, but they’re releasing it for a reason. They want to show that, “Hey, this is tighter than you think,” and we want more money. There’s no reason why you would release internal numbers unless you’re pushing a narrative.
Anne Kim: So a final question for you has to do with the future of your industry. You talked in the beginning about technological challenges and how the old ways of trying to get people to respond to polling just aren’t working anymore. What does that mean for polling? Are you going to be able to solve these problems?
Stefan Hankin: I think “solving” is strong. I think we’re just always constantly adapting. It used to be super easy because everyone had a landline, and we knew everyone was pretty much going to be home after 6 p.m., and you could call until 9 p.m., and you were going to get a pretty good representative sample.
I don’t think we’re ever going to find the perfect way to get it right.
At the base of it, we’re dealing with human beings who are less rational and less predictable than we’d like to think. Even if we could get the perfect sample and get the right people on the proverbial phone, you’re still going to have instances where people do something slightly differently, or they change their mind at the last minute. Humans are interesting, but they’re also hard to pin down sometimes.
Anne Kim: Right. So humans are predict unpredictable and fallible, so that means polls are too. That’s the bottom line.
Stefan Hankin: Exactly.
