When PolitiFact falls short of its name

Over the weekend, Jon Stewart talked to Chris Wallace on “Fox News Sunday,” and made an important observation: “In polls,” Stewart said, in a surprisingly angry tone, “who are the most consistently misinformed media viewers? The most consistently misinformed? Fox. Fox viewers. Consistently. Every poll.”

I defended the claim, pointing to data from the Program on International Policy Attitudes, NBC/Wall Street Journal, and the Pew Research Center. PolitiFact checked the same claim and rated Stewart’s comment “false.” So, which is it? Did Stewart and I get it wrong or did PolitiFact?

My friend Chris Mooney argued the other day that Stewart’s original claim was correct and cited five (1, 2, 3, 4, 5) separate public opinion studies in support of Stewart’s argument. Yesterday, Chris gave the PolitiFact piece a closer look and found the fact-checking site’s analysis was clearly mistaken.

What Stewart obviously meant — and what I mean — is that when it comes to politicized, contested issues where the facts have been made murky due to political biases, it is Fox viewers who are the most likely to believe incorrect things — to fall prey to misinformation. A quintessential example of such an issue is global warming, or whether Saddam Hussein’s Iraq possessed weapons of mass destruction or was collaborating with Al Qaeda. There are many, many others.

To rebut Stewart’s claim, Politfact relied upon irrelevant and off-point studies. Thus, the site cited a number of Pew surveys that examine basic political literacy and relate it to what kind of media citizens consume. E.g., questions like whether people know “who the vice president is, who the president of Russia is, whether the Chief Justice is conservative, which party controls the U.S. House of Representatives and whether the U.S. has a trade deficit.”

Too few citizens know the answers to such basic questions — which is lamentable, but also irrelevant in the current context…. [T]he bulk of the studies cited by Politifact have nothing to do with whether Fox viewers believe the truth, or falsehoods, on politicized and contested issues.

Between us, Chris and I pointed to seven different pieces of independent research, all of which show Fox News viewers being the most misinformed news consumers. PolitiFact ignored five of the seven, tries to dismiss a sixth, and then points to other data with no meaningful relevance to the subject.

Independent fact-checking outlets can play an important part of the political discourse, but the process breaks down when the fact-checkers are themselves wrong. When an outlet puts “fact” in its name, the standards are especially high. In this case, PolitiFact fell far short.

Last night, Stewart apologized for his error. Jon, if you read this, you weren’t wrong. The fact-checkers were.

Update: Adam Serwer argued along the same lines, explaining this morning, “Politfact’s fact check evaluates a claim Stewart wasn’t actually trying to make.”