Not long after President Donald Trump declared the media the “enemy of the people,” the Republican Party released a “poll” on the Donald J Trump website, entitled the “Mainstream Media Accountability Survey.”
It’s fairly obvious that the results of this effort will be used to bolster the President’s attacks against the press. As propaganda, the results of this survey might work to rally the Trump faithful, but it in no way resembles a serious piece of opinion research. As a professional pollster, I can confidently state that this survey pushes the very definition of what a survey is and violates just about every principle of polling.
For starters, there is the question of who is being surveyed. No poll is valid unless the sample reflects an accurate cross-section of the population (what’s called a “representative sample”). The short version is that the people taking the survey should reflect the entire population you are trying to reach. So if you are conducting a national survey of registered voters, you want to make sure that the gender, age, ethnicity, location and other demographic details of the people you are surveying are proportional to the population at large. If 53 percent of registered voters are female, for example, the same should be true of your sample. On top of this, we also typically use “quotas” to make sure that we speak to the right number of women, registered voters under the age of 30, etc., so that the results are not biased. For example, having too many liberal Democrats in your sample will bias the results. Quotas also ensure that we are not just surveying people who are home and willing to answer their phone at a certain time.
Surveys posted online like this one will collect what is called a “non-representative sample” – in this case, a self-selected group of people visiting Trump’s website. This is not to say the results will be meaningless, but the key is how the results are presented. If the GOP reports that the results from this survey are based on responses by visitors to its website, then that is fine because it is not claiming the survey to be representative of all Americans. If however, it reports that “according to our poll, X percent of Americans think XYZ,” this would be false information.
The sampling problem is almost the least troubling part of this survey, unless basic grammar, biased questions, and butchering the English language don’t bother you. This partial annotation points out some of the other egregious flaws of this survey:
“Question 1: Do you believe the mainstream media has reported unfairly on our movement?”
Fatal flaw: Leading question. The use of the phrase “our movement” makes it clear what the “right” response should be. The correct way to ask this question would be: “How do you feel the media has reported on President Trump and the people who voted for him — fairly or unfairly?” The way it is worded now would be similar to asking: “Tom Brady, great quarterback or greatest quarterback?” Additionally, using the term “our movement” ensures that if a non-Trump supporter stumbled upon this survey, their responses are not welcome.
“Question 5: On which issues does the mainstream media do the worst job of representing Republicans? (Select as many that apply)
- Pro-Life values
- Individual liberty
- Foreign policy
- Second Amendment rights”
Once again the poll is leading by asking which is “worst” versus having you rate the media’s job performance from good to bad. Secondly, something can’t be “the worst” if you are allowing more than one response. Third, there is no “none of the above” option for those who feel that the media does a good job on covering these issues, which again points to a desire to cherry-pick the respondents taking the survey. Fourth, there are some words missing in this question. I am guessing they meant “representing the views of Republicans,” since it is not the job of the media to represent a party, as the question seems to suggest. Fifth, “conservatism” is not an issue, it’s a philosophy.
“Question 9: Do you trust the mainstream media to tell the truth about the Republican Party’s positions and actions?”
This question basically repeats the first question in the survey, unless the Trump campaign believes that the “movement” is different from the Republican Party, which is technically possible. If that were the case, however, the questions should be asked together and asked correctly. The right way to ask these questions should be: “How do you feel the media has reported on President Trump?” and then “How do you feel the media has reported on the Republican Party overall?”
Another problem is that this question is now being asked after questions which are anti-media in nature, which further biases the results. Going back to Tom Brady, if a survey were to ask a few questions about his accomplishments and then ask your opinion of him, it would likely generate a much more positive response than if the survey asked about Brady in a vacuum.” A legitimate survey would not give you information about a topic without expecting that information to bias the results.
“Question 12: Were you aware that a poll was released revealing that a majority of Americans actually supported President Trump’s temporary restriction executive order?”
Citing another poll in a poll, especially in this manner, is bad practice. First and most importantly, it biases the results. If you inform a respondent that a majority of people share an opinion, it is very hard for that person to not agree. Second of all this is what pollsters refer to as a “quiz question,” which should be avoided at all costs. The main reason is that if you answer “no, I was not aware” you are basically making the respondent feel like they are saying “no, I am not up on current events,” which no one really likes to admit.
“Question 13: Do you believe that political correctness has created biased news coverage on both illegal immigration and radical Islamic terrorism?”
Not a terrible question in theory, but given the selection bias we discussed, I have a sneaking suspicion that the answer to this question is going to make it into a tweet sometime soon
“Question 14: Do you believe that contrary to what the media says raising taxes does not create jobs?”
This question isn’t even a double negative. This is the rarely ever seen triple negative. Clear and concise questions are the key to getting good results. This question is confusing to say the least, and the results will be completely meaningless. There is a reason why people should always edit their polls before having respondents answer them.
Publicly released polls should always be looked at with a skeptical eye, since the person or organization releasing the poll generally is doing so as a means to an end, but this goes far beyond the realm of good practices regardless of how it is described.
At best, this survey is a clumsy and amateurish attempt to bolster Trump’s support among his base and lend some semblance of “legitimacy” to Trump’s campaign against the media. But at worst, it speaks to a more sinister desire by Trump and his allies to manipulate information to meet their political needs, even if it means abandoning any semblance of professionalism in how it gathers and presents data.