Do politicians understand how polls work?

A recent CBS News/New York Times poll showed 80% of Americans do not think their family is financially better than four years ago:

Just 20 percent of Americans feel their family’s financial situation is better today than it was four years ago. Another 37 percent say it is worse, and 43 percent say it is about the same.

When asked about these specific results, Harry Reid has this to say about polls in general:

“I’m not much of a pollster guy. As everyone knows, there isn’t a poll in America that had me having any chance of being re-elected, but I got re-elected,” he told TheDC.

“I think this poll is so meaningless. It is trying to give the American people an idea of what 300 million people feel by testing several hundred people. I think the poll is flawed in so many different ways including a way that questions were asked. I don’t believe in polls generally and specifically not in this one.”

The cynical take on this is that Reid and politicians in general like polls when they are supportive of their positions and don’t like them when they do not favor them. If this is true, then you might expect politicians to cite polls when they are good but to ignore them or even try to discredit them if they are bad.

But, I would like to ask a more fundamental question: are politicians any better than average Americans in understanding polls? Reid seems to suggest that this poll has two major problems: it doesn’t ask questions of enough people to really understand all Americans (a sampling issue) and the questions are poor which leads to biased answers (an issue of how the questions are worded). Is Reid right? From the information at the bottom of the CBS story about the poll, it seems pretty standard:

This poll was conducted by telephone from March 7-11, 2012 among 1009 adults nationwide.

878 interviews were conducted with registered voters, including 301 with voters who said they plan to vote in a Republican primary. Phone numbers were dialed from samples of both standard land-line and cell phones. The error due to sampling for results based on the entire sample could be plus or minus three percentage points. The margin of error for the sample of registered voters could be plus or minus three points and six points for the sample of Republican primary voters. The error for subgroups may be higher. This poll release conforms to the Standards of Disclosure of the National Council on Public Polls.

Yes, the number of respondents seems low to be able to talk about all Americans but this is how all major polls work: you select a representative sample based on standard demographic factors (gender, race, age, etc.) and then you estimate how close the survey results are to the actual results if we asked all American adults these questions. This is why all polls have a margin of error: if you ask less people, you are less confident in the generalizability of the results (which is why there is a larger 6% gap for the smaller Republican primary voters subgroup) and if you ask more people, you can be more confident (though the payoff of asking more people usually diminishes between 1200-1500 respondents so it is not worth asking more at some point).

I don’t think Reid sounds very good in this soundbite: he attacks the scientific basis of polls with common objections. While polls may not “feel right” and may contradict anecdotal or personal evidence, they can be done well and with a good sample of around 1,000 people, you can be confident that the results are generalizable to the American people. If Reid does understand how polls work, he could raise other issues. For example, he could insist that this is a one-time poll and you would want to measure this again and again to see how it changes (perhaps this is an unusual trough?) or you would want other polling organizations to ask the same question and triangulate the results between the surveys (like what Real Clear Politics does by taking averages of polls). Or he could suggest that this question doesn’t matter much because asking about four years ago is a rather arbitrary point and philosophically, does life always have to get better over time?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s