Tag Archives: sampling

Combating abysmally low response rates for political polling

Posted on by

One pollster describes the difficulty today in reaching potential voters:

Photo by Breakingpic on Pexels.com

As the years drifted by, it took more and more voters per cluster for us to get a single voter to agree to an interview. Between 1984 and 1989, when caller ID was rolled out, more voters began to ignore our calls. The advent of answering machines and then voicemail further reduced responses. Voters screen their calls more aggressively, so cooperation with pollsters has steadily declined year-by-year. Whereas once I could extract one complete interview from five voters, it can now take calls to as many as 100 voters to complete a single interview, even more in some segments of the electorate…

I offer my own experience from Florida in the 2020 election to illustrate the problem. I conducted tracking polls in the weeks leading up to the presidential election. To complete 1,510 interviews over several weeks, we had to call 136,688 voters. In hard-to-interview Florida, only 1 in 90-odd voters would speak with our interviewers. Most calls to voters went unanswered or rolled over to answering machines or voicemail, never to be interviewed despite multiple attempts.

The final wave of polling, conducted Oct. 25-27 to complete 500 interviews, was the worst for cooperation. We could finish interviews with only four-tenths of one percent from our pool of potential respondents. As a result, this supposed “random sample survey” seemingly yielded, as did most all Florida polls, lower support for President Trump than he earned on Election Day.

After the election, I noted wide variations in completion rates across different categories of voters, but nearly all were still too low for any actual randomness to be assumed or implied.

This is a basic Research Methods class issue: if you cannot collect a good sample, you are going to have a hard time reflecting reality for the population.

Here is the part I understand less. This is not a new issue. As noted above, response rates have been falling for decades. Part of it is new technology. Some of it involves new behavior, such as ignoring phone calls or distrust of political polling. The amount of polling and data collection that takes place now can lead to survey fatigue.

But, it is interesting that the techniques used to collect this data are roughly the same. Of course, it has moved from land lines to cell phones and perhaps even texting or recruited online pools of potential voters. The technology has changed some but the idea is similar in trying to reach out to a broad set of people and hope a representative enough sample responds.

Perhaps it is time for new techniques. The old ones have some advantages including the ability to relatively quickly reach a large number of people and researchers and consultants are used to these techniques. And I do not have the answers for what might work better. Researchers embedded in different communities who could collect data over time? Finding public spaces frequented by diverse populations and approaching people there? Working more closely with bellwhether or representative places or populations to track what is going on there?

Even with these low response rates, polling can still tell us something. It is not as bad as picking randomly or flipping a coin. Yet, it is not accurate enough in recent years. If researchers want to collect valid and reliable polling data in the future, new approaches may be in order.

The ongoing politics of the 2020 Census

Posted on by

The dicennial Census is not just a counting exercise; it is a political matter as this commentary suggests.

According to recent documents from the Census Bureau and the Government Accountability Office, the bureau plans to substantially cut back on door-to-door surveying and, instead, use the internet, the Post Office and other means to determine who is living where.

The bureau thinks the 2020 survey will cost $5.2 billion less than the last one (an estimate the GAO questions), but the accuracy could be called into question. There will also likely be worries about fraud because many of the conclusions will be drawn through “imputations” — educated guesses.

In fact, fraud could affect the House of Representatives elections for years to come if someone isn’t watching.

During a recent hearing before the House Oversight Committee, which maintained control over the Census Bureau after the Obama-Emanuel caper, a key technology officer for the 2020 decennial admitted that a fraud prevention system won’t be fully in place until just a few months before the polling starts.

If the Census Bureau – often led by sociologists and other social scientists who have expertise in collecting and analyzing data – is fraudulent because certain parties don’t like the result, what can be left alone?

Sampling and estimation alone does not have to be a problem. Just because the Census can’t reach everyone – and they have certainly tried at points – doesn’t mean that there is room for fraud. If done well, the estimates are made based on accurate samples – meaning they generally match the proportions of the total population – and responsible people reporting on this data will always note that there is not 100% certainty in the data.

Journalist tries to summarize 8% of teens not on social media

Posted on by

Most American teenagers use social media. So, how should a journalist go about finding about those who do not?

Such abstention from social media places him in a small minority in his peer group. According to a 2015 report by the Pew Research Center, 92% of American teenagers (ages 13-17) go online daily, including 24% who say they are on their devices “almost constantly.” Seventy-one percent use Facebook, half are on Instagram, and 41% are Snapchat users. And nearly three-quarters of teens use more than one social-networking site. A typical teen, according to Pew, has 145 Facebook friends and 150 Instagram followers…

Most of the social-media abstainers whom I interviewed aren’t technophobes. On the contrary, they have mobile phones that they use to contact their friends, usually via text. They are internet-savvy and fully enmeshed in popular culture. And they are familiar with social media. They just don’t like it…

For many nonusers of social media, the immediacy of face-to-face interaction trumps the filtered intimacy of Facebook and Instagram. “I do love seeing kids otherwise attached to their phones equalize when they’re cut off,” says Katy Kunkel of McLean, Va., whose four children range in age from 7 to 12. None of them are on social media. Especially during the summer months, she notes, “The kids recalibrate much quicker than adults. They find a tribe, then fun or trouble in trees and creeks…. They are way more active by default.”

The children themselves don’t often feel that they are missing out. Even though “almost 100%” of his friends are on social media, Brian O’Neill says that he can’t recall a time when something important happened in his social circle and he didn’t hear about it. “They let me know if something is going on,” he said. Ms. Furman’s experience is similar: “Sometimes I wouldn’t understand a specific joke everyone was telling, but 90% of the time, it’s not really worth it—it’s just a joke.”

Small subpopulations like this – the 8% of teenagers not using social media – can be attractive to journalists and social scientists alike: what causes them to go against the pervasive social norms? However, studying such small groups is often difficult. Large-scale surveys will not pick up many of them as there aren’t many to find.

This journalist went the route of interviews which can provide more detail but take more time. Still, how do you find such teenagers to interview when they are not easy to track down online? (Well, these teenagers might be active on other parts of the web without being on social media.) Perhaps a snowball sample was used or a quota sample. And, how many teenagers should you interview? The article quotes just several teenagers – perhaps more were interviewed – and tries to suggest that these quotes are representative of the 8% of teenagers not on social media.

Does this article correctly identify the reasons behind why a few teenagers are not using social media? It is hard to know but I’m not too hopeful based on a limited number of interviews with teenagers who may or may not represent those 8%. This may work for a journalist but I hope it wouldn’t pass academic muster.

Mapping England’s emotional mood in real-time

Posted on by

A group of researchers have developed a real-time map of England’s emotions based on Twitter:

The team, from Loughborough University, say it can scan up to 2,000 tweets a second and rate them for expressions of one of eight human emotions…

The team, from the university’s new Centre for Information Management, say the system can extract a direct expression of anger, disgust, fear, happiness, sadness, surprise, shame and confusion from each tweet.

The academics said that using the Emotive software to geographically evaluate any mass mood could help police to track potential criminal behaviour or threats to public safety.

It may be able to guide national policy on the best way to react to major incidents, they added.

There have been several projects like this in recent years. The algorithms to sort out all of the language must be intricate. But, I’m skeptical about two things. One is a sampling issue. Just how many people in England are using Twitter? In the United States, the figures about regular Twitter users are still quite low. You can map the moods of Twitter users but this doesn’t necessarily represent the larger population. The Twitter population probably trends younger. At the same time, responding to vocal responses on the web or on Twitter might be effective for public relations. A second issue is how exactly tracking moods could be used to help police. So police will be sent to places that show high concentrations of disgust or anger and pay less attention to places experiencing happiness? Or, such a system might alert police to trouble spots? I suspect it is more complicated than this yet I imagine such talk could make Twitter users nervous about how exactly their moods will be analyzed.

Pew reminds us that Twitter users are not representative of the US population

Posted on by

In looking at this story, I was led to a recent Pew study that compared the political leanings of Twitter to the political opinions of the general US population. One takeaway: the two populations are not the same.

The lack of consistent correspondence between Twitter reaction and public opinion is partly a reflection of the fact that those who get news on Twitter – and particularly those who tweet news – are very different demographically from the public.

The overall reach of Twitter is modest. In the Pew Research Center’s 2012 biennial news consumption survey, just 13% of adults said they ever use Twitter or read Twitter messages; only 3% said they regularly or sometimes tweet or retweet news or news headlines on Twitter.

Twitter users are not representative of the public. Most notably, Twitter users are considerably younger than the general public and more likely to be Democrats or lean toward the Democratic Party. In the 2012 news consumption survey, half (50%) of adults who said they posted news on Twitter were younger than 30, compared with 23% of all adults. And 57% of those who posted news on Twitter were either Democrats or leaned Democratic, compared with 46% of the general public. (Another recent Pew Research Center survey provides even more detail on who uses Twitter and other social media.)

In another respect, the Twitter audience also is broader than the sample of a traditional national survey. People under the age of 18 can participate in Twitter conversations, while national surveys are limited to adults 18 and older. Similarly, Twitter conversations also may include those living outside the United States.

Perhaps most important, the Twitter users who choose to share their views on events vary with the topics in the news. Those who tweeted about the California same-sex marriage ruling were likely not the same group as those who tweeted about Obama’s inaugural or Romney’s selection of Paul Ryan.

This leads to me to three thoughts:

1. What does this mean for the archiving of Twitter being undertaken by the Library of Congress? While it is still an interesting data source, Twitter provides a very small slice of U.S. opinion.

2. This is emblematic of larger issues with relying on new technologies to do research: who uses newer technologies is not the same as the U.S. population. This can be corrected for, as a recent article titled “A More Perfect Poll” suggests, and technologies can eventually filter throughout the whole U.S. population. In the meantime, researchers need to be careful about what they conclude.

3. So…what do we do about a comparison of a non-representative sample to a population? Pew seems to admit this:

While this provides an interesting look into how communities of interest respond to different circumstances, it does not reliably correlate with the overall reaction of adults nationwide.

This is an odd way to conclude a statistical report.

Controversy in using sampling for the dicennial Census

Posted on by

In a story about the resignation of sociologist Robert Groves as director of the United States Census Bureau, there is an overview of some of the controversy over Groves’ nomination. The issue: the political implications of using statistical sampling.

Dr. Robert M. Groves announced on Tuesday that he was resigning his position as director of the U.S. Census Bureau in order to take over as provost of Georgetown University. “I’m an academic at heart,” Groves told The Washington Post. He will leave the Bureau in August. Unlike some government officials who recently have had to resign under a cloud, such as Regina Dugan of DARPA and Martha Johnson of the General Services Administration, Groves received universal praise for the job he did directing the 2010 Census, a herculean task he completed on time and almost $2 billion under budget.
At the time of Groves’ nomination, Rep. Darrell Issa, (R-California), chairman of the House Committee on Oversight and Government Reform, said that he found it “an incredibly troubling selection that contradicts the administration’s assurances that the census process would not be used to advance an ulterior political agenda.” However, by the time Groves announced that he was leaving, Issa had changed his tune and issued a statement that “His tenure is proof that appointing good people makes a big difference.”
When President Barack Obama nominated Groves on April 2, 2009, he was viewed as a generally uncontroversial professor of sociology.  However, his nomination turned out to be contentious anyway because his support for using statistical sampling, a statistical method commonly used to correct for errors and biases in the census, raised the ire of Republican critics, who believed that sampling would benefit minorities and the poor, who generally vote Democratic…
A specialist in survey methodology and statistics, Groves was no stranger to the Census Bureau, whose decennial census is one of the world’s largest and most sophisticated statistical exercises.  Groves served there early in his career as a visiting statistician in 1982, and later as associate director of Statistical Design, Standards, and Methodology from 1990 to 1992.  It was during the latter period that Groves became embroiled in the controversy over the proposed use of statistical sampling to correct known biases and deficiencies in the Census head count.  Groves and others at the Census Bureau proposed using sampling techniques to correct an admitted 1.2% undercount in the 1990 Census, which failed to include millions of homeless, minority and poor persons mainly living in big cities, which lost millions of dollars in federal funds when Republican Commerce Secretary Robert Mosbacher vetoed the sampling proposal.

Considering Groves’ track record in sociology, I’m not surprised that he is now regarded to have done a good job in this position.

Perhaps this is a silly question in today’s world but does everything have to become politicized? Is the ultimate goal to get the most accurate count of American residents or do both parties simply assume that the other side wants to use the occasion for political gain? If you want to limit funding to cities based on population, why not go after this funding rather than try to skew the count?

Of course, this is not the first time that the dicennial Census has been politicized…

Another note: a sociologist apparently saved the government $2 billion! That alone should draw some attention.

Do politicians understand how polls work?

Posted on by

A recent CBS News/New York Times poll showed 80% of Americans do not think their family is financially better than four years ago:

Just 20 percent of Americans feel their family’s financial situation is better today than it was four years ago. Another 37 percent say it is worse, and 43 percent say it is about the same.

When asked about these specific results, Harry Reid has this to say about polls in general:

“I’m not much of a pollster guy. As everyone knows, there isn’t a poll in America that had me having any chance of being re-elected, but I got re-elected,” he told TheDC.

“I think this poll is so meaningless. It is trying to give the American people an idea of what 300 million people feel by testing several hundred people. I think the poll is flawed in so many different ways including a way that questions were asked. I don’t believe in polls generally and specifically not in this one.”

The cynical take on this is that Reid and politicians in general like polls when they are supportive of their positions and don’t like them when they do not favor them. If this is true, then you might expect politicians to cite polls when they are good but to ignore them or even try to discredit them if they are bad.

But, I would like to ask a more fundamental question: are politicians any better than average Americans in understanding polls? Reid seems to suggest that this poll has two major problems: it doesn’t ask questions of enough people to really understand all Americans (a sampling issue) and the questions are poor which leads to biased answers (an issue of how the questions are worded). Is Reid right? From the information at the bottom of the CBS story about the poll, it seems pretty standard:

This poll was conducted by telephone from March 7-11, 2012 among 1009 adults nationwide.

878 interviews were conducted with registered voters, including 301 with voters who said they plan to vote in a Republican primary. Phone numbers were dialed from samples of both standard land-line and cell phones. The error due to sampling for results based on the entire sample could be plus or minus three percentage points. The margin of error for the sample of registered voters could be plus or minus three points and six points for the sample of Republican primary voters. The error for subgroups may be higher. This poll release conforms to the Standards of Disclosure of the National Council on Public Polls.

Yes, the number of respondents seems low to be able to talk about all Americans but this is how all major polls work: you select a representative sample based on standard demographic factors (gender, race, age, etc.) and then you estimate how close the survey results are to the actual results if we asked all American adults these questions. This is why all polls have a margin of error: if you ask less people, you are less confident in the generalizability of the results (which is why there is a larger 6% gap for the smaller Republican primary voters subgroup) and if you ask more people, you can be more confident (though the payoff of asking more people usually diminishes between 1200-1500 respondents so it is not worth asking more at some point).

I don’t think Reid sounds very good in this soundbite: he attacks the scientific basis of polls with common objections. While polls may not “feel right” and may contradict anecdotal or personal evidence, they can be done well and with a good sample of around 1,000 people, you can be confident that the results are generalizable to the American people. If Reid does understand how polls work, he could raise other issues. For example, he could insist that this is a one-time poll and you would want to measure this again and again to see how it changes (perhaps this is an unusual trough?) or you would want other polling organizations to ask the same question and triangulate the results between the surveys (like what Real Clear Politics does by taking averages of polls). Or he could suggest that this question doesn’t matter much because asking about four years ago is a rather arbitrary point and philosophically, does life always have to get better over time?

Sociology grad student: “the Internet is a sociologist’s playground”

Posted on by

A sociology graduate student makes an interesting claim: “the Internet is a sociologist’s playground“:

The Internet is a sociologist’s playground, says Scott Golder, a graduate student in sociology at Cornell University. Although sociologists have wanted to study entire societies in fine-grained detail for nearly a century, they have had to rely primarily upon large-scale surveys (which are costly and logistically challenging) or interviews and observations (which provide rich detail, but for small numbers of subjects). Golder hopes that data from the social Web will provide opportunities to observe the detailed activities of millions of people, and he is working to bring that vision to fruition.  The same techniques that make the Web run—providing targeted advertisements and filtering spam—can also provide insights into social life. For example, he has used Twitter archives to examine how people’s moods vary over time, as well as how network structure predicts friendship choices. Golder came to sociology by way of computer science, studying language use in online communities and using the Web as a tool for collecting linguistic data. After completing a B.A. at Harvard and an M.S. at the MIT Media Lab, he spent several years in an industrial research lab before beginning his Ph.D. in sociology at Cornell.

I would think that having a background in computer science would be a big plus for a sociologist today. Lots of people want to study social networking sites like Facebook and work with the data available online. But I wonder if there still aren’t a few issues to overcome before we can really tap this information:

1. Do companies that have a lot of this data, places like Google and Facebook, want to open it up to researchers or would they prefer to keep the data in-house in order to make money?

2. How will Internet users respond to the interest researchers have in studying their online behavior if they are often not thrilled about being tracked by companies?

3. Has the sampling issue been resolved? In other words, one of the problems with web surveys or working with certain websites is that theses users are not representative of the total US population. So while internet activity has increased among the population as a whole, isn’t internet usage, particularly among those who use it most frequently, still skewed in certain directions?

4. Just how much does online activity reveal about offline activity? Do the two worlds overlap so much that this is not an issue or are there important things that you can’t uncover through online activity?

I would think some of these issues could be resolved and the sociologists who can really tap this growing realm will have a valuable head start.

Looking for a new area of study? Try Twitterology

Posted on by

If it is in the New York Times, Twitterology must be a viable area of academic study:

Twitter is many things to many people, but lately it has been a gold mine for scholars in fields like linguistics, sociology and psychology who are looking for real-time language data to analyze.

Twitter’s appeal to researchers is its immediacy — and its immensity. Instead of relying on questionnaires and other laborious and time-consuming methods of data collection, social scientists can simply take advantage of Twitter’s stream to eavesdrop on a virtually limitless array of language in action…

One criticism of “sentiment analysis,” as such research is known, is that it takes a naïve view of emotional states, assuming that personal moods can simply be divined from word selection. This might seem particularly perilous on a medium like Twitter, where sarcasm and other playful uses of language often subvert the surface meaning…

Still, the Twitterologists will continue to have a tough row to hoe in justifying their research to those who think that Twitter is a trivial form of communication. No less a figure than Noam Chomsky has taken Twitter to task recently for its “superficiality.”

For more sociological thoughts about Chomsky’s comments, see this post from a few days ago.

Here is my quick take on Twitterology: it has some potential for gathering quick, on-the-ground information. But there are two big issues that this article doesn’t address:

1. Are Twitter users representative of the whole population? Probably not. Twitter feeds might be good for studying very specific groups and movements.

2. How can one make causal arguments with Twitter data? If we had more information about Twitter users from profiles, this might be doable but Twitter is less about Facebook-style profiles. We then need studies that collect the information about Twitter users as well as their Twitter activity. If we want to ask questions like whether Twitter was instrumental or even helped cause the Arab Spring movements, we need more data.

Twitterology may be trendy at the moment but I think it has a ways to go before we can use it to tackle typical questions that sociologists ask.

Sociologists tracking “global mood swings” through Twitter

Posted on by

New social media platforms like Facebook and Twitter are ripe data sources. A new study in Science done by two sociologists examines the world’s emotions through Twitter:

The research team, led by Scott Golder, a PhD doctoral student in the field of sociology, and Professor of Sociology Michael Macy, tracked 2.4 million people in 84 different countries over the past two years. Clearly the team working on the project didn’t read through 2.4 million people’s tweets. Instead, they used a text analysis program that quantified the emotional content of 509 million tweets. Their results, featured in the paper “Diurnal and Seasonal Mood Tracks Work, Sleep and Day Length Across Diverse Cultures,” were published September 29 in Science.

The researchers found that work, sleep, and the amount of daylight we get really does affect things like our enthusiasm, delight, alertness, distress, fear, and anger. They concluded that people tweet more positive things early in the morning and then again around midnight. This could suggest that people aren’t very happy while they’re working since their happy tweets are at the beginning and end of the day. Saturday and Sunday also saw more positive tweets in general. The weekend showed these peaks at about 2 hours later, which accounts for sleeping in and staying out late.

Of course, all of the trends weren’t the same throughout every country. For example, the United Arab Emirates tend to work Sunday through Thursday, so their weekend tweets happened on Friday and Saturdays. The results also found that people who live in countries that get more daylight (closer to the equator) aren’t necessarily happier than people in countries that get less daylight (closer to the North and South Poles). It seems that only people who have a lot of daylight during the summer and then very little in the winter feel the affect of the change in seasons as much.

Clearly the results of the research aren’t perfect. There may be some people who only share positive things on Twitter, or some people who love to be cynical and use Twitter to complain about problems.

This sounds interesting and the resulting maps and charts are intriguing.  However, I would first ask methodological questions that would get at whether this is worthwhile data or not. Does this really reflect global moods? Or does this simply tell us something about Twitter users, who are likely not representative of the population at large?

Another article does suggest this study makes methodological improvements over two common ways studies look at emotions:

None of these results are particularly surprising, but Golder and Macy suggest that using global tweets allows them to confirm previous studies that only looked at small samples of American undergraduates who were not necessarily representative of the wider world. Traditional studies also require participants to recall their past emotions, whereas tweets can be gathered in real time.

These are good things: more immediate data and a wider sample beyond college undergraduates. But this doesn’t necessarily mean that the Twitter data is good data. The sample still probably skews toward younger people and those who have the technological means to be on Twitter consistently. Additionally, immediate emotions can tell us one thing but inquiring about longer-term satisfaction often tells us something else.

On the whole, this sounds like better data than we have before but until we have more universal Twitter usage, this data source will have significant limitations.