Tag Archives: surveys

Census 2020 to go digital and online

Posted on by

The Census Bureau is developing plans to go digital in 2020:

The bureau’s goal is that 55% of the U.S. population will respond online using computers, mobile phones or other devices. It will mark the first time (apart from a small share of households in 2000) that any Americans will file their own census responses online. This shift toward online response is one of a number of technological innovations planned for the 2020 census, according to the agency’s recently released operational plan. The plan reflects the results of testing so far, but it could be changed based on future research, congressional reaction or other developments…

The Census Bureau innovations are driven by the same forces afflicting all organizations that do survey research. People are increasingly reluctant to answer surveys, and the cost of collecting their data is rising. From 1970 to 2010, the bureau’s cost to count each household quintupled, to $98 per household in 2010 dollars, according to the GAO. The Census Bureau estimates that its innovations would save $5.2 billion compared with repeating the 2010 census design, so the 2020 census would cost a total of $12.5 billion, close to 2010’s $12.3 billion price tag (both in projected 2020 dollars)…

The only households receiving paper forms under the bureau’s plan would be those in neighborhoods with low internet usage and large older-adult populations, as well as those that do not respond online.

To maximize online participation, the Census Bureau is promoting the idea that answering the census is quick and easy. The 2010 census was advertised as “10 questions, 10 minutes.” In 2020, bureau officials will encourage Americans to respond anytime and anywhere – for example, on a mobile device while watching TV or waiting for a bus. Respondents wouldn’t even need their unique security codes at hand, just their addresses and personal data. The bureau would then match most addresses to valid security codes while the respondent is online and match the rest later, though it has left the door open to restrict use of this option or require follow-up contact with a census taker if concerns of fraud arise.

Perhaps the marketing slogan could be: “Do the Census online to save your own taxpayer dollars!”

It will be interesting to see how this plays out. I’m sure there will be plenty of tests to (1) make sure the people responding are matched correctly to their address (and that fraud can’t be committed); (2) the data collected is as accurate as going door to door and mailing out forms; and (3) the technological infrastructure is there to handle all the traffic. Even after going digital, the costs will be high and I’m guessing more people will ask why all the expense is necessary. Internet response rates to surveys are notoriously low so it may take a lot of marketing and reminders to get a significant percentage of online respondents.

But, if the Census Bureau can pull this off, it could represent a significant change for the Census as well as other survey organizations.

(The full 192 page PDF file of the plan is here.)

A need to better measure financial support and wealth passed to Millennials

Posted on by

A look at how race affects the financial support given by parents to Millennials includes this bit about measurement:

Shapiro said the numbers of Millennials receiving support from family are “absolutely underestimated” because many survey questions are not as methodical and specific as those a sociologist might ask. “As much as 90 percent of what you’ll hear isn’t picked up in the survey,” he said.

Shapiro’s more careful research found this:

Shapiro’s work pays special attention to the role of intergenerational family support in wealth building. He coined the term “transformative assets” to refer to any money acquired through family that facilitates social mobility beyond what one’s current income level would allow for. And it’s not that parents and other family members are exceptionally altruistic, either. “It’s how we all operate,” Shapiro said. “Resources tend to flow to people who are more needy.”

Racial disparity in transformative assets became especially striking to Shapiro during interviews with middle-class black Americans. “They almost always talk about financial help they give family members. People come to them,” Shapiro said. But when he asked white interviewees if they were lending financial support to family members, he said, “I almost always get laughter. They’re still getting subsidized.”…

To many Millennials, the small influxes of cash from parents are a lifeline, a financial relief they’re hard pressed to find elsewhere. To researchers, however, it’s both a symptom and an exacerbating factor of wealth inequality. In a 2004 CommonWealth magazine interview, Shapiro explained that gifts like this are “often not a lot of money, but it’s really important money. It’s a kind of money that allows families to obtain something for themselves and for their children that they couldn’t do on their own.”

Two quick thoughts:

  1. Americans tend not to like to talk about passing down wealth but decades of sociological research (as well as research from others) shows that it happens frequently and is quite advantageous for those who have wealth passed to them. I recommend looking at Shapiro and Oliver’s book Black Wealth/White Wealth.
  2. Polls like those cited here from USA Today could lead to lots of problems just because the measurement is not great. Why not ask better poll questions in the first place? I understand there are likely limits to how many questions can be asked (it is costly to ask more and longer questions) but I’d rather have sociologists and other social scientists handling this rather than the media.

Evangelicals recommend four beliefs that should identify them on surveys

Posted on by

The National Association of Evangelicals and LifeWay Research suggest evangelicals should be identified by agreeing with four beliefs:

  • The Bible is the highest authority for what I believe.
  • It is very important for me personally to encourage non-Christians to trust Jesus Christ as their Savior.
  • Jesus Christ’s death on the cross is the only sacrifice that could remove the penalty of my sin.
  • Only those who trust in Jesus Christ alone as their Savior receive God’s free gift of eternal salvation.

More on the reasons for these four:

The statements closely mirror historian David Bebbington’s classic four-point definition of evangelicalism: conversionism, activism, biblicism, and crucicentrism. But this list emphasizes belief rather than behavior, said Ed Stetzer, executive director of LifeWay Research.

“Affiliation and behavior can be measured in addition to evangelical beliefs, but this is a tool for researchers measuring the beliefs that evangelicals—as determined by the NAE—believe best define the movement,” he said.

A few quick thoughts on this:

  1. On one hand, it can be helpful for religious groups to identify what they see as unique to them. Outsiders may not pick up on these things. On the other hand, outsiders might see beliefs or other characteristics that mark evangelicals.
  2. Measuring religiosity involves a lot more than just beliefs. From later in the article:

    “Identity, belief, and behavior are three different things when it comes to being an evangelical,” McConnell said. “Some people are living out the evangelical school of thought but may not embrace the label. And the opposite is also true.”

    So this is just one piece of the puzzle. And I think sociologists (and other social scientists) have contributed quite a bit here in looking at how these particular theological views relate to other social behavior from race relations to voting to charitable activity and more.

  3. The suggestion here is that research shows the “correct” number of evangelicals identify with these four statements – identifying evangelicals in other ways seems to get to similar percentages as working with these four beliefs. Yet, I wonder how many evangelicals would name these four statements if asked what they believe. How exactly are these statements taught and passed on within evangelicalism?

Can religion not be fully studied with surveys or do we not use survey results well?

Posted on by

In a new book (which I have not read), sociologist Robert Wuthnow critiques the use of survey data to explain American religion:

Bad stats are easy targets, though. Setting these aside, it’s much more difficult to wage a sustained critique of polling. Enter Robert Wuthnow, a Princeton professor whose new book, Inventing American Religion, takes on the entire industry with the kind of telegraphed crankiness only academics can achieve. He argues that even gold-standard contemporary polling relies on flawed methodologies and biased questions. Polls about religion claim to show what Americans believe as a society, but actually, Wuthnow says, they say very little…

Even polling that wasn’t bought by evangelical Christians tended to focus on white, evangelical Protestants, Wuthnow writes. This trend continues today, especially in poll questions that treat the public practice of religion as separate from private belief. As the University of North Carolina professor Molly Worthen wrote in a 2012 column for The New York Times, “The very idea that it is possible to cordon off personal religious beliefs from a secular town square depends on Protestant assumptions about what counts as ‘religion,’ even if we now mask these sectarian foundations with labels like ‘Judeo-Christian.’”…

These standards are largely what Wuthnow’s book is concerned with: specifically, declining rates of responses to almost all polls; the short amount of time pollsters spend administering questionnaires; the racial and denominational biases embedded in the way most religion polls are framed; and the inundation of polls and polling information in public life. To him, there’s a lot more depth to be drawn from qualitative interviews than quantitative studies. “Talking to people at length in their own words, we learn that [religion] is quite personal and quite variable and rooted in the narratives of personal experience,” he said in an interview…

In interviews, people rarely frame their own religious experiences in terms of statistics and how they compare to trends around the country, Wuthnow said. They speak “more about the demarcations in their own personal biographies. It was something they were raised with, or something that affected who they married, or something that’s affecting how they’re raising their children.”

I suspect such critiques could be leveled at much of survey research: the questions can be simplistic, the askers of the questions can have a variety of motives and skills in developing useful survey questions, and the data gets bandied about in the media and public. Can surveys alone adequately address race, cultural values, politics views and behaviors, and more? That said, I’m sure there are specific issues with surveys regarding religion that should be addressed.

I wonder, though , if another important issue here is whether the public and the media know what to do with survey results. This book review suggests people take survey findings as gospel. They don’t know about the nuances of surveys or how to look at multiple survey questions or surveys that get at similar topics. Media reports on this data are often simplistic and lead with a “shocking” piece of information or some important trend (even if the data suggests continuity). While more social science projects on religion could benefit from mixed methods or by incorporating data from the other side (whether quantitative or qualitative), the public knows even less about these options or how to compare data. In other words, surveys always have issues but people are generally innumerate in knowing what to do with the findings.

Call for changing sex and gender questions on major surveys

Posted on by

Two sociologists argue that survey questions about sex and gender don’t actually tell us much:

Traditional understandings of sex and gender found in social surveys – such as only allowing people to check one box when asked “male” or “female” – reflect neither academic theories about the difference between sex and gender nor how a growing number of people prefer to identify, Saperstein argues in a study she coauthored with Grand Valley State University sociology professor Laurel Westbrook.

In their analysis of four of the largest and longest-running social surveys in the United States, the sociologists found that the surveys not only used answer options that were binary and static, but also conflated sex and gender. These practices changed very little over the 60 years of surveys they examined.

“Beliefs about the world shape how surveys are designed and data are collected,” they wrote. “Survey research findings, in turn, shape beliefs about the world, and the cycle repeats.”…

“Characteristics from race to political affiliation are no longer counted as binary distinctions, and possible responses often include the category ‘other’ to acknowledge the difficulty of creating a preset list of survey responses,” they wrote…The researchers suggest the following changes to social surveys:

  • Surveys must consistently distinguish between sex and gender.
  • Surveys should rethink binary categories.
  • Surveys need to incorporate self-identified gender and acknowledge it can change over time.

Surveys have to change as social understandings change. Measurement of race and ethnicity has changed quite a bit in recent decades with the Census considering changes for 2020.

It sounds like the next step would be to do a pilot study of alternatives – have a major survey include standard questions as well as new options – and then (1) compare results and (2) see how the new information is related to other information collected by the survey.

The bias toward one party in 2014 election polls is a common problem

Posted on by

Nate Silver writes that 2014 election polls were generally skewed toward Democrats. However, this isn’t an unusual problem in election years:

This type of error is not unprecedented — instead it’s rather common. As I mentioned, a similar error occurred in 1994, 1998, 2002, 2006 and 2012. It’s been about as likely as not, historically. That the polls had relatively little bias in a number of recent election years — including 2004, 2008 and 2010 — may have lulled some analysts into a false sense of security about the polls.

Interestingly, this year’s polls were not especially inaccurate. Between gubernatorial and Senate races, the average poll missed the final result by an average of about 5 percentage points — well in line with the recent average. The problem is that almost all of the misses were in the same direction. That reduces the benefit of aggregating or averaging different polls together. It’s crucially important for psephologists to recognize that the error in polls is often correlated. It’s correlated both within states (literally every nonpartisan poll called the Maryland governor’s race wrong, for example) and amongst them (misses often do come in the same direction in most or all close races across the country).

This is something we’ve studied a lot in constructing the FiveThirtyEight model, and it’s something we’ll take another look at before 2016. It may be that pollster “herding” — the tendency of polls to mirror one another’s results rather than being independent — has become a more pronounced problem. Polling aggregators, including FiveThirtyEight, may be contributing to it. A fly-by-night pollster using a dubious methodology can look up the FiveThirtyEight or Upshot or HuffPost Pollster or Real Clear Politics polling consensus and tweak their assumptions so as to match it — but sometimes the polling consensus is wrong.

It’s equally important for polling analysts to recognize that this bias can just as easily run in either direction. It probably isn’t predictable ahead of time.

The key to the issue here seems to be the assumptions that pollsters make before the election: who is going to turn out? Who is most energized? How do we predict who exactly is a likely voter? What percentage of a voting district identifies as Republican, Democrat, or Independent?

One thing that Silver doesn’t address is how this affects both perceptions of and reliance on such political polls. To have a large number of these polls lean in one direction (or lean in Republican directions in previous election cycles) suggests there is more work to do in perfecting such polls. All of this isn’t an exact science yet the numbers seem to matter more than ever; both parties jump on the results to either trumpet their coming success or to try to get their base out to reverse the tide. I’ll be curious to see what innovations are introduced heading into 2016 when the polls matter even more for a presidential race.

The need for “the endangered art of ethnography”

Posted on by

To highlight a new award for ethnography, a British sociologist explains what ethnography brings to the table:

Day after day, we are bombarded with survey evidence about the lives and the times of our fellow citizens. This, we are told, is how the unemployed regard benefit fraud, how the Scottish middle class react to the idea of independence, what black youths feel about the police’s use of stop and search. But much of this evidence is collected over a short period of time by professional pollsters who have little sense of the context in which they ask their tick-box questions.

Ethnography is a necessary supplement and often an important antidote to this form of research. It takes time: several of the researchers on our shortlist, for example, had spent two to three years studying, and often living within, a specific culture or subculture. It also allows questions to arise during the course of the research rather than being pre-programmed. So when Howard Parker embarked on his classic ethnographic study of delinquent youth in Liverpool (View from the Boys: A Sociology of Downtown Adolescents, 1974), he was faced by the official assumption that the young people in his sample were persistent offenders, hardened and even dangerous delinquents. Only after two years of hanging around with the boys was Parker able to conclude that this was far from the case. The boys’ offending was “mundane, trivial, petty, occasional, and very little of a threat to anyone except themselves”.

In a very similar manner, Heidi Hoefinger’s Sex, Love and Money in Cambodia: Professional Girlfriends and Transactional Relationships (2013), one of the studies shortlisted for the award, began from the common belief that encounters in the so-called “sex bars” of Cambodia would be entirely cash-based and essentially sleazy. Only after spending long periods of time talking to the women who worked in the bars and their male clients was she able to show that the relationships fashioned in the bars also had an important emotional component. Another stereotype had been exploded…

But the award is not only an affirmation of the significance of ethnography. What also prompted the five-year agreement between the BBC and the BSA was a wish to recognise the personal qualities that are needed in someone who is prepared to leave their family and friends to spend extended periods of time in a culture that will be uncomfortable, alien and, at times, downright dangerous. We all happily dip into different cultures: watch the skateboarders going through their paces under the Royal Festival Hall, check out the street style of the Rastas at the Notting Hill Carnival, wander through Chinatown during the New Year celebrations. But this is a far cry from suspending our own cherished values and embracing those of others for months and even years.

I wonder if ethnography gets less attention these days because we live in an era where:

1. We want research results more quickly. In comparison, surveys can be quickly administered and analyzed.

2. The big data of today allows for broad understandings and patterns. Ethnographies tend to be more particular.

3. We like “scientific” data that appears more readily available in surveys and experiments. Ethnographies appear more dependent on the researcher and subjective as opposed to “scientific.”

At the same time, there are other social forces that would promote ethnographies including more humane and holistic understandings of the world (particularly compared to the sterility of multiple-choice questions and quick numbers) as well as needing more time to study complex social phenomena.

Putin claims actions in Crimera based on sociological polls

Posted on by

Did sociology surveys provide cover for Vladimir Putin to incorporate Crimea? Here is one source:

Russian President Vladimir Putin said the final decision on the inclusion of Crimea and Sevastopol into Russia was made in regards to a sociological poll conducted in Crimea.

And another source:

“Russia did not prepare to incorporate Crimea, the decision on the republic’s accession to Russia was made only after data were received about the mood of local residents”, President Vladimir Putin said at a meeting with activists of the All-Russian People’s Front on Thursday…

The Republic of Crimea and Sevastopol, a city with a special status on the Crimean Peninsula, where most residents are Russians, signed reunification deals with Russia on March 18 after a referendum two days earlier in which an overwhelming majority of Crimeans voted to secede from Ukraine and join the Russian Federation.

While the international community is not likely to accept this reasoning, it does highlight an interesting issue: what happens when surveys show that people in one country would prefer to be in another? What then happens to national boundaries if there is strong public opinion to leave the current country? Perhaps the big difference here is that the people of Crimea didn’t revolt against Ukraine and seek to join Russia; Putin stepped in and pushed for this. But, there are likely lots of people groups in the world who might prefer to have their own country or to leave their current nation.

Another question might be regarding how this survey was conducted. I vaguely remember hearing similar figures that many in eastern Ukraine consider themselves to be Russian rather than Ukrainian while figures in the western side of the country were nearly opposite. How good are these sociological results?

The difficulty in wording survey questions about American education

Posted on by

Emily Richmond points out some of the difficulties in creating and interpreting surveys regarding public opinion on American education:

As for the PDK/Gallup poll, no one recognizes the importance of a question’s wording better than Bill Bushaw, executive director of PDK. He provided me with an interesting example from the September 2009 issue of Phi Delta Kappan magazine, explaining how the organization tested a question about teacher tenure:

“Americans’ opinions about teacher tenure have much to do with how the question is asked. In the 2009 poll, we asked half of respondents if they approved or disapproved of teacher tenure, equating it to receiving a “lifetime contract.” That group of Americans overwhelmingly disapproved of teacher tenure 73% to 26%. The other half of the sample received a similar question that equated tenure to providing a formal legal review before a teacher could be terminated. In this case, the response was reversed, 66% approving of teacher tenure, 34% disapproving.”

So what’s the message here? It’s one I’ve argued before: That polls, taken in context, can provide valuable information. At the same time, journalists have to be careful when comparing prior years’ results to make sure that methodological changes haven’t influenced the findings; you can see how that played out in last year’s MetLife teacher poll. And it’s a good idea to use caution when comparing findings among different polls, even when the questions, at least on the surface, seem similar.

Surveys don’t write themselves nor is the interpretation of the results necessarily straightforward. Change the wording or the order of the questions and results can change. I like the link to the list of “20 Questions A Journalist Should Ask About Poll Results” put out by the National Council on Public Polls. Our public life would be improved if journalists, pundits, and the average citizen would pay attention to these questions.

Sociologist who studies fear at and collects stats for a haunted house

Posted on by

Here is one way to put sociological training into practice: working for a haunted house.

Ms. Kerr’s equivalent of a coffee break was the ScareHouse in Etna, which bills itself as “Pittsburgh’s ultimate haunted house” and has earned accolades from national publications, trade magazines, horror movie directors and other outlets to buttress the claim…

A part-time professor at Pitt and Robert Morris University, Ms. Kerr’s appreciation for the macabre also led to a job at ScareHouse, where she’s worked since 2008 as an administrator, statistician and resident sociologist…

Though the ScareHouse, which opened in 1999, had long taken customer surveys, Ms. Kerr added a new dimension, he says, polling not just on what aspects of the haunted house worked but what customers’ fear most deeply…

Ms. Kerr’s book, based on her haunted house experiences, deals with “the real benefits of experiencing thrilling or scary materials.” Those can range from the endorphin and adrenaline rush and confidence boost of surviving a dicey encounter to the stronger bonds formed in social groups that experience a scary situation together. Of course, there’s an important caveat.

“To really enjoy thrilling situations, you have to know that you’re safe,” she added.

“Thanks for experiencing our haunted house – now please take our exit survey.” Yet, it sounds like an interesting place to collect data. It would be interesting to hear how generalizable the findings about fear at a haunted house might be to other situations.

I often tell my statistics and research methods students that all sorts of organizations, from NGOs to corporations to religious groups to governments, are looking to collect and analyze data. Here is another example I can use that might prove more interesting than some other options…