Tag Archives: mixed methods

Researchers adjust as Americans say they are more religious when asked via phone versus responding online

Posted on by

Research findings suggest Americans answer questions about religiosity differently depending on the mode of the survey:

Photo by mentatdgt on Pexels.com

Researchers found the cause of the “noise” when they compared the cellphone results with the results of their online survey: social desirability bias. According to studies of polling methods, people answer questions differently when they’re speaking to another human. It turns out that sometimes people overstate their Bible reading if they suspect the people on the other end of the call will think more highly of them if they engaged the Scriptures more. Sometimes, they overstate it a lot…

Smith said that when Pew first launched the trend panel in 2014, there was no major difference between answers about religion online and over the telephone. But over time, he saw a growing split. Even when questions were worded exactly the same online and on the phone, Americans answered differently on the phone. When speaking to a human being, for example, they were much more likely to say they were religious. Online, more people were more comfortable saying they didn’t go to any kind of religious service or listing their religious affiliation as “none.”…

After re-weighting the online data set with better information about the American population from its National Public Opinion Reference Survey, Pew has decided to stop phone polling and rely completely on the online panels…

Pew’s analysis finds that, today, about 10 percent of Americans will say they go to church regularly if asked by a human but will say that they don’t if asked online. Social scientists and pollsters cannot say for sure whether that social desirability bias has increased, decreased, or stayed the same since Gallup first started asking religious questions 86 years ago.

This shift regarding studying religion highlights broader considerations about methodology that are always helpful to keep in mind:

  1. Both methods and people/social conditions change. More and more surveying (and other data collection) is done via the Internet and other technologies. This might change who responds, how people respond, and more. At the same time, actual religiosity changes and social scientists try to keep up. This is a dynamic process that should be expected to change over time to help researchers get better and better data.
  2. Social desirability bias is not the same as people lying to researchers or being dishonest with researchers. That implies an intentional false answer. This is more about context: the mode of the survey – phone or online – influences who the respondent is responding to. And with a human interaction, we might respond differently. In an interaction, we with impression management in mind where we want to be viewed in particular ways by the person with whom we are interacting.
  3. Studying any aspect of religiosity benefits from multiple methods and multiple approaches to the same phenomena under study. A single measure of church attendance can tell us something but getting multiple data points with multiple methods can help provide a more complete picture. Surveys have particular strengths but they are not great in other areas. Results from surveys should be put alongside other data drawn from interviews, ethnographies, focus groups, historical analysis, and more to see what consensus can be reached. All of this might be out of the reach of individual researchers or single research projects but the field as a whole can help find the broader patterns.

Researchers fact-checking their own ethnographic data

Posted on by

Toward the end of a long profile of sociologist Matthew Desmond is an interesting section regarding ethnographic methods:

Desmond has done an especially good job spelling out precisely how he went about his research and verified his findings, says Klinenberg. At the start of Evicted, an author’s note states that most of the events in the book took place between May 2008 and December 2009. Except where it says otherwise in the notes, Desmond writes, all events that happened between those dates were observed firsthand. Every quotation was “captured by a digital recorder or copied from official documents,” he adds. He also hired a fact-checker who corroborated the book by combing public records, conducting some 30 interviews, and asking him to produce field notes that verified a randomly selected 10 percent of its pages.

Desmond has been equally fastidious about taking himself out of the text. Unlike many ethnographic studies, including Goffman’s, his avoids the first person. He wants readers to react directly to the people in Evicted. “Ethnography often provokes very strong feelings,” he says. “So I wanted the book to do that. But not about me.”

Ethnographers should be more skeptical about their data, Desmond believes. In his fieldwork, for example, he saw women getting evicted at higher rates than men. But when he crunched the data, analyzing hundreds of thousands of court records, it turned out that was only the case in predominantly black and Latino neighborhoods. Women in white neighborhoods were not evicted at higher rates than men. The field had told him a half-truth.

Still, beyond acknowledging that the reception of Goffman’s book shaped his fact-checking, he will say nothing about the controversy. Even an old journalism trick — letting a silence linger, in the hope that an interviewee will fill it — fails to wring a quote from him. “This is such a good technique,” he says after a few seconds, “where you just kind of let the person talk.” Then he sips his Diet Coke, waiting for the next question.

This gets at some basic questions about what ethnography is. Should it be participant observation with a reflexive and involved researcher? Letting the research subjects speak for themselves with minimal interpretation? Should it involve fact-checking and verifying data? Each of these could have their merit and sociologists pursue different approaches. Contrasting the last two, for example, how people describe their own circumstances and understanding could be very important even if what is reported is not necessarily true. On the other hand, more and more ethnographies involve reflexive commentary from the researcher on how their presence and personal characteristics influenced the data collection and inteprretation.

It sounds to me like Desmond is doing some mixed methods work: starting with ethnographic data that he directly observes but then using secondary analysis (in the example above, using official records) to better understand both the micro level that he observed as well as the broader patterns. This means more work for each study but also more comprehensive data.