Tag Archives: bias

The bias toward one party in 2014 election polls is a common problem

Posted on by

Nate Silver writes that 2014 election polls were generally skewed toward Democrats. However, this isn’t an unusual problem in election years:

This type of error is not unprecedented — instead it’s rather common. As I mentioned, a similar error occurred in 1994, 1998, 2002, 2006 and 2012. It’s been about as likely as not, historically. That the polls had relatively little bias in a number of recent election years — including 2004, 2008 and 2010 — may have lulled some analysts into a false sense of security about the polls.

Interestingly, this year’s polls were not especially inaccurate. Between gubernatorial and Senate races, the average poll missed the final result by an average of about 5 percentage points — well in line with the recent average. The problem is that almost all of the misses were in the same direction. That reduces the benefit of aggregating or averaging different polls together. It’s crucially important for psephologists to recognize that the error in polls is often correlated. It’s correlated both within states (literally every nonpartisan poll called the Maryland governor’s race wrong, for example) and amongst them (misses often do come in the same direction in most or all close races across the country).

This is something we’ve studied a lot in constructing the FiveThirtyEight model, and it’s something we’ll take another look at before 2016. It may be that pollster “herding” — the tendency of polls to mirror one another’s results rather than being independent — has become a more pronounced problem. Polling aggregators, including FiveThirtyEight, may be contributing to it. A fly-by-night pollster using a dubious methodology can look up the FiveThirtyEight or Upshot or HuffPost Pollster or Real Clear Politics polling consensus and tweak their assumptions so as to match it — but sometimes the polling consensus is wrong.

It’s equally important for polling analysts to recognize that this bias can just as easily run in either direction. It probably isn’t predictable ahead of time.

The key to the issue here seems to be the assumptions that pollsters make before the election: who is going to turn out? Who is most energized? How do we predict who exactly is a likely voter? What percentage of a voting district identifies as Republican, Democrat, or Independent?

One thing that Silver doesn’t address is how this affects both perceptions of and reliance on such political polls. To have a large number of these polls lean in one direction (or lean in Republican directions in previous election cycles) suggests there is more work to do in perfecting such polls. All of this isn’t an exact science yet the numbers seem to matter more than ever; both parties jump on the results to either trumpet their coming success or to try to get their base out to reverse the tide. I’ll be curious to see what innovations are introduced heading into 2016 when the polls matter even more for a presidential race.

Getting around the anger or apathy students have for taking a sociology qualitative research methods class

Posted on by

In a review that describes how a book’s author practices “stealth sociology,” one sociologist describes how he tries to get his students excited about a qualitative research methods class:

Every semester, I teach a course in qualitative research methods. Revealing this at a dinner party or art opening invariably prompts sympathy, no response at all or variations on “Yuck! That was the worst course I ever had.”

Teaching what students dread and remember in anger robs my equilibrium. I tell students qualitative methods happen to be about stories, not numbers and measurements. And who doesn’t love a story and need one—many—daily? I merely teach ways to collect people’s stories, how to observe everyday life and narrate the encounter, and ways to discover stories “contained” in every human communication medium, from movies and tweets to objects of material culture, cars to casseroles.

Hearing this, students perk up. Momentarily. I continue in the liberal arts college spirit and urge students, “Bring to our class discussion and your research planning the skills you developed in English, literature and art classes.”

Hearing this, spirits deflate. Although some take to the freedom in narrative research methods, many students can’t give up the security they find in objective hypotheses, measured variables and reassuring numbers.

“How can we be objective about ourselves?” I argue. “How can anyone?”

Today in the wake of so-called identity studies, we sociologists and anthropologists expect each other to write ourselves into our research. We reveal our social addresses, identify our perspectives, and justify our intent. Sociologists and women’s studies scholars call it standpoint theory. No more pretense of the all-seeing-eye. No more fly on the wall invisibility.

As I think back on my experiences teaching lots of Intro to Sociology, Statistics, and Research Methods (involving both quantitative and qualitative methods), I have found the opposite to often be true: undergraduates more often understand the value or stories and narratives and have more difficulty thinking about scientifically studying people and society. Perhaps this is the result of a particular subculture that values personal relationships.

At the same time, sociologists collect stories in particular ways. It isn’t just about one person making an interpretation and other people can see very different things in the stories. This involves rigorous data collection and analysis by looking across cases. But, this is done without statistical tests and often having smaller samples (which can limit generalizability). Coding “texts” can be a time-consuming and involved process and interviews with people take quite a bit of work in crafting good questions, interacting with respondents in order to build rapport but not doing things to influence their answers, and then understanding and applying what you have heard. We know that we might bias the process, even in the selection of a research question, but we can find ways to limit this including utilizing multiple coders as well as sharing our work with others so they can check our findings and help us think through the implications.

Quick Review: League of Denial

Posted on by

I had a chance this past week to read the book League of Denial and see the PBS documentary by the same name. Some thoughts about the story of the NFL and concussion research (focusing mostly on the book which provides a more detailed narrative):

1. I know some fans are already complaining of “concussion fatigue” but it is hard to think of football the same way after hearing this story. For decades, we have held up players for their toughness and yet it may be ruining their brains.

2. The human story in all of this is quite interesting. This includes some of the former football players who have been driven to the edge by their football-related brain injuries. At the same time, the story amongst the doctors is also pretty fascinating, the chase for fame, publishing articles, and acquiring brains. Running through the whole book is this question of “who is really doing this research for the right reasons?” Even if the NFL research appears to be irrevocably tainted, are the researchers on the other side completely neutral or pure of heart?

3. The whole scientific process is laid out in the book (glossed over more in the documentary)…and I’m not sure how it fares. You have scientists fighting each other to acquire brains. You have peer-reviewed research – supposed to help prevent erroneous findings – that is viewed by many as erroneous from the start. You have scientists fighting for funding, an ongoing battle for all researchers as they must support their work and have their own livelihoods. In the end, consensus seems to be emerging but the book and documentary highlight the messy process it takes to get there.

4. The comparisons of the NFL to Big Tobacco seem compelling: the NFL tried to bury concussions research for a few decades and still doesn’t admit to a long-term impact of concussions on its players. One place where the comparison might break down for the general public (and scientific research could change this in the near future): the worst problems seem to be in long-time NFL players. When exactly does CTE start in the brains of football players? There is some evidence younger players, college or high school, might already have CTE but we need more evidence of this to be sure. If that is established, that perhaps kids as young as junior high already have CTE and that CTE is derived from regular hits at a young age (not the big knock-out blows), the link to Big Tobacco might be complete.

5. It is not really part of this story but I was struck again by how relatively little we know about the brain. Concussion research didn’t really take off until the 1990s, even as this had happened with football players for decades. (One sports area where it had been studied: boxing.) Much of this research is quite new and is a reminder that we humans don’t know as much as we might think.

6. This also provides a big reminder that the NFL is big business. Players seem the most aware of this: they can be cut at any time and an injury outside of their control could end their careers. The league and owners do not come off well here as they try to protect their holdings. The employees – the players – are generally treated badly: paid well if they perform but thrown aside otherwise. This may lead to a “better product” on the field but the human toll is staggering.

7. How exactly you change people’s opinions, both fans and players, regarding concussions will be fascinating to watch. It will take quite a shift among players from the tough-guy image to being willing to consider their futures more carefully. For fans, they may become more understanding as their favorite players consider what concussions might do to their lives. Will the NFL remain as popular? Hard to say though I imagine most fans this past weekend of football had little problem watching lots of gridiron action Saturday and Sunday.

Argument: sociologists not aware when they are gentrifiers

Posted on by

Two sociologists have published a paper that suggests some sociologists are gentrifiers themselves even as they critically address gentrification:

Few groups, Schlichtman contends, are more hypocritical than urbanists discussing gentrification. As he and fellow sociologist Jason Patch write in a rather unusual article in the International Journal of Urban and Regional Research, “many (dare we say most — ‘mainstream’ and critical) urbanists are gentrifiers themselves.” They mean this is an academic context, although the charge could reasonably be applied more broadly.

The point is not that these sociologists should stop talking about and researching the process of gentrification, but rather that they could do so with a self-awareness that might lead to a more nuanced understanding of what the word really means. Schlichtman and Patch, themselves, are owning up to the label. (The title of their article: “Gentrifier? Who Me? Interrogating the Gentrifier in the Mirror.”)…

Sociologists have backed themselves into a theoretical corner, he argues, with the caricature of the middle-class, latte-drinking urban pioneer whose inevitable taste for wine bars and boutiques drives up the rent and drives out the poor. If any middle-class presence in a diverse neighborhood is evidence of gentrification, he and Patch write, then it’s impossible for a middle-class person not to gentrify. “Is there any room,” they wonder, “for an ethical housing choice by the middle class?”

Is it necessarily unethical for a white middle-class family that wants to live in a racially and economically diverse neighborhood to move into one? How should that family reconcile that its presence on the block may signal unwelcome change to neighbors? As we’ve previously written, the idea of fair housing is as much about opening up high-opportunity neighborhoods to low-income people as it is enabling new investment in traditionally disinvested places, some of which will encourage new families to move in.

This leads me to a few thoughts:

1. I’m not sure there is much publishing space for sociologists to reflect on their own actions or own identities. For example, anthropologists are often open about their own personal backgrounds when writing an ethnography but sociologists are more tight-lipped. Perhaps this has to do with sociology’s more scientific aspirations.

2. I’ve seen how this plays out when talking about McMansions around sociologists. In that case, they are often quick to distance themselves from such homes.

3. What exactly do sociologists think about the middle class? Take the middle class choosing (or being pushed toward by policies and powerful interests) suburbia: this has been criticized by all sorts of academics for decades. What about the values and cultural preferences of the middle class? I remember one sociologist suggesting to a class that if they wanted to interact with regular Americans, they should go to Walmart. But, how many sociologists would want to go to Walmart or shop there themselves?

Using plagiarism detection software to examine anti-Muslim bias in post-9/11 news coverage

Posted on by

A new sociological study suggests mainstream media sources tended to rely on the rhetoric of certain anti-Muslim groups after 9/11:

“The vast majority of organisations competing to shape public discourse about Islam after the September 11 attacks delivered pro-Muslim messages, yet my study shows that journalists were so captivated by a small group of fringe organisations that they came to be perceived as mainstream,” the paper’s author, University of North Carolina assistant professor of sociology Christopher Bail, told Wired.co.uk…

Bail and his team used plagiarism detection software to compare 1,084 press releases produced by 120 different organisations with more than 50,000 television transcripts and newspaper articles produced between 2001 and 2008. The software picked up damning similarities between the releases and stories from news outlets including the New York Times, USA Today, the Washington Times, CBS News, CNN and Fox News Channel.

“We learned the American media almost completely ignored public condemnations of terrorist events by prominent Muslim organisations in the United States,” Bail told Wired.co.uk. “Inattention to these condemnations, combined with the emotional warnings of anti-fringe organisations, has created a very distorted representation of the community of advocacy organisations, think tanks, and religious groups competing to shape the representation of Islam in the American public sphere.”…

Bail’s paper, published in the American Sociological Review, is part of a wider study which will investigate how the influence of these fringe groups has spread beyond media and in to the real world, where doors have been opened to elite conservative social circles and conservative think tanks — the first steps to influencing public policy and national opinion. Bail touched upon this in the current study after analysing publicly available information on the organisations’ membership, which revealed troubling crossovers between fringe and mainstream organisations.

Four quick thoughts:

1. It sounds like there could be some importance influence of social networks. These fringe groups may be on the edges of public discourse but they have connections or means to which to reach more mainstream media sources. How much of this reporting is built on previous personal connections?

2. This sounds like a clever use of plagiarism software. Such software is intended to catch students in using published material incorrectly but it can also be used to track common quotes, phrases, and narratives.

3. In general, how much does the media today rely on press releases and reports from mainstream or fringe groups without interviews, fact-checking, and sorting through all the information?

4. Would a similar study involving elite liberal social circles and think tanks find similar things?

NYT lays out three options for how personal religious faith could influence sociological work

Posted on by

At the end of a column looking at this summer’s public debate over research findings from sociologist Mark Regnerus, the writer suggests there are three ways personal religious faith could influence a sociologist’s work:

So if there is not really a Christian method in sociology, but there is a role for a self-described Christian in sociology, as Dr. Regnerus once averred, then what is that role? One can imagine several answers.

First, the religious — or atheist, for that matter — sociologist might have a set of topics that she finds particularly relevant to her beliefs. Given their traditions’ emphasis on traditional family, for example, a conservative Catholic or evangelical Protestant could reasonably gravitate toward the study of family structure.

Second, a scholar might have faith that good research ultimately brings people to God or furthers his plans. A Christian historian might trust that even a modest study of the Spanish-American War, or of Rhode Island history, would do a small part to reveal the providential nature of all history.

Finally, a scholar might be a “Christian scholar” by virtue of the pride he takes in his faith, especially in the secular academy. Dr. Regnerus was a proud Christian witness, once upon a time. But these days he won’t discuss his faith, even with a Christian magazine. Two weeks ago, Christianity Today ran a lengthy interview with Dr. Regnerus in which he said nothing about his religious beliefs.

Option one presented here seems to be the one that would probably be most acceptable to the broader scientific community. Lots of researchers have personal interests that help guide them to particular areas of study but then we tend to assume (or hope), a la Weber’s arguments about value-free sociology, that the findings will not necessarily be influenced by these personal interests. At the same time, some might argue that completely separating personal life and research results may be a modernist dream.

I suspect options two and three wouldn’t get as much broad support.

It would also be interesting to see how this would play out if we weren’t talking about personal religious beliefs but other personal beliefs. For example, Jonathan Haidt has been looking at politics within social psychology and thinking about how these personal (and more collective) beliefs might influence a whole field.