Tag Archives: methodology

SurveyMonkey made good 2014 election predictions based on experimental web polls

Posted on by

Here is an overview of some experimental work at SurveyMonkey in doing political polls ahead of the 2014 elections:

For this project, SurveyMonkey took a somewhat different approach. They did not draw participants from a pre-recruited panel. Instead, they solicited respondents from the millions of people that complete SurveyMonkey’s “do it yourself” surveys every day run by their customers for companies, schools and community organizations. At the very end of these customer surveys, they asked respondents if they could answer additional questions to “help us predict the 2014 elections.” That process yielded over 130,000 completed interviews across the 45 states with contested races for Senate or governor.

SurveyMonkey tabulated the results for all adult respondents in each state after weighting to match Census estimates for gender, age, education and race for adults — a relatively simple approach analogous to the way most pollsters weight random sample telephone polls. SurveyMonkey provided HuffPollster with results for each contest tabulated among all respondents as well as among subgroups of self-identified registered voters and among “likely voters — those who said they had either already voted or were absolutely certain or very likely to vote (full results are published here).

“We sliced the data by these traditional cuts so we could easily compare them with other surveys,” explains Jon Cohen, SurveyMonkey’s vice president of survey research, “but there’s growing evidence that we shouldn’t necessarily use voters’ own assessments of whether or not they’ll vote.” In future elections, Cohen adds, they plan “to dig in and build more sophisticated models that leverage the particular attributes of the data we collect.” (In a blog post published separately on Thursday, Cohen adds more detail about how the surveys were conducted).

The results are relatively straightforward. The full SurveyMonkey samples did very well in forecasting winners, showing the ultimate victor ahead in all 36 Senate races and missing in just three contests for Governor (Connecticut, Florida and Maryland)…

The more impressive finding is the way the SurveyMonkey samples outperformed the estimates produced by HuffPost Pollster’s poll tracking model. Our models, which are essentially averages of public polls, were based on all available surveys and calibrated to corresponded to results from the non-partisan polls that had performed well in previous elections. SurveyMonkey’s full samples in each state showed virtually no bias, on average. By comparison, the Pollster models overstated the Democrats’ margins against Republican candidates by an average 4 percent. And while SurveyMonkey’s margins were off in individual contests, the spread of those errors was slightly smaller than the spread of those for the Pollster averages (as indicated by the total error, the average of the absolute values of the error on the Democrat vs Republican margins).

The general concerns with web surveys involve obtaining a representative sample, either because it is difficult to identify the particular respondents who would meet the appropriate demographics or the survey is open to everyone. But, SurveyMonkey was able to produce good predictions for this past election cycle. Was it because they had (a) large enough samples that their data was a better approximation of the general population (they were able to reach a large number of people who use their services or (b) their weighting was particularly good?

The real test of this will be when a major organization, particularly a media outlet, solely utilizes web polls ahead of a major election. Given these positive results, perhaps we will see this in 2016. Yet, I imagine there may be some kinks to work out of the system or some organizations would only be willing to do that if they paired the web data with more traditional forms of polling.

Sociologists walking every block not just in New York City

Posted on by

A sociologist who walked every block of New York City drew attention but can you also learn from walking every block of Tyler, Texas? One sociologist explains:

Because of his interest in the community, Moody said, he has walked every street in Tyler twice. “It took 12 years to do it the first time; 11 years the second time,” Moody said…

“It (walking) is part of my research interests in society,” Moody, who taught sociology and other subjects at different times in six area colleges, said…

“I’m sure there are people who have lived here all their life and never been in parts of this town. If we understand and love one another, we will have a better community and I believe we will have more unity. We should never turn down an opportunity to learn from someone, whether it’s a homeless person, a wino or a wealthy billionaire,” Moody said…

n his walks around town, Moody said he has attended services or toured every church, synagogue and mosque, although he is a Southern Baptist.

Moody added that he has toured every hospital in Tyler, day care centers, nonprofit agencies, television and radio stations, the newspaper office and nursing homes as well as East Texas juvenile correctional facilities, state mental hospitals and prisons.

Two quick thoughts:

1. Tyler may not be New York City but it is still a sizable city of around 100,000 people. Sociology has a long history of community studies and the experiences of people in places like Tyler may hold a lot of interesting research potential. Yet, I’m not sure the field is really interested in the sorts of Middletown studies that once were more common.

2. People who really want to know their communities could use this method. This may be a sort of fad but not for those really invested in their community. I’m thinking of local politicians who claim this but this is typically based on their social connections. While these certainly matter, it is another thing to physically walk everything.

The American Dream and how Chicago magazine determined “Chicago’s Best Places to Live”

Posted on by

Chicago has a new list of the best places to live that includes 12 Chicago neighborhoods and 12 Chicago suburbs. Here are the factors the magazine used to identify these communities

First we looked at the factor that tends to be uppermost in the minds of families these days: safety. We eliminated from contention all community areas that notched violent crime rates higher than 7.0 offenses per 1,000 inhabitants last year (the city average: 9.3 per 1,000). That meant tossing out the Loop (9.9 per 1,000) and the historic South Side neighborhood of Pullman (11.2 per 1,000), for example. And we eliminated suburbs with violent crime rates above their county’s average—which removed from contention such otherwise appealing places as Evanston (2.2 per 1,000) and Oak Park (2.7 per 1,000), both in Cook County (2.1 per 1,000).

Then we turned to education. If a town or community lacks a public school whose students score above average on standardized tests, we dinged it. And because raising kids in an area that’s at least somewhat diverse is a goal of most parents, we nixed spots where more than 92 percent of residents are of any one race. (Bye-bye, Kenilworth, Western Springs, and Winnetka.)

For the places that remained, we looked at ease of transportation downtown, giving extra points to those that have several el stops and at least one Metra stop. (Places with outstanding schools and low rates of property crime also got bonus points.) And we considered how home prices in these places have fared in recent years compared with prices in neighboring areas, as well as whether buyers there can get good value for their money—which is not the same thing as paying the smallest amount. (For detailed price charts covering all Chicago suburbs and neighborhoods with at least 20 home sales in 2013, see this page with all the housing data.)

Finally, I hit the pavement to assess which spots possess those hard-to-define qualities that matter hugely when you’re looking for somewhere to live. Things like vibrancy (are there lots of bustling restaurants and shops?). Beauty (are there architecturally interesting buildings or just cookie-cutter developments?). Friendliness (does the community have a natural center that brings people together?). Is it, quite simply, a great place to call home?

So it boils down to safety, good schools, good transportation, higher than average housing values, and quality of life. Such measures are not uncommon on Best Places to Live lists.

However, it struck me upon reading this list that these traits tend to match a particular vision of a good community. If I may put it this way, it is a middle to upper-class ideal where kids are safe and nurtured and communities are protected from the difficulties of the world. It roughly matches the American Dream where people can live in small-town type places (even though most Americans do not live in such areas, they harbor the ideal of living in a tight-knit community – even if they may not want to contribute much to it) in relative comfort.

Just to take the two examples from DuPage County, Wheaton and Hinsdale, these communities partly derive their ranking from protecting this particular vision over the decades. Not everyone can move into these communities; it requires a certain amount of money as the affordable housing the communities discuss has much more to do with allowing senior citizens and recent college graduates to have somewhere to live rather than truly addressing low-income residents.

Maybe this methodology does reflect what many Americans want. The suburbs on the list are nice and the Chicago neighborhoods, while having more diversity, tend to be the more sought-after ones. At the same time, such lists could reinforce the notion that protected and wealthy places are the best ones, the ones we should all aspire to live.

The difficulties in finding out the most popular street name in the United States

Posted on by

FiveThirtyEight tries to find out the most common street name in the US and this leads to comparing Census information from 1993 with a Reddit user’s work:

The chart on Reddit that sparked your question looks very different from the 1993 list of most common street names from the Census Bureau.

Why, for example are there 3,238 extra Main streets in that chart compared with the census records in 1993? To find out, I got in touch with “darinhq,” whose name is Darin Hawley when he’s not producing charts on Reddit. After speaking to him, I think there are three explanations for the difference between his chart and the official data.

First, some new streets may have been built over the past 20 years (Hawley used 2013 census data to make his chart). Second, some streets may have changed their names: If a little town grows, it might change the name of its principal street from Tumbleweed Lane to Main Street.

Third, I don’t know how the Census Bureau produced its 1993 list (I asked, and a spokesperson told me the researcher who made it can’t recall his methodology), so Hawley might have simply used a different methodology to produce his chart. Because I wasn’t able to find any data on the frequency that American streets are renamed or the rate at which new streets are being built, I’m going to stake my money on this third explanation. Hawley told me that he counted “Main St N” and “N Main St” as two separate streets in his data. If the Census Bureau counted them as just one street, that could account for the difference.

That’s not the only executive decision Hawley made when he was summarizing this data. He set a minimum of how far away one Elm Street in Maine had to be from another Elm Street in Maine to qualify as two separate streets. That’s a problem because streets can break and resume in unexpected ways.

In other words, getting an answer requires making some judgment calls with the available data. While this is the sort of question that exemplifies the intriguing things we can all learn from the Internet, it is also a question that likely isn’t important enough to spend a lot of time with it. As an urban sociologist, this is an interesting question but what would I learn from the frequencies of street names? What hypothesis could I test? It might roughly tell us the names that Americans give to roads. What we value may just be reflected in these road names. For example, the Census data suggests that numbered streets and references to nature dominate the top 20. Does this mean we like order (a pragmatic approach) and idyllic yet vague nature terms (park, view, lake, tree names) over other things? Yet, the list has limitations as these communities and roads were built at different times, roads can be renamed, and we do have to make judgment calls about what specifies separate streets.

Two other thoughts:

1. The Census researcher who did this back in the early 1990s can’t remember the methodology. Why wasn’t it part of the report?

2. Is this something that would be best left up to marketers (who might find some advertising value in this) or GIS firms (who have access to comprehensive map data)?

Don’t see social media as representative of full populations

Posted on by

This should be obvious but computer scientists remind us that social media users are not representative populations:

One of the major problems with sites like Twitter, Pinterest or Facebook is ‘population bias’ where platforms are populated by a very narrow section of society.

Latest figures on Twitter suggest that just five per cent of over 65s use the platform compared with 35 per cent for those aged 18-29. Similarly far more men use the social networking site than women.

Instagram has a particular appeal to younger adults, urban dwellers, and non-whites.

In contrast, the picture-posting site Pinterest is dominated by females aged between 25 and 34. LinkedIn is especially popular among graduates and internet users in higher income households.

Although Facebook is popular across a diverse mix of demographic groups scientists warn that postings can be skewed because there is no ‘dislike’ button. There are also more women using Facebook than men, 76 per cent of female internet users use the site compared with 66 per cent of males.

Who does the data from social media represent? The people who use social media who, as pointed out above, tend to skew younger across the board and have other differences based on the service. Just because people are willing to put information out there doesn’t mean that it is a widely shared perspective, even if a Twitter account has millions of followers or a Facebook group has a lot of likes. Until we have a world where everyone participates in social media in similar ways and makes much of the same information public, we need to be careful about social media samples.

“Sociology’s most cited papers by decade”

Posted on by

Kieran Healy looks at the patterns among the most cited sociology papers:

Again, we’re looking at the Top 10 most-cited papers that were published in the 1950s, 1960s, and so on. This means that while the eleventh most-cited paper from the 1980s might outscore the fourth most-cited paper from the 1950s in terms of cumulative citations, the former does not appear here whereas the latter does. There are some striking patterns. One thing to notice is the rise of articles from the Annual Review of Sociology in the 2000s. Another is the increasing heterogeneity of outlets. Of the top ten papers written in the 1950s or before, seven appear in the American Sociological Review, two in the American Journal of Sociology, and one in Social Forces. (That is SF’s only entry in the list, as it happens.) ASR and AJS rule the 1960s, too. After that, though, there’s more variety. Strikingly, for the 2000s only one of the ten most-cited articles is from ASR and none is from AJS—a complete reversal of the pattern of the ‘50s and ‘60s. You can also see the long shadow of post-war university expansion and “Boomer Sociology”. The most-cited work from before 1970 is not nearly as widely cited as the most-cited work from the ‘70s and ‘80s, despite having been around longer. The drop-off in citation numbers in the Top 10s from the ‘90s and ‘00s is to be expected as those papers are younger. American dominance—or insularity—is also evident, as the only non-U.S. journal to make any of the lists is Sociology, and that was in the 1970s.

Turning to the subject matter of the papers, I think you can see the importance of articles whose main contribution is either a methodological technique or a big idea. There are fewer papers where a specific empirical finding is the main contribution. If you want to hang in there as one of the most-remembered papers from your decade, it seems, give people a good concept to work with or a powerful tool to use. Of course, it’s also true that people tend to have a lot of unread books lying around the house and unused drill attachments in the garage.

It is tempting to connect these two patterns in the data. To speculate: ASR and AJS remain amongst the journals with the very highest impact factors in the discipline. Publishing in them has become more important than ever to people’s careers. Yet the most-cited papers of the last two decades appeared elsewhere. These journals demand the papers they publish meet high standards in methods and ideally also innovate theoretically, along with making an empirical contribution to knowledge. That, together with a more competitive and professionalized labor market, produces very high-quality papers. But perhaps it also makes these journals less likely than in the past to publish purely technical or purely theoretical pieces, even though some papers of that sort will in the end have the most influence on the field.

Outlets like Sociological Methods and Research and Sociological Methodology now publish articles that might in the past have appeared in more general journals. Similarly, big-idea pieces that might once have gotten in at ASR or AJS may now be more likely to find a home at places like Theory and Society or Gender and Society. At the same time—perhaps because the state of theory in the field is more confused than that of methods—theoretical papers may also have been partially displaced by ARS articles that make an argument for some idea or approach, but under the shield of a topical empirical literature review. In a relatively fragmented field, it’s also easier for methodological papers to be more widely cited across a range of substantive areas than it is for a theory paper to do the same.

These seem like reasonable arguments to me. It is also interesting to see that a few subfields attract more attention, like theory and methodology but also social networks, social movements, gender, and cultural sociology, while other subfields are not among the most cited.

The bias toward one party in 2014 election polls is a common problem

Posted on by

Nate Silver writes that 2014 election polls were generally skewed toward Democrats. However, this isn’t an unusual problem in election years:

This type of error is not unprecedented — instead it’s rather common. As I mentioned, a similar error occurred in 1994, 1998, 2002, 2006 and 2012. It’s been about as likely as not, historically. That the polls had relatively little bias in a number of recent election years — including 2004, 2008 and 2010 — may have lulled some analysts into a false sense of security about the polls.

Interestingly, this year’s polls were not especially inaccurate. Between gubernatorial and Senate races, the average poll missed the final result by an average of about 5 percentage points — well in line with the recent average. The problem is that almost all of the misses were in the same direction. That reduces the benefit of aggregating or averaging different polls together. It’s crucially important for psephologists to recognize that the error in polls is often correlated. It’s correlated both within states (literally every nonpartisan poll called the Maryland governor’s race wrong, for example) and amongst them (misses often do come in the same direction in most or all close races across the country).

This is something we’ve studied a lot in constructing the FiveThirtyEight model, and it’s something we’ll take another look at before 2016. It may be that pollster “herding” — the tendency of polls to mirror one another’s results rather than being independent — has become a more pronounced problem. Polling aggregators, including FiveThirtyEight, may be contributing to it. A fly-by-night pollster using a dubious methodology can look up the FiveThirtyEight or Upshot or HuffPost Pollster or Real Clear Politics polling consensus and tweak their assumptions so as to match it — but sometimes the polling consensus is wrong.

It’s equally important for polling analysts to recognize that this bias can just as easily run in either direction. It probably isn’t predictable ahead of time.

The key to the issue here seems to be the assumptions that pollsters make before the election: who is going to turn out? Who is most energized? How do we predict who exactly is a likely voter? What percentage of a voting district identifies as Republican, Democrat, or Independent?

One thing that Silver doesn’t address is how this affects both perceptions of and reliance on such political polls. To have a large number of these polls lean in one direction (or lean in Republican directions in previous election cycles) suggests there is more work to do in perfecting such polls. All of this isn’t an exact science yet the numbers seem to matter more than ever; both parties jump on the results to either trumpet their coming success or to try to get their base out to reverse the tide. I’ll be curious to see what innovations are introduced heading into 2016 when the polls matter even more for a presidential race.

Mismatches in sociology grad student interests, job openings – 2013 edition

Posted on by

The ASA reports more job openings in sociology in recent years but the interests of sociology PhD graduates and the specializations of the new jobs don’t always line up. Here is part of the full table from the report (page six):

 

ASAJobAreas2013

There is some overlap here with most categories represented on both sides. However, other areas have some bigger differences:

1. Methodology – research methodology with 50 jobs and quantitative methods with 47 jobs and only 5 students with an interest in quantitative methodology. 23 jobs with statistics and 5 students. The figures for jobs in qualitative methods or ethnography better match the number of jobs available.

2. Another area of difference is criminology or criminal justice: 89 jobs in crime/delinquency and 70 in criminal justice with 66 students in criminology.

3. Sex and gender is particularly popular among students (108 interests) while only 31 jobs. (Granted, certain topics – like race, class, and gender – can easily cut across other subfields.)

4. Education has 83 students and 9 jobs.

This isn’t a complete analysis and these are the areas that struck me. Looking at methodology, it is a reminder that being interested in methods goes a long way on the job market as departments need people who can teach these skills and work with students in these areas.

The need for “the endangered art of ethnography”

Posted on by

To highlight a new award for ethnography, a British sociologist explains what ethnography brings to the table:

Day after day, we are bombarded with survey evidence about the lives and the times of our fellow citizens. This, we are told, is how the unemployed regard benefit fraud, how the Scottish middle class react to the idea of independence, what black youths feel about the police’s use of stop and search. But much of this evidence is collected over a short period of time by professional pollsters who have little sense of the context in which they ask their tick-box questions.

Ethnography is a necessary supplement and often an important antidote to this form of research. It takes time: several of the researchers on our shortlist, for example, had spent two to three years studying, and often living within, a specific culture or subculture. It also allows questions to arise during the course of the research rather than being pre-programmed. So when Howard Parker embarked on his classic ethnographic study of delinquent youth in Liverpool (View from the Boys: A Sociology of Downtown Adolescents, 1974), he was faced by the official assumption that the young people in his sample were persistent offenders, hardened and even dangerous delinquents. Only after two years of hanging around with the boys was Parker able to conclude that this was far from the case. The boys’ offending was “mundane, trivial, petty, occasional, and very little of a threat to anyone except themselves”.

In a very similar manner, Heidi Hoefinger’s Sex, Love and Money in Cambodia: Professional Girlfriends and Transactional Relationships (2013), one of the studies shortlisted for the award, began from the common belief that encounters in the so-called “sex bars” of Cambodia would be entirely cash-based and essentially sleazy. Only after spending long periods of time talking to the women who worked in the bars and their male clients was she able to show that the relationships fashioned in the bars also had an important emotional component. Another stereotype had been exploded…

But the award is not only an affirmation of the significance of ethnography. What also prompted the five-year agreement between the BBC and the BSA was a wish to recognise the personal qualities that are needed in someone who is prepared to leave their family and friends to spend extended periods of time in a culture that will be uncomfortable, alien and, at times, downright dangerous. We all happily dip into different cultures: watch the skateboarders going through their paces under the Royal Festival Hall, check out the street style of the Rastas at the Notting Hill Carnival, wander through Chinatown during the New Year celebrations. But this is a far cry from suspending our own cherished values and embracing those of others for months and even years.

I wonder if ethnography gets less attention these days because we live in an era where:

1. We want research results more quickly. In comparison, surveys can be quickly administered and analyzed.

2. The big data of today allows for broad understandings and patterns. Ethnographies tend to be more particular.

3. We like “scientific” data that appears more readily available in surveys and experiments. Ethnographies appear more dependent on the researcher and subjective as opposed to “scientific.”

At the same time, there are other social forces that would promote ethnographies including more humane and holistic understandings of the world (particularly compared to the sterility of multiple-choice questions and quick numbers) as well as needing more time to study complex social phenomena.

Hard to measure school shootings

Posted on by

It is difficult to decide on how to measure school shootings and gun violence:

What constitutes a school shooting?

That five-word question has no simple answer, a fact underscored by the backlash to an advocacy group’s recent list of school shootings. The list, maintained by Everytown, a group that backs policies to limit gun violence, was updated last week to reflect what it identified as the 74 school shootings since the massacre in Newtown, Conn., a massacre that sparked a national debate over gun control.

Multiple news outlets, including this one, reported on Everytown’s data, prompting a backlash over the broad methodology used. As we wrote in our original post, the group considered any instance of a firearm discharging on school property as a shooting — thus casting a broad net that includes homicides, suicides, accidental discharges and, in a handful of cases, shootings that had no relation to the schools themselves and occurred with no students apparently present.

None of the incidents rise to the level of the massacre that left 27 victims, mostly children, dead in suburban Connecticut roughly 18 months ago, but multiple reviews of the list show how difficult quantifying gun violence can be. Researcher Charles C. Johnson posted a flurry of tweets taking issue with incidents on Everytown’s list. A Hartford Courant review found 52 incidents involving at least one student on a school campus. (We found the same, when considering students or staff.) CNN identified 15 shootings that were similar to the violence in Newtown — in which a minor or adult was actively shooting inside or near a school — while Politifact identified 10.

Clearly, there’s no clean-cut way to quantify gun violence in the nation’s schools, but in the interest of transparency, we’re throwing open our review of the list, based on multiple news reports per incident. For each, we’ve summarized the incident and included casualty data where available.

This is a good example of the problems of conceptualization and operationalization. The idea of a “school shooting” seems obvious until you start looking at a variety of incidents and have to decide whether they hang together as one definable phenomenon. It is interesting here that the Washington Post then goes on to provide more information about each case but doesn’t come down on any side.

So how might this problem be solved? In the academic or scientific world, scholars would debate this through publications, conferences, and public discussions until some consensus (or at least some agreement about the contours of the argument) emerges. This takes time, a lot of thinking, and data analysis. This runs counter to more media or political-driven approaches that want quick, sound bite answers to complex social problems.