Tag Archives: methodology

Sociologists = people who look at “boring data compiled during endless research”

Posted on by

If this is how a good portion of the public views what sociologists do, sociologists may be in trouble:

Anthony Campolo is a sociologist by trade, used to looking at boring data compiled during endless research.

Data collection and analysis may not be glamorous but a statement like this suggests sociologists may have some PR issues. Data collection and analysis are often time consuming and even tedious. But, there are reasons for working so hard to get data and do research: so sociologists can make substantiated claims about how the social world works. Without rigorous methods, sociologists would just be settling for interpretation, opinion, or anecdotal evidence. For example, we might be left with stories like that of a homeless man in Austin, Texas who was “testing”  which religious groups contributed more money to him. Of course, his one case tells us little to nothing.

Perhaps this opening sentence should look something like this: time spent collecting and analyzing data will pay off in stronger arguments.

 

Spreadsheet errors, austerity, ideology, and social science

Posted on by

The graduate student who found some spreadsheet errors in an influential anti-austerity paper discusses what happened. Here is part of the conversation about the process of finding this error:

Q. You say, don’t you, that their use of data was faulty?

A. Yes. The terms we used about their data—”selective” and “unconventional”—are appropriate ones. The reasons for the choices they made needed to be given, and there was nowhere where they were.

Q. And how about their claim that your findings support their thesis that growth slows as debt rises?

A. That is not our interpretation of our paper, at all. If you read their paper, it’s interesting how they handle causality. They waffle between strong and weak claims. The weak claim is that it’s just a negative association. If that’s all they claim, then it’s not really relevant for policy. But they also make a strong claim, more in public than in the paper, that there’s causality going from high debt to drops in growth. They haven’t been obvious about that…

Q. Paul Krugman wrote in The New York Times that your work confirms what many economists have long intuitively thought. Was that your intuition?

A. Yes. I just thought it was counterintuitive when I first saw their claim. It wasn’t plausible.

Q. This is more than a spreadsheet error, then?

A. Yes. The Excel error wasn’t the biggest error. It just got everyone talking about this. It was an emperor-has-no-clothes moment.

This would make for a good case study in a methodology class in the social sciences: how much of this is about actual data errors versus different interpretations? You have people who are clearly staking out space on either side of a policy discussion and it is a bit unclear how much does this color their interpretation of “facts”/data. I suspect some time will help sort this out – if the spreadsheet was indeed wrong, shouldn’t this lead to a correction or a retraction?

I do like the fact that the original authors were willing to share their data – this is something that could happen more often in the social sciences and give people the ability to look at the data for themselves.

Argument: Big Data reduces humans to something less than human

Posted on by

One commentator suggests Big Data can’t quite capture what makes humans human:

I have been browsing in the literature on “sentiment analysis,” a branch of digital analytics that—in the words of a scientific paper—“seeks to identify the viewpoint(s) underlying a text span.” This is accomplished by mechanically identifying the words in a proposition that originate in “subjectivity,” and thereby obtaining an accurate understanding of the feelings and the preferences that animate the utterance. This finding can then be tabulated and integrated with similar findings, with millions of them, so that a vast repository of information about inwardness can be created: the Big Data of the Heart. The purpose of this accumulated information is to detect patterns that will enable prediction: a world with uncertainty steadily decreasing to zero, as if that is a dream and not a nightmare. I found a scientific paper that even provided a mathematical model for grief, which it bizarrely defined as “dissatisfaction.” It called its discovery the Good Grief Algorithm.

The mathematization of subjectivity will founder upon the resplendent fact that we are ambiguous beings. We frequently have mixed feelings, and are divided against ourselves. We use different words to communicate similar thoughts, but those words are not synonyms. Though we dream of exactitude and transparency, our meanings are often approximate and obscure. What algorithm will capture “the feel of not to feel it?/?when there is none to heal it,” or “half in love with easeful Death”? How will the sentiment analysis of those words advance the comprehension of bleak emotions? (In my safari into sentiment analysis I found some recognition of the problem of ambiguity, but it was treated as merely a technical obstacle.) We are also self-interpreting beings—that is, we deceive ourselves and each other. We even lie. It is true that we make choices, and translate our feelings into actions; but a choice is often a coarse and inadequate translation of a feeling, and a full picture of our inner states cannot always be inferred from it. I have never voted wholeheartedly in a general election.

For the purpose of the outcome of an election, of course, it does not matter that I vote complicatedly. All that matters is that I vote. The same is true of what I buy. A business does not want my heart; it wants my money. Its interest in my heart is owed to its interest in my money. (For business, dissatisfaction is grief.) It will come as no surprise that the most common application of the datafication of subjectivity is to commerce, in which I include politics. Again and again in the scholarly papers on sentiment analysis the examples given are restaurant reviews and movie reviews. This is fine: the study of the consumer is one of capitalism’s oldest techniques. But it is not fine that the consumer is mistaken for the entirety of the person. Mayer-Schönberger and Cukier exult that “datafication is a mental outlook that may penetrate all areas of life.” This is the revolution: the Rotten Tomatoes view of life. “Datafication represents an essential enrichment in human comprehension.” It is this inflated claim that gives offense. It would be more proper to say that datafication represents an essential enrichment in human marketing. But marketing is hardly the supreme or most consequential human activity. Subjectivity is not most fully achieved in shopping. Or is it, in our wired consumerist satyricon?

“With the help of big data,” Mayer-Schönberger and Cukier continue, “we will no longer regard our world as a string of happenings that we explain as natural and social phenomena, but as a universe comprised essentially of information.” An improvement! Can anyone seriously accept that information is the essence of the world? Of our world, perhaps; but we are making this world, and acquiescing in its making. The religion of information is another superstition, another distorting totalism, another counterfeit deliverance. In some ways the technology is transforming us into brilliant fools. In the riot of words and numbers in which we live so smartly and so articulately, in the comprehensively quantified existence in which we presume to believe that eventually we will know everything, in the expanding universe of prediction in which hope and longing will come to seem obsolete and merely ignorant, we are renouncing some of the primary human experiences. We are certainly renouncing the inexpressible. The other day I was listening to Mahler in my library. When I caught sight of the computer on the table, it looked small.

I think there are a couple of arguments possible about the limitations of big data and Wieseltier is making a particular argument. He does not appear to be saying that big data can’t predict or model human complexity. And fans of big data would probably say the biggest issue is that we simply don’t have enough data yet and we are developing better and better models. In other words, our abilities and data will eventually catch up to the problem of complexity. But I think Wieseltier is arguing something else: he, along with many others, does not want humans to be reduced to information. Even if we had the best models, it is one thing to see people as complex individuals and yet another to say they are simply another piece of information. Doing the latter takes away the dignity of people. Reducing people to data means we stop seeing people as people that can change their minds, be creative, and confound predictions.

It will be interesting to see how this plays out in the coming years. I think this is the same fear many people have about statistics. Particularly in our modern world where we see ourselves as sovereign individuals, describing statistical trends to people strikes them as reducing their own agency and negating their own experiences. Of course, this is not what statistics is about and something more training in statistics could help change. But, how we talk about data and its uses might go a long way to how big data is viewed in the future.

Pollster provides concise defense of polls

Posted on by

The chief pollster for Fox News defends polls succinctly here. The conclusion:

Likewise, we don’t need to contact every American — more than 230 million adults — to find out what the public is thinking. Suffice it to say that with proper sampling and random selection of respondents so that every person has an equal chance of being contacted, a poll of 800-1,000 people provides an incredibly accurate representation of the country as a whole. It’s a pretty amazing process if you think about it.

Still, many people seem to have a love-hate relationship with polls. Even if they enjoy reading the polls, some people can turn into skeptics if they personally don’t feel the same as the majority. Maybe they don’t even know anyone who feels the same as the majority.  Yet assuming everyone shares your views and those of your friends and neighbors would be like the cook skimming a taste from just the top of the pot without stirring the soup first.

Basic but a staple of many a statistics and research methods course. Unfortunately, more people need this kind of education in a world where statistics are becoming more and more common.

Lincoln, Nebraska #1 city in well-being

Posted on by

A new survey from Gallup and Healthways puts Lincoln, Nebraska as the number one city in the U.S. for well-being:

Lincoln, Neb., had the highest Well-Being Index score (72.8) in the U.S. across the 189 metropolitan areas that Gallup and Healthways surveyed in 2012. Also in the top 10 are Boulder, Colo.; Provo-Orem, Utah; Ann Arbor, Mich.; Honolulu, Hawaii; Fort Collins-Loveland, Colo.; and Burlington-South Burlington, Vt…

At 60.8, Charleston, W.Va., had the lowest Well-Being Index score, displacing Huntington-Ashland, W.Va.-Ky.-Ohio, which held this position the two previous years. Huntington-Ashland’s score of 61.2 is up from 58.1 in 2010, which is the lowest score on record for any metro area across five years of data collection. Mobile, Ala.; Utica-Rome, N.Y.; Hickory-Lenoir-Morganton, N.C.; and Fort Smith, Ark.-Okla.; join Charleston and Huntington-Ashland as frequent occupants of the bottom 10 list each year…

Washington, D.C.-Arlington-Alexandria, Va.-Md.-W.Va., residents reported the highest wellbeing among the nation’s 52 largest metropolitan areas, defined as those with 1 million residents or more, followed by San Francisco-Oakland-Fremont, Calif. These two metros have been in the top five among large metro areas in each of the past five years…

The Gallup-Healthways Well-Being Index score is an average of six sub-indexes, which individually examine life evaluation, emotional health, work environment, physical health, healthy behaviors, and access to basic necessities. The overall score and each of the six sub-index scores are calculated on a scale from 0 to 100, where a score of 100 represents the ideal. Gallup and Healthways have been tracking these measures daily since January 2008.

Interesting as there are more cities from the Great Plains and Midwest than I expected.

A few thoughts about the methodology:

1. After all is added up across these six measures, there isn’t much variation between the top and the bottom. Lincoln had the highest score at 72.8 and Charleston had the lowest at 60.8. So on a scale of 0 to 100, the range was just 12. This suggests there is not much variation in these measures and that this index may not tell us a whole lot. Are Americans simply generally optimistic about these topics or are they realistically not that different across cities?

2. What exactly does Gallup and Healthways do with this information that it requires daily polling? This is not a small sample:

Results are based on telephone interviews conducted as part of the Gallup-Healthways Well-Being Index survey Jan. 2-Dec. 29, 2012, with a random sample of 353,563 adults, aged 18 and older, living in all 50 U.S. states and the District of Columbia, selected using random-digit-dial sampling.

Perhaps there is some marketing edge to this surveying or it is related to some big research project.

Bonus well-being info: for occupations, doctors and then K-12 teachers lead the way and manufacturing-production workers and then transportation workers are at the bottom.

Social network of email between countries shows homophily between culturally similar nations

Posted on by

A new study of email traffic between countries finds some patterns:

The Internet was supposed to let us bridge continents and cultures like never before. But after analyzing more than 10 million e-mails from Yahoo! mail, a team of computer researchers noticed an interesting phenomenon: E-mails tend to flow much more frequently between countries with certain economic and cultural similarities.

Among the factors that matter are GDP, trade, language, non-Commonwealth colonial relations, and a couple of academic-sounding cultural metrics, like power-distance, individualism, masculinity and uncertainty…

To this point, of course, the study amounts to little more than very interesting trivia. The real conclusion comes toward the end, when the researchers posit it as possible evidence for Samuel Huntington’s controversial “Clash of Civilizations” theory. From the paper:

In this respect we cautiously assign a level of validity to Huntington’s contentions, with a few caveats. The ?rst issue was already mentioned – overlap between civilizations and other factors contributing to countries’ level of association. Huntington’s thesis is clearly re?ected in the graph presented in Figure 3, but some of these civilizational clusters are found to be explained by other factors in Table 5. The second limitation concerns the fact that we investigated a communication network. There is no necessary “clash” between countries that do not communicate, and Huntington’s thesis was concerned primarily with ethnic con?ict.

Interesting what can be done with data from more than 10 million emails.

I wonder if it is even worth doing this analysis at the country level. Isn’t this too broad? Aren’t there likely to be important patterns within and across countries that are obscured by this broader lens?

Another possible issue: is Yahoo mail a representative sample of emails or does it provide a particular slice of of email traffic? I would assume it involves more personal email as opposed to business activity.

Pew reminds us that Twitter users are not representative of the US population

Posted on by

In looking at this story, I was led to a recent Pew study that compared the political leanings of Twitter to the political opinions of the general US population. One takeaway: the two populations are not the same.

The lack of consistent correspondence between Twitter reaction and public opinion is partly a reflection of the fact that those who get news on Twitter – and particularly those who tweet news – are very different demographically from the public.

The overall reach of Twitter is modest. In the Pew Research Center’s 2012 biennial news consumption survey, just 13% of adults said they ever use Twitter or read Twitter messages; only 3% said they regularly or sometimes tweet or retweet news or news headlines on Twitter.

Twitter users are not representative of the public. Most notably, Twitter users are considerably younger than the general public and more likely to be Democrats or lean toward the Democratic Party. In the 2012 news consumption survey, half (50%) of adults who said they posted news on Twitter were younger than 30, compared with 23% of all adults. And 57% of those who posted news on Twitter were either Democrats or leaned Democratic, compared with 46% of the general public. (Another recent Pew Research Center survey provides even more detail on who uses Twitter and other social media.)

In another respect, the Twitter audience also is broader than the sample of a traditional national survey. People under the age of 18 can participate in Twitter conversations, while national surveys are limited to adults 18 and older. Similarly, Twitter conversations also may include those living outside the United States.

Perhaps most important, the Twitter users who choose to share their views on events vary with the topics in the news. Those who tweeted about the California same-sex marriage ruling were likely not the same group as those who tweeted about Obama’s inaugural or Romney’s selection of Paul Ryan.

This leads to me to three thoughts:

1. What does this mean for the archiving of Twitter being undertaken by the Library of Congress? While it is still an interesting data source, Twitter provides a very small slice of U.S. opinion.

2. This is emblematic of larger issues with relying on new technologies to do research: who uses newer technologies is not the same as the U.S. population. This can be corrected for, as a recent article titled “A More Perfect Poll” suggests, and technologies can eventually filter throughout the whole U.S. population. In the meantime, researchers need to be careful about what they conclude.

3. So…what do we do about a comparison of a non-representative sample to a population? Pew seems to admit this:

While this provides an interesting look into how communities of interest respond to different circumstances, it does not reliably correlate with the overall reaction of adults nationwide.

This is an odd way to conclude a statistical report.

Looking at the data behind the claim that more black men are in jail than college

Posted on by

A scholar looks at his own usage of a statistic and where it came from:

About six years ago I wrote, “In 2000, the Justice Policy Institute (JPI) found evidence that more black men are in prison than in college,” in my first “Breaking Barriers” (pdf) report. At the time, I did not question the veracity of this statement. The statement fit well among other stats that I used to establish the need for more solution-focused research on black male achievement…

Today there are approximately 600,000 more black men in college than in jail, and the best research evidence suggests that the line was never true to begin with. In this two-part entry in Show Me the Numbers, the Journal of Negro Education’s monthly series for The Root, I examine the dubious origins, widespread use and harmful effects of what is arguably the most frequently quoted statistic about black men in the United States…

In September 2012, in response to the Congressional Black Caucus Foundation’s screening of the film Hoodwinked, directed by Janks Morton, JPI issued a press release titled, “JPI Stands by Data in 2002 on Education and Incarceration.” However, if one examines the IPEDS data from 2001 to 2011, it is clear that many colleges and universities were not reporting JPI’s data 10 years ago.

In 2011, 4,503 colleges and universities across the United States reported having at least one black male student. In 2001, only 2,734 colleges and universities reported having at least one black male student, with more than 1,000 not reporting any data at all. When perusing the IPEDS list of colleges with significant black male populations today but none reported in 2001, I noticed several historically black colleges and universities, including Bowie State University, and my own alma mater, Temple University. Ironically, I was enrolled at Temple as a doctoral candidate in 2001.

When I first saw this, I first thought it might be an example of what sociologist Joel Best calls a “mutant statistic.” This is a statistic that might originally be based in fact but at some point undergoes a transformation and keeps getting repeated until it seems unchallengeable.

There might be some mutant statistic going here but it also appears to be an issue of methodology. As Toldson points out, it looks like this was a missing data issue: the 2001 survey did not include data from over 1,000 colleges. When more colleges were counted in 2011, the findings changed. If it is a methodological issue, then this issue should have been caught at the beginning.

As Best notes, it can take some time for bad statistics to be reversed. It will be interesting to see how long this particular “fact” continues to be repeated.

Why Public Policy Polling (PPP) should not conduct “goofy polls”

Posted on by

Here is an explanation why the polling firm Public Policy Polling (PPP) conducts “goofy polls”:

But over the past year, PPP has been regularly releasing goofy, sometimes pointless polls about every other month. In early January, one such survey showed that Congress was less popular than traffic jams, France and used-car salesmen. According to their food-centric surveys released this week, Americans clearly prefer Ronald McDonald over Burger King for President; Democrats are more likely to get their chicken at KFC than Chick-fil-A, and Republicans are more apt to order pancakes than waffles. “We’re obviously doing a lot of polling on the key 2014 races,” says Jensen. “That kind of polling is important. We also like to do some fun polls.”

PPP, which has a left-leaning reputation, releases fun polls in part because they’re entertaining but mostly in an attempt to set themselves apart as an approachable polling company. Questions for polls are sometimes crowd-sourced via Twitter. The outfit does informal on-site surveys about what state they should survey next. And when the results of offbeat polls come out, the tidbits have potential to go viral. “We’re not trying to be the next Gallup or trying to be the next Pew,” Jensen says. “We’re really following a completely different model where we’re known for being willing to poll on stuff other people aren’t willing to poll on.” Like whether Republicans are willing to eat sushi (a solid 64% are certainly not).

Which means polls about “Mexican food favorability” are a publicity stunt on some level. Jensen says PPP, which has about 150 clients, gets more business from silly surveys and the ethos it implies than they do cold-calling. One such client was outspoken liberal Bill Maher, who hired PPP to poll for numbers he could use on his HBO show Real Time. That survey, released during the 2012 Republican primaries, found that Republicans were more likely to vote for a gay candidate than an atheist candidate—and that conservative virgins preferred Mitt Romney, while Republicans with 20 or more sexual partners strongly favored Ron Paul.

Jensen argues that the offbeat polls do provide some useful information. One query from the food survey, for instance, asks respondents whether they consider themselves obese: about 20% of men and women said yes, well under the actual American obesity rate of 35.7%.  Information like that could give health crusaders some fodder for, say, crafting public education PSAs. Still, the vast majority of people are only going to use these polls to procrastinate at work: goodness knows it’s hard to resist a “scientific” analysis of partisans’ favorite pizza toppings (Republicans like olives twice as much!).

Here is my problem with this strategy: it is short-sighted and privileges PPP. While polling firms do need to market themselves as there are a number of organizations that conduct national polls, this strategy can harm the whole field. When the average American sees the results of “goofy polls,” is it likely to improve their view of the polling in general? I argue there is already enough suspicion in America about polls and their validity without throwing in polls that don’t tell us as much. This suspicion contributes to lower response rates across the board, a problem for all survey researchers.

In the end, the scientific nature of polling takes a hit when any firm is willing to reduce polling to marketing.

How can Lake County, Illinois be #9 on the list of “America’s Most Miserable Cities”?

Posted on by

Forbes just put out their 2013 list of “America’s Most Miserable Cities.” Out of the top 20, there is one that is not like the others: Lake County, Illinois at #9. Here is the short description of why Lake County made the list:

The Chicago suburb is one of the richest counties in the U.S., as measured by per capita income. But home prices are down 29% over the past 5 years. Other drawbacks: long commutes and lousy weather.

There are numerous problems with this:

1. Calling an entire county a suburb is strange. Lake County is made up of dozens of suburbs which are quite varied. For example, look at quick overviews of Deerfield versus Grayslake versus Waukegan. Lumping them all together is silly and is one of the traps many people make when looking at the suburbs: they are not all the same kind of places.

2. How does a county end up on this list when the rest of the top 20 are cities? In terms of categories, a suburban county is not in the same category as a city. While there might be some identity in saying one is from “Lake County,” it is nowhere close to being a singular city.

3. Just glancing at this description and the top 20 cities on the list, I have to wonder how Lake County could even make the list. According to this list, Lake County is the 56th wealthiest county in the United States with a median household income of $74,266. Here is a bit more on the methodology:

We looked at the 200 largest metropolitan statistical areas and divisions in the U.S. to determine America’s Most Miserable Cities. The minimum population to be eligible was 259,000. We ranked each area on 9 factors, including average unemployment rate between 2010 and 2012; median commute times to work for 2011 based on U.S. Census data; violent crimes per capita from the FBI’s 2011 Uniform Crime Report.

We included three housing metrics: the change in median home prices between 2009 and 2012; foreclosure rates in 2012, as compiled by RealtyTrac; and property tax rates based on median real estate taxes paid and median home values in 2011 per the U.S. Census. We factored in income tax rates and the weather in each metro on factors relating to temperature, precipitation and humidity. The data metrics are weighted equally in the final scoring.

We tweaked the methodology in this year’s list in response to feedback from readers, dropping our rankings of both pro sports team success and political corruption, since both were based on regional, rather than city-specific data. We also added a new measure—net migration—which we see as a clear gauge of whether or not residents feel a community is worth living in.

If this methodology puts Lake County at #9, Forbes may want to revisit their criteria.