Tag Archives: methodology

Is this meaningful data: Chicago the “slowest-growing major city” between 2011 and 2012?

Posted on by

New figures from the Census show that Chicago doesn’t fare well compared to other cities in recent population growth:

Chicago gained nearly 10,000 people from July 2011 to July 2012, but was the slowest-growing major city in the country according to U.S. Census Bureau estimates released Thursday.

It was the second year in a row that population grew here, but the increase so far shows no signs of making up for the loss of 200,000 people over the previous decade…

Among cities with more than one million people, sun-belt metropolises like Dallas, San Antonio, Phoenix, Houston and San Diego all posted gains of more than 1.3 percent, while Chicago grew by little more than one-third of 1 percent.

With a total estimated population of 2,714,856. Chicago held on to its spot as the third largest city. But the two largest cities padded their leads, with New York City adding 67,000 in 2012 and No. 2 Los Angeles gaining 34,000 people.

While I’m sure some will use these figures to judge Chicago’s politics and development efforts, I’m not sure these figures mean anything. Here’s why:

1. The data only cover one year. This is just one time point. The story does a little bit to provide a wider context by referencing the 2010-2010 population figures but it would also be helpful to know the year-to-year figures for the last two years. In other words, what is the trend in the last several years in Chicago? Is the nearly 10,000 new people much different from 2011 or 1010 or 1009?

2. These are population estimates meaning there is a margin of error for the estimate. Thus, that error might cover a decent amount of population growth in all of these cities.

In the end, we need more data over time to know whether there are long-term trends going on in these major cities.

Two other interesting notes from the Census data:

1. The population growth in the Sunbelt continues:

Eight of the 15 fastest-growing large U.S. cities and towns for the year ending July 1, 2012 were in Texas, according to population estimates released today by the U.S. Census Bureau. The Lone Star State also stood out in terms of the size of population growth, with five of the 10 cities and towns that added the most people over the year…

No state other than Texas had more than one city on the list of the 15 fastest-growing large cities and towns. However, all but one were in the South or West.

This fits with what Joel Kotkin has been saying for a while.

2. Many Americans continue to live in communities with fewer than 50,000 people:

Of the 19,516 incorporated places in the United States, only 3.7 percent (726) had populations of 50,000 or more in 2012.

However, many of these smaller communities are suburbs near big cities. It’s too bad there aren’t figures here about what percentage of Americans live in those 726 communities of 50,000 or more.

Defining what makes for a luxury home

Posted on by

Here is how one data firm defines what it means to be a luxury housing unit:

Although upscale housing is selling better in some cities than in others, a monthly analysis by the Altos Research data firm for the Institute for Luxury Home Marketing says that overall, that segment of the market is gaining momentum and prices are rising…

Q: “Luxury home” is probably one of the most abused phrases in real estate-ese. How do you define it?

A: A price range that’s considered the high end of the market in one place might be something that’s average in another. So, “luxury” is local: Our organization generally defines it as the top 10 percent of an area’s sales in the past 12 months. But for the purposes of the research that we do with Altos for our monthly Luxury Market Report, we’ve taken the ZIP codes within each of 31 markets that have the highest median prices, and for about five years we’ve tracked the sales of homes in those (areas) that are $500,000 and above.

There are two techniques proposed here:

1. The highest 10 percent of a local housing market. Thus, the prices are all relative and the data is based on the highest end in each place. So, there could be some major differences in luxury prices across zip codes or metropolitan regions.

2. Breaking it down first by geography to the wealthiest places (so this is based on geographic clustering) and then setting a clear cut point at $500,000. In these wealthiest zip codes, wouldn’t most of the units be over $500,000? Why the 31 wealthiest markets and not 20 or 40?

Each of these approaches have strengths and weaknesses but I imagine the data here could change quite a bit based on what operationalization is utilized.

Interestingly, the firm found that luxury sales rebounded quicker than the rest of the market:

The interesting thing about this recovery is that the luxury segment, that group of affluent households, was able to recover fairly quickly. They shifted their assets around, and a lot of them were able to see opportunities in the down market. By 2010, there were almost as many high-end households as before the downturn, not just in the United States, but internationally, as well. This group focused on residential real estate as a pretty desirable asset — for them, a second or third home turned out to be a portfolio play.

This shouldn’t be too surprising – when an economic crisis hits, the wealthier members of society have more of a cushion. While the upper end is doing all right, others have argued the bottom end, those looking for starter homes, are having a tougher time.

Sociologists = people who look at “boring data compiled during endless research”

Posted on by

If this is how a good portion of the public views what sociologists do, sociologists may be in trouble:

Anthony Campolo is a sociologist by trade, used to looking at boring data compiled during endless research.

Data collection and analysis may not be glamorous but a statement like this suggests sociologists may have some PR issues. Data collection and analysis are often time consuming and even tedious. But, there are reasons for working so hard to get data and do research: so sociologists can make substantiated claims about how the social world works. Without rigorous methods, sociologists would just be settling for interpretation, opinion, or anecdotal evidence. For example, we might be left with stories like that of a homeless man in Austin, Texas who was “testing”  which religious groups contributed more money to him. Of course, his one case tells us little to nothing.

Perhaps this opening sentence should look something like this: time spent collecting and analyzing data will pay off in stronger arguments.

 

Spreadsheet errors, austerity, ideology, and social science

Posted on by

The graduate student who found some spreadsheet errors in an influential anti-austerity paper discusses what happened. Here is part of the conversation about the process of finding this error:

Q. You say, don’t you, that their use of data was faulty?

A. Yes. The terms we used about their data—”selective” and “unconventional”—are appropriate ones. The reasons for the choices they made needed to be given, and there was nowhere where they were.

Q. And how about their claim that your findings support their thesis that growth slows as debt rises?

A. That is not our interpretation of our paper, at all. If you read their paper, it’s interesting how they handle causality. They waffle between strong and weak claims. The weak claim is that it’s just a negative association. If that’s all they claim, then it’s not really relevant for policy. But they also make a strong claim, more in public than in the paper, that there’s causality going from high debt to drops in growth. They haven’t been obvious about that…

Q. Paul Krugman wrote in The New York Times that your work confirms what many economists have long intuitively thought. Was that your intuition?

A. Yes. I just thought it was counterintuitive when I first saw their claim. It wasn’t plausible.

Q. This is more than a spreadsheet error, then?

A. Yes. The Excel error wasn’t the biggest error. It just got everyone talking about this. It was an emperor-has-no-clothes moment.

This would make for a good case study in a methodology class in the social sciences: how much of this is about actual data errors versus different interpretations? You have people who are clearly staking out space on either side of a policy discussion and it is a bit unclear how much does this color their interpretation of “facts”/data. I suspect some time will help sort this out – if the spreadsheet was indeed wrong, shouldn’t this lead to a correction or a retraction?

I do like the fact that the original authors were willing to share their data – this is something that could happen more often in the social sciences and give people the ability to look at the data for themselves.

Argument: Big Data reduces humans to something less than human

Posted on by

One commentator suggests Big Data can’t quite capture what makes humans human:

I have been browsing in the literature on “sentiment analysis,” a branch of digital analytics that—in the words of a scientific paper—“seeks to identify the viewpoint(s) underlying a text span.” This is accomplished by mechanically identifying the words in a proposition that originate in “subjectivity,” and thereby obtaining an accurate understanding of the feelings and the preferences that animate the utterance. This finding can then be tabulated and integrated with similar findings, with millions of them, so that a vast repository of information about inwardness can be created: the Big Data of the Heart. The purpose of this accumulated information is to detect patterns that will enable prediction: a world with uncertainty steadily decreasing to zero, as if that is a dream and not a nightmare. I found a scientific paper that even provided a mathematical model for grief, which it bizarrely defined as “dissatisfaction.” It called its discovery the Good Grief Algorithm.

The mathematization of subjectivity will founder upon the resplendent fact that we are ambiguous beings. We frequently have mixed feelings, and are divided against ourselves. We use different words to communicate similar thoughts, but those words are not synonyms. Though we dream of exactitude and transparency, our meanings are often approximate and obscure. What algorithm will capture “the feel of not to feel it?/?when there is none to heal it,” or “half in love with easeful Death”? How will the sentiment analysis of those words advance the comprehension of bleak emotions? (In my safari into sentiment analysis I found some recognition of the problem of ambiguity, but it was treated as merely a technical obstacle.) We are also self-interpreting beings—that is, we deceive ourselves and each other. We even lie. It is true that we make choices, and translate our feelings into actions; but a choice is often a coarse and inadequate translation of a feeling, and a full picture of our inner states cannot always be inferred from it. I have never voted wholeheartedly in a general election.

For the purpose of the outcome of an election, of course, it does not matter that I vote complicatedly. All that matters is that I vote. The same is true of what I buy. A business does not want my heart; it wants my money. Its interest in my heart is owed to its interest in my money. (For business, dissatisfaction is grief.) It will come as no surprise that the most common application of the datafication of subjectivity is to commerce, in which I include politics. Again and again in the scholarly papers on sentiment analysis the examples given are restaurant reviews and movie reviews. This is fine: the study of the consumer is one of capitalism’s oldest techniques. But it is not fine that the consumer is mistaken for the entirety of the person. Mayer-Schönberger and Cukier exult that “datafication is a mental outlook that may penetrate all areas of life.” This is the revolution: the Rotten Tomatoes view of life. “Datafication represents an essential enrichment in human comprehension.” It is this inflated claim that gives offense. It would be more proper to say that datafication represents an essential enrichment in human marketing. But marketing is hardly the supreme or most consequential human activity. Subjectivity is not most fully achieved in shopping. Or is it, in our wired consumerist satyricon?

“With the help of big data,” Mayer-Schönberger and Cukier continue, “we will no longer regard our world as a string of happenings that we explain as natural and social phenomena, but as a universe comprised essentially of information.” An improvement! Can anyone seriously accept that information is the essence of the world? Of our world, perhaps; but we are making this world, and acquiescing in its making. The religion of information is another superstition, another distorting totalism, another counterfeit deliverance. In some ways the technology is transforming us into brilliant fools. In the riot of words and numbers in which we live so smartly and so articulately, in the comprehensively quantified existence in which we presume to believe that eventually we will know everything, in the expanding universe of prediction in which hope and longing will come to seem obsolete and merely ignorant, we are renouncing some of the primary human experiences. We are certainly renouncing the inexpressible. The other day I was listening to Mahler in my library. When I caught sight of the computer on the table, it looked small.

I think there are a couple of arguments possible about the limitations of big data and Wieseltier is making a particular argument. He does not appear to be saying that big data can’t predict or model human complexity. And fans of big data would probably say the biggest issue is that we simply don’t have enough data yet and we are developing better and better models. In other words, our abilities and data will eventually catch up to the problem of complexity. But I think Wieseltier is arguing something else: he, along with many others, does not want humans to be reduced to information. Even if we had the best models, it is one thing to see people as complex individuals and yet another to say they are simply another piece of information. Doing the latter takes away the dignity of people. Reducing people to data means we stop seeing people as people that can change their minds, be creative, and confound predictions.

It will be interesting to see how this plays out in the coming years. I think this is the same fear many people have about statistics. Particularly in our modern world where we see ourselves as sovereign individuals, describing statistical trends to people strikes them as reducing their own agency and negating their own experiences. Of course, this is not what statistics is about and something more training in statistics could help change. But, how we talk about data and its uses might go a long way to how big data is viewed in the future.

Pollster provides concise defense of polls

Posted on by

The chief pollster for Fox News defends polls succinctly here. The conclusion:

Likewise, we don’t need to contact every American — more than 230 million adults — to find out what the public is thinking. Suffice it to say that with proper sampling and random selection of respondents so that every person has an equal chance of being contacted, a poll of 800-1,000 people provides an incredibly accurate representation of the country as a whole. It’s a pretty amazing process if you think about it.

Still, many people seem to have a love-hate relationship with polls. Even if they enjoy reading the polls, some people can turn into skeptics if they personally don’t feel the same as the majority. Maybe they don’t even know anyone who feels the same as the majority.  Yet assuming everyone shares your views and those of your friends and neighbors would be like the cook skimming a taste from just the top of the pot without stirring the soup first.

Basic but a staple of many a statistics and research methods course. Unfortunately, more people need this kind of education in a world where statistics are becoming more and more common.

Lincoln, Nebraska #1 city in well-being

Posted on by

A new survey from Gallup and Healthways puts Lincoln, Nebraska as the number one city in the U.S. for well-being:

Lincoln, Neb., had the highest Well-Being Index score (72.8) in the U.S. across the 189 metropolitan areas that Gallup and Healthways surveyed in 2012. Also in the top 10 are Boulder, Colo.; Provo-Orem, Utah; Ann Arbor, Mich.; Honolulu, Hawaii; Fort Collins-Loveland, Colo.; and Burlington-South Burlington, Vt…

At 60.8, Charleston, W.Va., had the lowest Well-Being Index score, displacing Huntington-Ashland, W.Va.-Ky.-Ohio, which held this position the two previous years. Huntington-Ashland’s score of 61.2 is up from 58.1 in 2010, which is the lowest score on record for any metro area across five years of data collection. Mobile, Ala.; Utica-Rome, N.Y.; Hickory-Lenoir-Morganton, N.C.; and Fort Smith, Ark.-Okla.; join Charleston and Huntington-Ashland as frequent occupants of the bottom 10 list each year…

Washington, D.C.-Arlington-Alexandria, Va.-Md.-W.Va., residents reported the highest wellbeing among the nation’s 52 largest metropolitan areas, defined as those with 1 million residents or more, followed by San Francisco-Oakland-Fremont, Calif. These two metros have been in the top five among large metro areas in each of the past five years…

The Gallup-Healthways Well-Being Index score is an average of six sub-indexes, which individually examine life evaluation, emotional health, work environment, physical health, healthy behaviors, and access to basic necessities. The overall score and each of the six sub-index scores are calculated on a scale from 0 to 100, where a score of 100 represents the ideal. Gallup and Healthways have been tracking these measures daily since January 2008.

Interesting as there are more cities from the Great Plains and Midwest than I expected.

A few thoughts about the methodology:

1. After all is added up across these six measures, there isn’t much variation between the top and the bottom. Lincoln had the highest score at 72.8 and Charleston had the lowest at 60.8. So on a scale of 0 to 100, the range was just 12. This suggests there is not much variation in these measures and that this index may not tell us a whole lot. Are Americans simply generally optimistic about these topics or are they realistically not that different across cities?

2. What exactly does Gallup and Healthways do with this information that it requires daily polling? This is not a small sample:

Results are based on telephone interviews conducted as part of the Gallup-Healthways Well-Being Index survey Jan. 2-Dec. 29, 2012, with a random sample of 353,563 adults, aged 18 and older, living in all 50 U.S. states and the District of Columbia, selected using random-digit-dial sampling.

Perhaps there is some marketing edge to this surveying or it is related to some big research project.

Bonus well-being info: for occupations, doctors and then K-12 teachers lead the way and manufacturing-production workers and then transportation workers are at the bottom.

Social network of email between countries shows homophily between culturally similar nations

Posted on by

A new study of email traffic between countries finds some patterns:

The Internet was supposed to let us bridge continents and cultures like never before. But after analyzing more than 10 million e-mails from Yahoo! mail, a team of computer researchers noticed an interesting phenomenon: E-mails tend to flow much more frequently between countries with certain economic and cultural similarities.

Among the factors that matter are GDP, trade, language, non-Commonwealth colonial relations, and a couple of academic-sounding cultural metrics, like power-distance, individualism, masculinity and uncertainty…

To this point, of course, the study amounts to little more than very interesting trivia. The real conclusion comes toward the end, when the researchers posit it as possible evidence for Samuel Huntington’s controversial “Clash of Civilizations” theory. From the paper:

In this respect we cautiously assign a level of validity to Huntington’s contentions, with a few caveats. The ?rst issue was already mentioned – overlap between civilizations and other factors contributing to countries’ level of association. Huntington’s thesis is clearly re?ected in the graph presented in Figure 3, but some of these civilizational clusters are found to be explained by other factors in Table 5. The second limitation concerns the fact that we investigated a communication network. There is no necessary “clash” between countries that do not communicate, and Huntington’s thesis was concerned primarily with ethnic con?ict.

Interesting what can be done with data from more than 10 million emails.

I wonder if it is even worth doing this analysis at the country level. Isn’t this too broad? Aren’t there likely to be important patterns within and across countries that are obscured by this broader lens?

Another possible issue: is Yahoo mail a representative sample of emails or does it provide a particular slice of of email traffic? I would assume it involves more personal email as opposed to business activity.

Pew reminds us that Twitter users are not representative of the US population

Posted on by

In looking at this story, I was led to a recent Pew study that compared the political leanings of Twitter to the political opinions of the general US population. One takeaway: the two populations are not the same.

The lack of consistent correspondence between Twitter reaction and public opinion is partly a reflection of the fact that those who get news on Twitter – and particularly those who tweet news – are very different demographically from the public.

The overall reach of Twitter is modest. In the Pew Research Center’s 2012 biennial news consumption survey, just 13% of adults said they ever use Twitter or read Twitter messages; only 3% said they regularly or sometimes tweet or retweet news or news headlines on Twitter.

Twitter users are not representative of the public. Most notably, Twitter users are considerably younger than the general public and more likely to be Democrats or lean toward the Democratic Party. In the 2012 news consumption survey, half (50%) of adults who said they posted news on Twitter were younger than 30, compared with 23% of all adults. And 57% of those who posted news on Twitter were either Democrats or leaned Democratic, compared with 46% of the general public. (Another recent Pew Research Center survey provides even more detail on who uses Twitter and other social media.)

In another respect, the Twitter audience also is broader than the sample of a traditional national survey. People under the age of 18 can participate in Twitter conversations, while national surveys are limited to adults 18 and older. Similarly, Twitter conversations also may include those living outside the United States.

Perhaps most important, the Twitter users who choose to share their views on events vary with the topics in the news. Those who tweeted about the California same-sex marriage ruling were likely not the same group as those who tweeted about Obama’s inaugural or Romney’s selection of Paul Ryan.

This leads to me to three thoughts:

1. What does this mean for the archiving of Twitter being undertaken by the Library of Congress? While it is still an interesting data source, Twitter provides a very small slice of U.S. opinion.

2. This is emblematic of larger issues with relying on new technologies to do research: who uses newer technologies is not the same as the U.S. population. This can be corrected for, as a recent article titled “A More Perfect Poll” suggests, and technologies can eventually filter throughout the whole U.S. population. In the meantime, researchers need to be careful about what they conclude.

3. So…what do we do about a comparison of a non-representative sample to a population? Pew seems to admit this:

While this provides an interesting look into how communities of interest respond to different circumstances, it does not reliably correlate with the overall reaction of adults nationwide.

This is an odd way to conclude a statistical report.

Looking at the data behind the claim that more black men are in jail than college

Posted on by

A scholar looks at his own usage of a statistic and where it came from:

About six years ago I wrote, “In 2000, the Justice Policy Institute (JPI) found evidence that more black men are in prison than in college,” in my first “Breaking Barriers” (pdf) report. At the time, I did not question the veracity of this statement. The statement fit well among other stats that I used to establish the need for more solution-focused research on black male achievement…

Today there are approximately 600,000 more black men in college than in jail, and the best research evidence suggests that the line was never true to begin with. In this two-part entry in Show Me the Numbers, the Journal of Negro Education’s monthly series for The Root, I examine the dubious origins, widespread use and harmful effects of what is arguably the most frequently quoted statistic about black men in the United States…

In September 2012, in response to the Congressional Black Caucus Foundation’s screening of the film Hoodwinked, directed by Janks Morton, JPI issued a press release titled, “JPI Stands by Data in 2002 on Education and Incarceration.” However, if one examines the IPEDS data from 2001 to 2011, it is clear that many colleges and universities were not reporting JPI’s data 10 years ago.

In 2011, 4,503 colleges and universities across the United States reported having at least one black male student. In 2001, only 2,734 colleges and universities reported having at least one black male student, with more than 1,000 not reporting any data at all. When perusing the IPEDS list of colleges with significant black male populations today but none reported in 2001, I noticed several historically black colleges and universities, including Bowie State University, and my own alma mater, Temple University. Ironically, I was enrolled at Temple as a doctoral candidate in 2001.

When I first saw this, I first thought it might be an example of what sociologist Joel Best calls a “mutant statistic.” This is a statistic that might originally be based in fact but at some point undergoes a transformation and keeps getting repeated until it seems unchallengeable.

There might be some mutant statistic going here but it also appears to be an issue of methodology. As Toldson points out, it looks like this was a missing data issue: the 2001 survey did not include data from over 1,000 colleges. When more colleges were counted in 2011, the findings changed. If it is a methodological issue, then this issue should have been caught at the beginning.

As Best notes, it can take some time for bad statistics to be reversed. It will be interesting to see how long this particular “fact” continues to be repeated.