Tag Archives: methodology

Disagreement on whether there are 7 billion people on earth just yet

Posted on by

There have been a number of recent stories about how the world’s population has reached 7 billion. Interestingly, not everyone agrees that this has happened yet:

According to United Nations demographers, 6,999,999,999 other Earthlings potentially felt the same way on Monday when the world’s population topped seven billion. But if you’d rather go by the United States Census Bureau’s projections, you’ve got some breathing room. The bureau estimates that even with the world’s population increasing by 215,120 a day, it won’t reach seven billion for about four months.

How do the dueling demographic experts reconcile a difference, as of Monday, of 28 million, which is more than all the people in Saudi Arabia?

They don’t.

“No one can know the exact number of people on the globe,” Gerhard Heilig, chief of the population estimates and projections section of the United Nations Population Division, acknowledges.

Even the best individual government censuses have a margin of error of at least 1 percent, he said, which would translate in the global aggregation to “a window of uncertainty of six months before or six months after Oct. 31.” An error margin of even as little as 2 percent would mean that Monday’s estimate of seven billion actually was 56 million off (which is more people than were counted in South Africa).

Figuring this out is not an easy task. It requires a central group to tabulate results from all of the countries around the world. Could there be a difference in the reliability and validity of the results across nations? For example, can we trust population counts from honed operations in the United States and other Western nations more than counts from Third World countries? (I wish the article went into this: how accurate are population figures from different countries? How big might the margins of errors be?) I’ve seen this before when doing some research in graduate school on suicide figures that the United Nations has collected – in the period I was looking at, roughly 1950 to 1970, some countries didn’t report, some had rougher estimates, and countries could have different definitions about what constitutes a suicide. Absolute population counts should be more straight forward but I imagine there could be a number of complications.

Will we get another round of news stories when the Census Bureau says we have hit 7 billion? I wonder if the perceived global authority of the United Nations versus that of the Census Bureau plays a role. For example, did the New York Times report the 7 billion figure as front-page news and then print this caveat story later in the news section?

A final note: the story ends by suggesting the two estimates are not that far off. If we could be so lucky that all of our estimates have only a 1% margin of error, science would benefit greatly. But it is a reminder that official figures are estimates, not 100% counts of social phenomenon.

Looking for a new area of study? Try Twitterology

Posted on by

If it is in the New York Times, Twitterology must be a viable area of academic study:

Twitter is many things to many people, but lately it has been a gold mine for scholars in fields like linguistics, sociology and psychology who are looking for real-time language data to analyze.

Twitter’s appeal to researchers is its immediacy — and its immensity. Instead of relying on questionnaires and other laborious and time-consuming methods of data collection, social scientists can simply take advantage of Twitter’s stream to eavesdrop on a virtually limitless array of language in action…

One criticism of “sentiment analysis,” as such research is known, is that it takes a naïve view of emotional states, assuming that personal moods can simply be divined from word selection. This might seem particularly perilous on a medium like Twitter, where sarcasm and other playful uses of language often subvert the surface meaning…

Still, the Twitterologists will continue to have a tough row to hoe in justifying their research to those who think that Twitter is a trivial form of communication. No less a figure than Noam Chomsky has taken Twitter to task recently for its “superficiality.”

For more sociological thoughts about Chomsky’s comments, see this post from a few days ago.

Here is my quick take on Twitterology: it has some potential for gathering quick, on-the-ground information. But there are two big issues that this article doesn’t address:

1. Are Twitter users representative of the whole population? Probably not. Twitter feeds might be good for studying very specific groups and movements.

2. How can one make causal arguments with Twitter data? If we had more information about Twitter users from profiles, this might be doable but Twitter is less about Facebook-style profiles. We then need studies that collect the information about Twitter users as well as their Twitter activity. If we want to ask questions like whether Twitter was instrumental or even helped cause the Arab Spring movements, we need more data.

Twitterology may be trendy at the moment but I think it has a ways to go before we can use it to tackle typical questions that sociologists ask.

“What’s Your Problem?” misses an opportunity to explain survey research

Posted on by

The “What’s Your Problem?” column in the Chicago Tribune tackles the problems of consumers. Yesterday’s column involved a woman who had been called multiple times by a survey firm even after she asked to not be called again:

Over the following weeks, Scarborough representatives called Riedell repeatedly, asking her to participate in a 15-minute phone survey.

No matter how many times she refused their overtures, the calls kept coming.

Riedell said she asked each time to have her name taken off the call list but was told that representatives were not authorized to do so.

And so it continued through late summer and early fall. By the sixth call, Riedell decided she had heard enough. She emailed What’s Your Problem?

When contacted by the Tribune, the survey firm had this reponse which did not please Riedell:

Dercher said Riedell did not leave her name and phone number when she called Scarborough’s toll-free number, which are critical pieces of information so that the company can remove a respondent from the calling list.

Although her number could be randomly picked for another survey in the future, the odds are against that happening, Dercher said.

After reading Dercher’s email, Riedell said Scarborough’s response was, well, lame.

“The response says that their interviewers are not allowed to remove the name of a respondent from their calling list since the respondent’s name is confidential, but the interviewer already has the respondent’s name and phone number, otherwise they wouldn’t have been able to reach me by phone or address me by my name when I answered the phone,” Riedell said. “Sounds like gibberish to me.”

The column is clearly geared toward Riedell’s point of view and frankly, who likes to be called repeatedly by companies or survey organizations after refusing to participate? At the same time, let’s flip this around to see it from the opposite angle:

-Riedell was selected for the survey by random digit dialing. This is not unusual and telephone surveys are not covered by the Do Not Call registry.

-It doesn’t sound unusual that the survey interviewers didn’t have the power to remove her name from their lists. They were likely handed lists of numbers and told to call until they had an answer.

-Surveys often select their initial batch of respondents and then do whatever they can to get responses from them. The US Census Bureau goes to housing units repeated times in order to collect data because they want accurate data. (Of course, one Census worker who was doing his job last year was arrested for trespassing in Hawaii.) If survey companies simply gave up on people after one attempt, they would spend a lot more time and money and doing so might mess up their calibrated samples which are meant to represent larger populations.

In the end, Riedell may not like the system but in order to collect good data, survey companies may have to contact selected respondents multiple times. Since participation is voluntary, Riedell can opt out and perhaps Scarborough does need to have a more clearly delineated method by which people can opt out. Additionally, there may be some complications because Scarborough is a market survey research firm (tagline: Scarborough Research measures our shopping, media, and lifestyle behaviors) and are not academic researchers or political researchers (though push polls are very problematic). But this column could be much more informative about how survey research works and how consumers can respond to common requests for information rather than just suggesting that this woman should be able to more easily avoid telephone survey questions.

Cities ranked by the “Trick or Treat Index”

Posted on by

Richard Florida has put some of his data to use to answer an important question: what are the best cities in the united States for trick or treating on Halloween?

According to National Retail Federation projections, Americans will spend $6.86 billion on Halloween this year, up from $3.3 billion in 2005 when a lot fewer of us were out of work. But even as Halloween edges up on Christmas as a shopping opportunity, the trick-or-treating experience is a lot less universal than it was. In some towns, you see hardly any unsupervised trick-or-treaters after dark; in other places—Brooklyn Heights or my neighborhood in Toronto leap to mind—there are more kids than you can imagine.

Herewith the 2011 edition of the Trick-or-Treater Index developed with the ever-able number-crunching of my Martin Prosperity Institute colleague, Charlotta Mellander. The 2011 Index is based on the following five metrics: the share of children aged 5 to 14; median household income (figuring the haul will be better in more affluent metros), population density, walkability (measured as the percentage of people who walk or bike to work) and creative spirit (which we measured as the percentage of artists, designers, and other cultural creatives). The data are from the American Community Survey and cover all U.S. metro areas, both their cores and suburbs…

As for the top ranking metros, Bridgeport-Stamford-Norwalk, Connecticut, comes in first again this year. Greater New York has moved up to second place, followed by Chicago, greater Washington, D.C., and the twin cities of Minneapolis-St. Paul. Los Angeles, last year’s runner-up, has dropped to 7th place. Big metros dominate the top spots, but Lancaster, Pennsylvania, has moved all the way up from 16th last year to 6th on our 2011 rankings. And college towns like Ann Arbor, Michigan, Boulder, Colorado, and New Haven, Connecticut, also rank among the top 25.

This reminds me of another recent odd use of data that ranked the luckiest cities. So people with more money, who are more creative, and live in more walkable areas necessarily give more or better candy? Might they also be the people who are more likely to give substitutes to candy? Could this also be related to health measures, like obesity or life expectancy? This seems like opportunistic, atheoretical data mining meant to get a few page clicks (like me).

And since there are probably few people who would go to a whole new metropolitan area just to get candy, wouldn’t this be a better analysis if it was at a zip code, community, or census block level?

How to rank the luckiest cities in the United States

Posted on by

Perhaps we have taken these rankings lists too far: Men’s Health has ranked the luckiest cities in the United States.

Luck is like that dark matter stuff scientists have spent billions of dollars trying to find with the Large Hadron Collider—a powerful presence that people surmise exists but no one has actually seen. The difference is that we found luck. Using statistics instead of protons, we pinpointed the location of a large supply in, of all places, San Diego.

Wondering how Vegas didn’t hit this jackpot? Here’s our definition of good luck: the most winners of Powerball, Mega Millions, and Publishers Clearing House sweepstakes; most hole-in-ones (PGA); fewest lightning strikes (including the fatal kind) and deaths from falling objects (Vaisala Inc., National Climatic Data Center, CDC); and least money lost on lottery tickets and race betting (Bureau of Labor Statistics).

San Diego is number one on the list with Baltimore, Phoenix, Wilmington (Delaware), and Richmond rounding out the top five. Chicago is #36. The bottom five: Sioux Falls, Memphis, Jackson (Mississippi), Tampa, and Charleston (West Virginia).

What I like about this is that they are straightforward with what factors went into the rankings (though they might have been weighted). These are what we might consider “very rare” and cultural conditioned lucky events. The lottery is perhaps the poster child for this. If someone wins more than once, some suspicions might surface (see a story about a four-time Texas winner here). What about lesser luck, such as avoiding a car accident at the last minute or local sports teams coming up with miraculous plays at the end of a game or avoiding natural disasters? Such things would be much more difficult to measure and it might always be an open statistical question of whether strange occurrences could be explained by some other unmeasured or unknown factor.

Should anyone move to the luckier cities to really improve their chances? No, the statistical odds of any of these things happening is still quite small. In fact, it would be interesting to see how much really separates the luckiest cities from the unluckiest – are we talking a difference of 1 in a million? Ten in a million?

In the end, I think these rankings don’t really tell us much about anything. People shouldn’t use them as a guide and measuring luck is fraught with difficulty. Take the lottery winnings: could this simply reflect the fact that people in certain cities buy more tickets or their states have bigger lottery jackpots which encourages more participation? This is a story that uses real numbers to make a nebulous point in order to gain website clicks (guilty as charged) and sell magazines.

Sociologist tells how time diaries provide six insights in the study of national well-being

Posted on by

An Oxford sociologist gives six findings regarding national well-being based on time diary data.

Time diaries allow researchers to get at daily activities and move past some of the memory and social desirability issues that come up in interviews or surveys.

Sociologists tracking “global mood swings” through Twitter

Posted on by

New social media platforms like Facebook and Twitter are ripe data sources. A new study in Science done by two sociologists examines the world’s emotions through Twitter:

The research team, led by Scott Golder, a PhD doctoral student in the field of sociology, and Professor of Sociology Michael Macy, tracked 2.4 million people in 84 different countries over the past two years. Clearly the team working on the project didn’t read through 2.4 million people’s tweets. Instead, they used a text analysis program that quantified the emotional content of 509 million tweets. Their results, featured in the paper “Diurnal and Seasonal Mood Tracks Work, Sleep and Day Length Across Diverse Cultures,” were published September 29 in Science.

The researchers found that work, sleep, and the amount of daylight we get really does affect things like our enthusiasm, delight, alertness, distress, fear, and anger. They concluded that people tweet more positive things early in the morning and then again around midnight. This could suggest that people aren’t very happy while they’re working since their happy tweets are at the beginning and end of the day. Saturday and Sunday also saw more positive tweets in general. The weekend showed these peaks at about 2 hours later, which accounts for sleeping in and staying out late.

Of course, all of the trends weren’t the same throughout every country. For example, the United Arab Emirates tend to work Sunday through Thursday, so their weekend tweets happened on Friday and Saturdays. The results also found that people who live in countries that get more daylight (closer to the equator) aren’t necessarily happier than people in countries that get less daylight (closer to the North and South Poles). It seems that only people who have a lot of daylight during the summer and then very little in the winter feel the affect of the change in seasons as much.

Clearly the results of the research aren’t perfect. There may be some people who only share positive things on Twitter, or some people who love to be cynical and use Twitter to complain about problems.

This sounds interesting and the resulting maps and charts are intriguing.  However, I would first ask methodological questions that would get at whether this is worthwhile data or not. Does this really reflect global moods? Or does this simply tell us something about Twitter users, who are likely not representative of the population at large?

Another article does suggest this study makes methodological improvements over two common ways studies look at emotions:

None of these results are particularly surprising, but Golder and Macy suggest that using global tweets allows them to confirm previous studies that only looked at small samples of American undergraduates who were not necessarily representative of the wider world. Traditional studies also require participants to recall their past emotions, whereas tweets can be gathered in real time.

These are good things: more immediate data and a wider sample beyond college undergraduates. But this doesn’t necessarily mean that the Twitter data is good data. The sample still probably skews toward younger people and those who have the technological means to be on Twitter consistently. Additionally, immediate emotions can tell us one thing but inquiring about longer-term satisfaction often tells us something else.

On the whole, this sounds like better data than we have before but until we have more universal Twitter usage, this data source will have significant limitations.

Crowd Counting 101

Posted on by

Every now and then, often connected to politically contentious events like the “Restoring Honor” or “Rally to Restore Sanity” in 2010 or Egyptians taking to the streets in early 2011, you will see articles about how officials and media sources estimate the number of people who attend. Here is a primer on crowd counting. Some of the possible new methods could help give us accurate and not politically-driven counts:

And, as Yip said in a statement about his study, a good way to count crowds could cut through the politically motivated stats we put up with now. “In the absence of any accurate estimation methods, the public are left with a view of the truth colored by the beliefs of the people making the estimates. The public would be better served by estimates less open to political bias.”

I look forward to improved crowd counting.

h/t Instapundit

Americans are coolest nationality according to Badoo.com poll

Posted on by

A new poll from Badoo.com finds that Americans are the coolest nationality:

Social networking site Badoo.com asked 30,000 people across 15 countries to name the coolest nationality and also found that the Spanish were considered the coolest Europeans, Brazilians the coolest Latin Americans and Belgians the globe’s least cool nationality.

“We hear a lot in the media about anti-Americanism,” says Lloyd Price, Badoo’s Director of Marketing. “But we sometimes forget how many people across the world consider Americans seriously cool.”…

“America,” says Price, “boasts the world’s coolest leader, Obama; the coolest rappers, Jay-Z and Snoop Dogg; and the coolest man in technology, Steve Jobs of Apple, the man who even made geeks cool.”

Brazilians are ranked the second coolest nationality in the Badoo poll and the coolest Latin Americans, ahead of Mexicans and Argentinians. The Spanish, in third place, are the coolest Europeans.

At least one marketer is happy.

Two thoughts:

1. I would be very hesitant about accepting the results of this poll. If this is a web survey of social network site users, it is probably not very representative of people within these countries. Serious news organizations should report on the methodology and discuss the downsides (and advantages) of this approach when reporting this information. But, if it is an accurate take on social network site users, generally younger, plugged-in populations, perhaps this is exactly what American companies would want to hear.

2. America has military, political, and economic power but this hints at another, less-recognized dimension: cultural power and influence. For better or worse, American values, celebrities, products, and ideas have spread throughout the world. Even if our economic and political power goes into a relative decline, this cultural influence will live on for some time. (A bonus: a Badoo poll from earlier this summer also said Americans are the funniest nationality!)

3. Is being “cool” really something to aspire to as a nation? In an America dominated by celebrity, media, and consumption, it may be hard to know that this is not the primary objective.

(Some background on Badoo.com.)

How jobless Americans are spending their time

Posted on by

Some new research suggests that unemployed Americans are doing a variety of things:

One study last year found that much of the extra time gets spent sleeping and watching TV–leading to news reports that the jobless “frittered away” their time. Another analysis–this one released in January and co-written by Princeton economist Alan Krueger, who was announced Monday as the White House’s pick to serve as the chief economic adviser to President Obama–pointed in the same direction. It found that people tend to devote fewer hours to job searches the longer they’ve been unemployed, and that sleep–especially “sleep in the morning hours”–increases as joblessness goes on. Together, the studies appeared to create a picture of the unemployed as lazy and unproductive.But a sophisticated new analysis (pdf) complicates that picture. In a paper written for the National Bureau of Economic Research, Mark A. Aguiar, Erik Hurst, and Loukas Karabarbounis, using data from the American Time Use Survey, found that the jobless do spend about 30 percent of their extra time–the time they would otherwise have spent working–sleeping or watching TV, and another 20 percent on other leisure activities. But around 35 percent is spent doing unpaid but nonetheless important work, like child-care and housework. And other investments–things like education, health-care, and volunteer work –account for another 10 percent.

The notion advanced by some that jobless benefits are being used to support a life of leisure is, at best, simplistic.

But as Nancy Folbre, an economics professor at the University of Massachusetts, Amherst, notes, there’s a limit to how much useful unpaid work the jobless can do. “They lack the capital, land, tools and skills needed to flexibly shift from wage employment to production for their own use.,” she writes. “Even when they can make a partial shift, their productivity is likely to be lower in unpaid work than paid work.”

I’m a little surprised by the quote from an economist at the end: unpaid work still needs to be done by someone whether they currently have the skills for it or not. Perhaps she is referring to longer-term issues: do the unemployed go back to work (perhaps by changing fields or getting educated in new areas) or do they adjust to a life of unpaid work? In the meantime, there is a transition that has to be made. But I can imagine that some people would see this quote and wonder what this means for people who have always done unpaid work, particularly mothers.

Another way to interpret the earlier study that the unemployed enjoy a life of leisure is that this is due to feelings of restlessness and perhaps even depression.

In general, I find time use studies to be quite interesting. When you ask people general questions about how they spend their time, like how long they spend at work, the numbers can be quite inflated. The better studies require logs or diaries and ask questions about recent time periods where memories will not be as distorted. Here is how the American Time Use Study describes some of its methodology (starting at page 11 of this document):

The ATUS sample is randomized by day, with 50 percent of the sample reporting about
weekdays, Monday through Friday, and 50 percent reporting about Saturday and
Sunday. Designated persons must report about their activities on their designated day,
without any substitution of days…

The ATUS interview is a combination of structured questions and conversational
interviewing. It consists of four major topics: the household roster, the time diary, the
summary questions, and a section related to information collected in the eighth CPS
interview. The portion of the interview relating to the CPS is divided into four sections:
labor force status, looking for work, industry and occupation, and earnings and school
enrollment. These questions are used to update or confirm time-sensitive CPS data or
to fill in missing CPS data. Each section is described below in more detail…

For all parts of the interview except the collection of the time-use diary data (in
section 4, above), interviewers read scripted text on the CATI screen and enter the
reported responses.

For the time-use diary, the interviewer uses conversational interviewing rather than
asking scripted questions. This is a more flexible interviewing technique designed to
allow the respondent to report on his or her activities comfortably and accurately. This
technique also allows interviewers to use methods to guide respondents through memory lapses, to probe in a nonleading way for the level of detail required to code activities, and to redirect respondents who are providing unnecessary information. As each activity is reported, the interviewer records the verbatim responses on a new activity line. The interviewers are trained to ensure that the respondent reports
activities (and activity durations) actually done on the previous (diary) day, not
activities done on a “usual” day. Interviewers do this by placing continual emphasis on
the word “yesterday” throughout the interview.

This study relies on both a diary and asking questions about yesterday.