Tag Archives: statistics

Controversy in using sampling for the dicennial Census

Posted on by

In a story about the resignation of sociologist Robert Groves as director of the United States Census Bureau, there is an overview of some of the controversy over Groves’ nomination. The issue: the political implications of using statistical sampling.

Dr. Robert M. Groves announced on Tuesday that he was resigning his position as director of the U.S. Census Bureau in order to take over as provost of Georgetown University. “I’m an academic at heart,” Groves told The Washington Post. He will leave the Bureau in August. Unlike some government officials who recently have had to resign under a cloud, such as Regina Dugan of DARPA and Martha Johnson of the General Services Administration, Groves received universal praise for the job he did directing the 2010 Census, a herculean task he completed on time and almost $2 billion under budget.
At the time of Groves’ nomination, Rep. Darrell Issa, (R-California), chairman of the House Committee on Oversight and Government Reform, said that he found it “an incredibly troubling selection that contradicts the administration’s assurances that the census process would not be used to advance an ulterior political agenda.” However, by the time Groves announced that he was leaving, Issa had changed his tune and issued a statement that “His tenure is proof that appointing good people makes a big difference.”
When President Barack Obama nominated Groves on April 2, 2009, he was viewed as a generally uncontroversial professor of sociology.  However, his nomination turned out to be contentious anyway because his support for using statistical sampling, a statistical method commonly used to correct for errors and biases in the census, raised the ire of Republican critics, who believed that sampling would benefit minorities and the poor, who generally vote Democratic…
A specialist in survey methodology and statistics, Groves was no stranger to the Census Bureau, whose decennial census is one of the world’s largest and most sophisticated statistical exercises.  Groves served there early in his career as a visiting statistician in 1982, and later as associate director of Statistical Design, Standards, and Methodology from 1990 to 1992.  It was during the latter period that Groves became embroiled in the controversy over the proposed use of statistical sampling to correct known biases and deficiencies in the Census head count.  Groves and others at the Census Bureau proposed using sampling techniques to correct an admitted 1.2% undercount in the 1990 Census, which failed to include millions of homeless, minority and poor persons mainly living in big cities, which lost millions of dollars in federal funds when Republican Commerce Secretary Robert Mosbacher vetoed the sampling proposal.

Considering Groves’ track record in sociology, I’m not surprised that he is now regarded to have done a good job in this position.

Perhaps this is a silly question in today’s world but does everything have to become politicized? Is the ultimate goal to get the most accurate count of American residents or do both parties simply assume that the other side wants to use the occasion for political gain? If you want to limit funding to cities based on population, why not go after this funding rather than try to skew the count?

Of course, this is not the first time that the dicennial Census has been politicized…

Another note: a sociologist apparently saved the government $2 billion! That alone should draw some attention.

Statistics learning opportunity: “Hunger Games Survival Analysis”

Posted on by

Fun with statistics: a survival analysis of The Hunger Games (quick reviews of the books and movie). According to the final analysis, the only significant factor is the rating of each participant:

My interpretation of this is that the Gamemakers know what they’re doing when they assign the ratings. They’ve been doing this for years, so they give scores that are so accurate that they’re actually better predictors of survival time than whether a tribute is a volunteer, a Career, male or female, or forms an alliance. Pretty impressive.

An alternate and more cynical interpretation is that the Gamemakers are concerned about their own reputations and thus engineer the games so as to confirm their ratings, occasionally killing off players who do better or worse than expected based on the ratings, all so that the Gamemakers can look like they knew what they were doing all along. Unfortunately, the political system of Panem ranks so slow on Freedom House’s annual scores that we simply can’t tell what’s going on behind the scenes at all. To cut through their lies we simply need more data.

If you read this, you just also learn something about survival analysis and event history analysis. Bonus: the data and Stata code is also available for download!

Thinking about the event history class I took during grad school, we didn’t look at any data that was remotely close to popular culture.

Also, why not include the data from the second and third books? Granted, the games change a bit in the sequels to ratchet up the tension but that would provide more data to work with…

Five main methods of detecting patterns in data mining

Posted on by

Here is a summary of five of the main methods utilized to uncover patterns when data mining:

Anomaly detection : in a large data set it is possible to get a picture of what the data tends to look like in a typical case. Statistics can be used to determine if something is notably different from this pattern. For instance, the IRS could model typical tax returns and use anomaly detection to identify specific returns that differ from this for review and audit.

Association learning: This is the type of data mining that drives the Amazon recommendation system. For instance, this might reveal that customers who bought a cocktail shaker and a cocktail recipe book also often buy martini glasses. These types of findings are often used for targeting coupons/deals or advertising. Similarly, this form of data mining (albeit a quite complex version) is behind Netflix movie recommendations.

Cluster detection: one type of pattern recognition that is particularly useful is recognizing distinct clusters or sub-categories within the data. Without data mining, an analyst would have to look at the data and decide on a set of categories which they believe captures the relevant distinctions between apparent groups in the data. This would risk missing important categories. With data mining it is possible to let the data itself determine the groups. This is one of the black-box type of algorithms that are hard to understand. But in a simple example – again with purchasing behavior – we can imagine that the purchasing habits of different hobbyists would look quite different from each other: gardeners, fishermen and model airplane enthusiasts would all be quite distinct. Machine learning algorithms can detect all of the different subgroups within a dataset that differ significantly from each other.

Classification: If an existing structure is already known, data mining can be used to classify new cases into these pre-determined categories. Learning from a large set of pre-classified examples, algorithms can detect persistent systemic differences between items in each group and apply these rules to new classification problems. Spam filters are a great example of this – large sets of emails that have been identified as spam have enabled filters to notice differences in word usage between legitimate and spam messages, and classify incoming messages according to these rules with a high degree of accuracy.

Regression: Data mining can be used to construct predictive models based on many variables. Facebook, for example, might be interested in predicting future engagement for a user based on past behavior. Factors like the amount of personal information shared, number of photos tagged, friend requests initiated or accepted, comments, likes etc. could all be included in such a model. Over time, this model could be honed to include or weight things differently as Facebook compares how the predictions differ from observed behavior. Ultimately these findings could be used to guide design in order to encourage more of the behaviors that seem to lead to increased engagement over time.

Several of these seem similar to methods commonly used by sociologists:

1. Anomaly detection seems like looking for outliers. On one hand, outliers can throw off basic measures of central tendency or dispersion. On the other hand, outliers can help prompt researchers to reassess their models and/or theories to account for the unusual cases.

2. Cluster detection and/or classification appear similar to factor analysis. This involves a statistical analysis of a set of variables to see which ones “hang together.” This can be helpful for finding categories and reducing the number of variables in an analysis to a lesser number of important concepts.

3. Regression is used all the time both for modeling and predictions.

This all reminds me of what I heard in graduate school about the difference between data mining and statistical research: data mining amounted to atheoretical analysis. In other words, you might find relationships  between variables (or apparent relationships between variables – could always be a spurious association or there could be suppressor or distorter effects) but you wouldn’t have compelling explanations for these relationships. While you might be able to develop some explanations, this is a different process than hypothesis testing where you set out to look and test for relationships and patterns.

US government (and “statistical bureaucracy”) looking to measure well-being

Posted on by

The federal government is looking into ways to measure well-being as a new indicator of social life:

Funded by the U.S. Department of Health and Human Services, a panel of experts in psychology and economics, including Nobel laureate Daniel Kahneman, began convening in December to try to define reliable measures of “subjective well-being.” If successful, these could become official statistics.

But as the United States ventures into the squishy realm of feelings, statisticians will first have to define happiness and then how to measure it. Neither is a trivial matter. There is even some doubt whether people, when polled, can accurately say whether they are happy…

The panel, organized by the nonprofit National Academies, has already met with two of the key figures in the U.S. statistical bureaucracy: Robert Groves, the director of the U.S. Census Bureau, and Steve Landefeld, the director of the Bureau of Economic Analysis, the federal agency that puts out the gross domestic product figures.

According to proponents, a measure of happiness could help assess the success or failure of a range of government policies. It could gauge the virtues of a health benefit or establish whether education has more value than simply higher incomes. It might also detect extremes of inequality or imbalances in how people divide their time between work and leisure.

I’m not sure why there is opposition to this. There are plenty of social scientists who study this topic and have developed established measures of “happiness.” I’ve written on this topic a number of times looking at the effect of income on happiness, how religion leads to greater life satisfaction through interaction with others, and an argument that we need to study flourishing rather than happiness. As I’ve noted before, measuring happiness requires looking at both short- and long-term satisfaction. This panel may have to work on applying these measures onto a national scale but they are not creating a whole new field of study.

The cost issue may be driven more by the current budget troubles than anything else. If you are studying the effectiveness of programs and policies, why not include a measure of well-being? We tend to measure many things in terms of economics and pragmatic factors alone. Overall, it could make government statistics more holistic. A measure of well-being doesn’t have to be the only number that matters in the future but it can play an important role.

Three other thoughts:

1. The panel might consider avoiding the term “happiness” as this seems too subjective to a lot of people. In popular usage, the emotion is considered to be ephemeral. Instead, stick with well-being or life satisfaction.

2. Tying this panel to the idea of the “pursuit of happiness” in the Declaration of Independence seems silly. This doesn’t provide evidence for or against this sort of panel.

3. I’m very amused at the mention of a “statistical bureaucracy.” This might be the worst nightmare for some people: statistics plus government. Just a reminder: one member of the bureaucracy, Robert Graves at the Census Bureau, is a sociologist with a lot of experience with surveys.

 

The median college loan debt: $12,800

Posted on by

Growing calls for ways to deal with college loan debt can lead to a statistical question: just how much does “the average” college student owe? Here are some of the figures:

Meet Kelli Space. She went to Northeastern University to get a degree in sociology. And she graduated in $200,000 of student loan debt. In the economy’s newest trillion-dollar crisis, she is the 1 percent.

Kelli is not the face of America’s student debt problem. Among the 37 million people in this country with student loans to pay off, the median balance is $12,800. A whole 72 percent of borrowers have less than $25,000 left in debt, according to data from the Federal Reserve Bank of New York.

No, students like Kelli are the rarities, the white rhinos. Only about 5 percent of borrowers owe more than $75,000.

The key figures for me: the median is $12,800, meaning that half of people with student loans have less than this figure, half have more. Yet, nearly three-quarters of those with loans have less than $25,000 to pay off. Only 11% have more than $50k in debt.

So why do we keep hearing stories about those who owe mega amounts of money? Perhaps we might think of them as canaries in the mine shaft, students who show how bad the college finance system might be today. But, on the other hand, the statistics suggest that these students are rarities, people who have unusual debt. From these anecdotal and relatively rare stories, it seems like there is a pattern: a student goes to a prestigious school banking on the name of the school to pay off. (One common argument you will find online is that the major should be blamed – this usually puts more creative disciplines, the humanities, and subjects like sociology at the center of blame.) But, the name doesn’t always pay off, the student can’t find a good enough job to start paying off these debts, and the interest just continues to grow.

Overall, we need to work with the statistics more than the anecdotes: most college students do not have more than $25,000 of debt. This is not a small amount but it can be tackled (though the economy doesn’t help).

We need a more complex analysis of how taxes affect income inequality

Posted on by

One current blogosphere discussion about whether taxes could help reduce income inequality would benefit from more complex analyses. Here is the discussion thus far according to TaxProf Blog:

There have been a number of reports published recently that purport to show a link between rising inequality and changes in tax policy — especially tax cuts for the so-called rich. The latest installment comes from Berkeley professor Emmanuel Saez, Striking it Richer: The Evolution of Top Incomes in the United States.

Saez and others who write on this issue seem so intent on proving a link between tax policy and inequality that they overlook the major demographic changes that are occurring in America that can contribute to — or at least give the appearance of — rising inequality; a few of these being, differences in education, the rise of dual-earner couples, the aging of our workforce, and increased entrepreneurship.

Today, we will look at the link between education and income. Recent census data comparing the educational attainment of householders and income shows about as clearly as you can that America’s income gap is really an education gap and not the result of tax cuts for the rich.

The chart below shows that as people’s income rise, so too does the likelihood that they have a college degree or higher. By contrast, those with the lowest incomes are most likely to have a high school education or less. Just 8% of those at the lowest income level have a college degree while 78% of those earning $250,000 or more have a college degree or advanced degree. At the other end of the income scale, 69% of low-income people have a high school degree or less, while just 9% of those earning over $250,000 have just a high school degree.

This analysis starts in the right direction: looking at a direct relationship between two variables such as tax rates and income inequality is difficult to do in isolation of other factors. While some factors may be more influential than others, there are a number of reasons for income inequality. In other words, graphs with two variables are not enough. Pulling out one independent variable at a time doesn’t give us the full picture.

But, then the supposedly better way is that we were just looking at the wrong variable’s influence on income and should have been looking at education instead! So after saying that the situation was more complex, we get another two variable graph that shows that as education goes up, so does income so perhaps it really isn’t about taxes at all.

What we need here is some more complex statistical analysis, preferably including regression analysis where we can see how a variety of factors at the same time influence income inequality. Some of this might be a little harder to model since you would want to account for changing tax rates but arguing over two variable graphs isn’t going to get us very far. Indeed, I wonder if this is more common now in debates: both sides like simpler analyses because it allows each to make the point they want without considering the full complexity of the matter. In other words, easier to make graphs line up more with ideological commitments rather than an interest in truly sorting out what factors are more influential in affecting income inequality.

Credibility, statistics, and the legal profession

Posted on by

Elie Mystal at Above the Law has this take on a recent story involving credibility, statistics, and the legal profession:

This week, the law schools at Columbia, NYU, and Fordham have come under fire for their allegedly inflated employment statistics. A story in the New York Post specifically called out the top New York-area law schools for shady reporting of graduate outcomes when it comes to graduates employed by the schools….

I want to take a step back and look at what we’re really fighting about here: some of the best law schools in New York City have put out a statistic about how many graduates get jobs, and the New York Post and a bunch of other people immediately called “bulls**t.” Think about that. Even if the law schools can somehow convince people that, technically, their published information isn’t riddled with lies, we’re living in a world where such data can be assumed to be false absent a long and detailed explanation and discussion from the law schools. When somebody notices a discrepancy between a school’s numbers and what’s in the newspaper, we assume the school was full of crap, not that the newspaper got it wrong.

I suppose this isn’t very flattering to either newspapers or law schools.  Perhaps Americans now trust journalists more than lawyers (or at least legal educators)?

Commenting a few months ago on a scandal within academic sociology, Brian suggested several approaches to dealing with uncertain statistics:

This reminds me of Joel Best’s recommendations regarding dealing with statistics. One common option is to simply trust all statistics. Numbers look authoritative, often come from experts, and they can be overwhelming. Just accepting them can be easy. At the other pole is the common option of saying that all statistics are simply interpretation and are manipulated so we can’t trust any of them. No numbers are trustworthy. Neither approaches are good options but they are relatively easy options. The better route to go when dealing with scientific studies is to have the basic skills necessary to understand whether they are good studies or not and how the process of science works [emphasis added].

Brian’s point is a good one.  Unfortunately, it’s not possible to implement his “third way” here because the root problem is the lack of raw information rather than the inability to duplicate experimental/study results.  The question is not, in the theoretical abstract:  how many law students will get jobs when the economy is in condition X?  The question is rather:  as a matter of historical fact, how many 2010 law school graduates (or 2011, or 2009, or whatever) actually had jobs by date Y?

Elie faults the American Bar Association, which oversees and accredits law schools, for the current disaster of unreliable data:

The ABA is supposed to represent lawyers and law schools to the public. It’s supposed to relegate them so that the public can trust that moral and ethical standards are being upheld and enforced. And on that scale, the ABA has been an unmitigated failure. It’s done a disservice to all law schools. Nobody can trust any law school because the ABA has failed to impose effective oversight over all of them.

That’s tragic. A society is supposed to be proud of its institutions of higher learning, but the ABA has robbed us of that pride in our nation’s law schools. We no longer get to feel like our justice system is populated by people trained to the highest ethical standards, because we can’t even trust our law schools to tell us the truth about how many people got hired.

If the numbers published by law schools under the oversight of the ABA are unreliable, it goes without saying that it’s very difficult to derive these numbers through other means, especially in a form that allows for legitimate comparisons between schools and over time.  There are workarounds, of course, like journalistic attempts to compile and/or verify employment statistics independently of the law schools.  But those are obviously imperfect solutions, as The Economist recently noted in its analysis of the (surprisingly analogous) problem of Argentinan inflation statistics:

Since 2007 Argentina’s government has published inflation figures that almost nobody believes. These show prices as having risen by between 5% and 11% a year. Independent economists, provincial statistical offices and surveys of inflation expectations have all put the rate at more than double the official number (see article). The government has often granted unions pay rises of that order….

We [The Economist] hope that we can soon revert to an official consumer-price index for Argentina. That would require INDEC to be run by independent statisticians working unhindered. Until then, readers are better served by a credible unofficial figure than a bogus official one.

Unfortunately, for the foreseeable future, I think we’re going to need to start seeing a lot more credible unofficial figures out there, both for Argentinian inflation and for law school placement statistics.

“Copyright math”

Posted on by

Since this blog regularly covers issues ranging from intellectual property law to statistics, Rob Reid’s recent TED talk on “Copyright Math” seems particularly salient:

http://www.youtube-nocookie.com/embed/GZadCj8O1-0

Why two media sources ranking the world’s wealthiest people is a good thing

Posted on by

While Forbes had the corner on the market for years in compiling a ranking of the world’s richest people, there is now another option: this week Bloomberg released its Billionaires Index. One commentator thinks we don’t need both Forbes and Bloomberg examining this topic:

The Forbes list, available online today, is published every March. (Its companion, the “Forbes 400” list of richest Americans published in September.) It’s hard to not feel that Bloomberg’s outing takes some of the air out of Forbes usually-hyped cover story on who are the world’s richest people. This year’s edition proves unexciting not only because there were few shake-ups in the top spots from 2011’s list, but also because these rankings don’t appear all that different from Bloomberg’s.

Highlights from 2012’s version: With $69 billion, Mexico’s Carlos Slim Helu ranks No. 1 again for the third year in the row. (The magazine also profiled him.) Helu was followed by another 1,225 billionaires, starting with Bill Gates, Warren Buffett, and Bernard Arnault (of Louis Vuitton fame), who were also two through four last year. But beside no one being knocked off the top of this year’s Forbes list, it’s markedly similar to how rich Bloomberg News told us these folks were. Here’s a side-by-side comparison, with Forbes on the left and Bloomberg on the right.

So there are slight differences. Bloomberg has Arnault one spot lower and places fashion mogul Amancio Ortega down to seventh. Bloomberg puts the Koch brothers in the top 10, whereas Forbes had them both pegged at 12th. But isn’t this hair-splitting? If anything, the discrepancies show how hard it is to measure rich people’s riches.

What today’s Forbes list shows more than anything is that we don’t need two billionaires lists reminding us how wealthy the wealthy are. If we had to choose one, we’d go with Bloomberg’s, since it’s updated daily instead of once a year. But we doubt that will stop Forbes from producing its longstanding annual issue as long folks keep buying it.

I disagree. Here is why: I think that having two media sources looking at this topic will actually give readers better information. With two publications tackling the subject, I hope this improves their measurement of wealth for both publications. Perhaps we could average the rankings across the publications to get a more accurate assessment of what is going on. In the end, two sets of people looking at the data is better than one. Because Bloomberg is updating this list daily, perhaps this will push Forbes to update their lists more frequently and move away from a magazine era schedule to an Internet era schedule. The two lists do have some differences and this is not inconsequential. Lots of people are interested in this list and I’m sure some of the people at the top of the list have some interest in where they rank. Of course, these differences can indicate “how hard it is to measure rich people’s riches” but this doesn’t mean we should just throw up our hands and go with one list. Just because these people are really wealthy doesn’t mean that we shouldn’t have more fine-grained analysis of their financial holdings. (This sometimes seems to happen quite a bit in sociology: we assume we know about the elites and so spend more time studying marginalized groups but we have fewer in-depth studies of the elites who do have a lot of influence in society.)

A second issue: Bloomberg obviously thinks there is a market for another list that is updated daily and so this is a market decision as well as a journalistic interest in updating this information more frequently. The Forbes list always gets a lot of attention and Bloomberg probably wants to draw away some of that market. I imagine there is enough room in the market for both lists to survive, particularly as the two could serve different markets. However, it will be interesting to see how the rest of the media responds to changes in the Bloomberg list: if someone moves up from #3 to #2 in the next few days, will there be news stories about it? Will journalists providing background information about the wealthy reference the Forbes or the Bloomberg list?

Santorum claims college pushes people away from religion, experts push back

Posted on by

Republican presidential candidate Rick Santorum recently suggested that going to college pushes people away from the church and faith. Those who study the subject disagree:

Santorum told talk show host Glenn Beck on Thursday that “62% of kids who go into college with a faith commitment leave without it.”

Thom Rainer, president of LifeWay Christian Resources, a Nashville evangelical research and marketing agency, said, “There is no statistical difference in the dropout rate among those who attended college and those that did not attend college. Going to college doesn’t make you a religious drop out.”…

The real causes [of leaving the faith]: lack of “a robust faith,” strongly committed parents and an essential church connection, Rainer said.

“Higher education is not the villain,” said sociologist William D’Antonio of Catholic University of America. Since 1986, D’Antonio’s surveys of American Catholics have asked about Mass attendance, whether they rate their religion as very important in their life, and whether they have considered leaving Catholicism. The percentage of Catholics who scored low on all three points hovers between 18% in 1993 and 14% in 2011. But the percentage of people who are highly committed fell from 27% to 19%.

Recent research also disputes this: several 2011 studies found that those with education are actually more religious than those with less education.

So what was Santorum getting at with his statement? Three thoughts:

1. Conservative Christians commonly cite alarmist statistics to show that the church needs to redouble its efforts or to demonstrate that the church is under attack. See this classic article “Evangelicals Behaving Badly with Statistics,” a good article titled “Curing Christians’ Stats Abuse,” and the book Christians are Hate-Filled Hypocrites…

2. He is hitting back against “elitist academia,” responding to but also feeding the perception college classrooms are filled with atheists and agnostics who want to disabuse students of their faith. Of course, there are many people of faith in academia. This is a larger battle over a perceived liberal, atheist elite versus a faith-filled “average America.”

2a. If Santorum were correct, does this mean that people of faith should not send their kids to college? Or alternatively, do these ideas continue to boost attendance at religious colleges?

3. To compound matters, Santorum was talking to Glenn Beck and this argument was aimed at Beck’s audience. At the same time, it appears Santorum made this a more general argument on the campaign trail:

“President Obama once said he wants everybody in America to go to college. What a snob,” Santorum said Saturday at a campaign stop in Troy, Mich. “There are good, decent men and women who go out and work hard every day and put their skills to test that aren’t taught by some liberal college professor that [tries] to indoctrinate them.”

In the end, this seems like another plank in a moral argument, rather than a political or social argument, for Republicans.