Tag Archives: methodology

Reassessing Mead versus Freeman in their studies of Samoa

Posted on by

A new look at anthropologist Derek Freeman’s critique of Margaret Mead’s famous study of sex in Samoa suggests Freeman may have manipulated data:

But Shankman’s new analysis — following his excellent 2009 book, The Trashing of Margaret Mead: Anatomy of an Anthropological Controversy — shows that Freeman manipulated “data” in ways so egregious that it might be time for Freeman’s publishers to issue formal retractions…Now Shankman has delved even deeper into the sources; in 2011, he obtained from Freeman’s archives the first key interview with one of the supposed “joshing” informants, a woman named Fa’apua’a. This interview, conducted in 1987, allegedly bolstered Freeman’s contention that Mead had based her “erroneous” portrait of Samoan sexuality on what Fa’apua’a and her friend Fofoa had jokingly told Mead back in the 1920s.

But Shankman shows that the interview was conducted and then represented in deeply problematic ways. The 1987 interview with Fa’apua’a was arranged and carried out by Fofoa’s son, a Samoan Christian of high rank who was convinced that Mead had besmirched the reputation of Samoans by portraying his mother, her friend Fa’apua’a, and other Samoans as sexually licentious…

But why did Freeman get it so wrong? Shankman’s book suggests Freeman was obsessed with Mead and with what he saw as her dangerous stories about the flexibility of human cultures. He saw himself as a brave “heretic,” a man saving true science from Mead’s mere ideology.

I wonder if Shankman’s work is the start to a solution to this debate. If two anthropologists disagree so much, wouldn’t bringing in other anthropologists to review the data or conduct their own fieldwork a possible answer to adjudicating who got it more right? There is a time factor here that makes the issue more complicated but people in addition to Shankman could review the notes and comparisons could be made to other societies which might be similar and offer insights.

More broadly, I wonder how much incentive there is for researchers to follow up on famous studies. Freeman made a name for himself by arguing against Mead’s famous findings but what if he had gone through the trouble and then found Mead was right? He likely would not have gotten very far.

 

Trying to ensure more accountability in US News & World Report college ranking data

Posted on by

The US News & World Report college rankings are big business but also a big headache in data collection. The company is looking into ways to ensure more trustworthy data:

A new report from The Washington Post‘s Nick Anderson explores the increasingly common problem, in which universities submit inflated standardized test scores and class rankings for members of their incoming classes to U.S. News, which doesn’t independently verify the information. Tulane University, Bucknell University, Claremont McKenna College, Emory University, and George Washington University have all been implicated in the past year alone. And those are just the schools that got caught:

A survey of 576 college admissions officers conducted by Gallup last summer for the online news outlet Inside Higher Ed found that 91 percent believe other colleges had falsely reported standardized test scores and other admissions data. A few said their own college had done so.

For such a trusted report, the U.S. News rankings don’t have many safeguards ensuring that their data is accurate. Schools self-report these statistics on the honor system, essentially. U.S. News editor Brian Kelly told Inside Higher Ed’s Scott Jaschik, “The integrity of data is important to everybody … I find it incredible to contemplate that institutions based on ethical behavior would be doing this.” But plenty of institutions are doing this, as we noted back in November 2012 when GWU was unranked after being caught submitting juiced stats. 

At this point, U.S. News shouldn’t be surprised by acknowledgment like those from Tulane and Bucknell. It turns out that if you let schools misreport the numbers — especially in a field of fierce academic competition and increasingly budgetary hardship — they’ll take you up on the offer. Kelly could’ve learned that by reading U.S. News‘ own blog, Morse Code. Written by data researcher Bob Morse, almost half of the recent posts have been about fraud. To keep schools more honest, the magazine is considering requiring university officials outside of enrollment offices to sign a statement vouching for submitted numbers. But still, no third party accountability would be in place, and many higher ed experts are already saying that the credibility of the U.S. News college rankings is shot.

Three quick thoughts:

1. With the amount of money involved in the entire process, this should not be a surprise. Colleges want to project the best image they can so having a weakly regulated system (and also a suspect methodology and set of factors to start with) can lead to abuses.

2. If the USNWR rankings can’t be trusted, isn’t there someone who could provide a more honest system? This sounds like an opportunity for someone.

3. I wonder if there are parallels to PED use in baseball. To some degree, it doesn’t matter if lots of schools are gaming the system as long as the perception among schools is that everyone else is doing it. With this perception, it is easier to justify one’s own cheating because colleges need to catch up or compete with each other.

Chicago traffic bad and, perhaps worse, unpredictable

Posted on by

Having heavy traffic is bad enough but Chicago also has unpredictable traffic, according to a new report.

Residents of the Chicago area are accommodating that increasing uncertainty by setting aside more time each day — just in case — for the commute, new research shows.

For the most important trips, such as going to work, medical appointments, the airport or making a 5:30 p.m. pickup at the child care center to avoid late fees, drivers in northeastern Illinois and northwest Indiana should count on allotting four times as much time as it would take to travel in free-flowing traffic, according to the “Urban Mobility Report” to be released Tuesday by the Texas A&M Transportation Institute. The analysis is based on 2011 data, which are the most recent available.

It is the first time that travel reliability was measured in the 30-year history of the annual report. The researchers created a Planning Time Index geared toward helping commuters reach their destinations on time in more than 95 percent of the trips. A second index, requiring less padding of travel time, would get an employee to work on time four out of five days a week…

The Chicago region ranked No. 7 among very large urban areas and 13th among 498 U.S. cities on a scale of the most unreliable highway travel times. The Washington area was the worst. A driver using the freeway system in the nation’s capital and surrounding suburbs should budget almost three hours to complete a high-priority trip that would take only 30 minutes in light traffic, the study said.

This sounds like an interesting new way to measure traffic. The absolute amount of time spent in traffic is interesting in itself but this study gives us a sort of confidence interval for time spent in traffic. This suggests that traffic is not just an issue of getting stuck but it is the threat of getting stuck that would affect a lot of behavior. Just the threat could lead to a lot more lost time and productivity.

It would also be interesting to look at how often the average driver gives themselves this time cushion. Could traffic be improved if people planned to take more time to get to their destination?

Mapping NFL fandom by county with Facebook likes

Posted on by

Facebook has put their massive data trove to use and examined the geographies of NFL fandom. Here is what they came up with:

The National Football League is one of the most popular sports in America with some incredibly devoted fans. At Facebook we have about 35 million account holders in the United States who have Liked a page for one of the 32 teams in the league, representing one of the most comprehensive samples of sports fanship ever collected. Put another way, more than 1 in 10 Americans have declared their support for an NFL team on Facebook…

While winning seems to matter, NFL teams have local followings that are probably heavily influenced by family ties and/or where a person grew up,  so we were obviously curious to see where the fans for various teams live now. By considering the physical locations of NFL fans, we can construct a map of the top team for each county in the US. It tells an interesting story about the ways that football rivalries and allegiances alternately divide and unite the country, and sometimes even individual states.

In some cases, whole states and even entire regions of the country uniformly support a single team.  For instance the Vikings are easily the only game in town in Minnesota, while New England appears to be comprised of entirely Patriots fans except for a small portion of Connecticut.

There are some states which are divided into regions by teams.  Florida has three teams–the Tampa Bay Bucs, Miami Dolphins, and the Jacksonville Jaguars–and Facebook users there seems fractured in their support, with some counties even defecting to teams from the North. Ohio is another interesting story, with the Cleveland Browns in the North, Cincinatti Bengals in the South, and Pittsburgh Steelers fans occupying the middle of the state.

Some teams, like the Steelers, Cowboys, and Packers, seem to transcend geography, with pockets of fans all over the country. On the other end of the spectrum, the Jets have to share New York with the Giants and are only the most popular team for a single stronghold county in Long Island.

Five quick thoughts:

1. There are few other organizations that could put together such a map without undertaking a major survey (since this is measured at the county level).

2. The best part for Facebook: users voluntarily provided this data.

3. Could Facebook end up being the most important future source for telling us about American society? There are still difficulties: users have to opt in (in this particular case, they had to “like” a NFL team), not everyone is involved (though it seems like pretty close), and not all users are putting everything in their profiles.

4. Is there a way to weight this map with population density? For example, the Cowboys may have a really broad geographic reach but many of those counties have fewer people. In contrast, teams like the Jets or Eagles have smaller reaches yet more people live in those areas.

5. Is there a way to show the percentage of county respondents who liked the dominant team? I imagine there are plenty of counties where one team does not have a strong majority, let alone even much of a plurality. For example, Jets fans barely show up on the map because they are only the top team in one county. Yet, there are plenty of Jets fans.

Bill Gates: we can make progress with goals, data, and a feedback loop

Posted on by

Bill Gates argues in the Wall Street Journal that significant progress can be made around the world if organizations and residents participate in a particular process:

In the past year, I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal—in a feedback loop similar to the one Mr. Rosen describes.

This may seem basic, but it is amazing how often it is not done and how hard it is to get right. Historically, foreign aid has been measured in terms of the total amount of money invested—and during the Cold War, by whether a country stayed on our side—but not by how well it performed in actually helping people. Closer to home, despite innovation in measuring teacher performance world-wide, more than 90% of educators in the U.S. still get zero feedback on how to improve.

An innovation—whether it’s a new vaccine or an improved seed—can’t have an impact unless it reaches the people who will benefit from it. We need innovations in measurement to find new, effective ways to deliver those tools and services to the clinics, family farms and classrooms that need them.

I’ve found many examples of how measurement is making a difference over the past year—from a school in Colorado to a health post in rural Ethiopia. Our foundation is supporting these efforts. But we and others need to do more. As budgets tighten for governments and foundations world-wide, we all need to take the lesson of the steam engine to heart and adapt it to solving the world’s biggest problems.

Gates doesn’t use this term but this sounds like a practical application of the scientific method. Instead of responding to a social problem by going out and trying to “do something,” the process should be more rigorous, involve setting goals, collecting good data, interpreting the data, and then adjusting the process from the beginning. This is related to other points about this process:

1. It is one thing to be able to collect data (and this is often its own complicated process) but it is another to know what to do with it once you have it. Compared to the past, data is relatively easy to obtain today but using it well is another matter.

2. Another broad issue in this kind of feedback loop is developing the measurements and what counts as “success.” Some of this is fairly easy; when Gates praises the UN Millennium Goals, reducing occurrences of disease or boosting incomes has face validity for getting at what matters. But, measuring teacher’s performances or what makes a quality college are a little trickier to define in the first place. Gates calls this developing goals but this could be a lengthy process in itself.

It is interesting that Gates mentions the need for such loops in colleges so that students “could know where they would get the most for their tuition money.” The Gates Foundation has put money into studying public schools and just a few weeks ago released some of their findings:

After a three-year, $45 million research project, the Bill and Melinda Gates Foundation believes it has some answers.

The most reliable way to evaluate teachers is to use a three-pronged approach built on student test scores, classroom observations by multiple reviewers and teacher evaluations from students themselves, the foundation found…

The findings released Tuesday involved an analysis of about 3,000 teachers and their students in Charlotte; Dallas; Denver; Memphis; New York; Pittsburgh; and Hillsborough County, Fla., which includes Tampa. Researchers were drawn from the Educational Testing Service and several universities, including Harvard, Stanford and the University of Virginia…

Researchers videotaped 3,000 participating teachers and experts analyzed their classroom performance. They also ranked the teachers using a statistical model known as value-added modeling, which calculates how much an educator has helped students learn based on their academic performance over time. And finally, the researchers surveyed the students, who turned out to be reliable judges of their teacher’s abilities, Kane said.

All this takes quite a few resources and time. For those interested in quick action, this is not the process to follow. Hopefully, however, the resources and time pay off with better solutions.

Average new house size expected to drop to 2,150 square feet by 2015

Posted on by

A short report on McMansions links to a National Association of Home Builders survey that suggests building professionals believe the square footage of the average new American house will fall by 2015:

Respondents expect the average, new single-family detached home in 2015 to be about 2,152 square feet, 10 percent smaller than the average size of single-family homes started in the first three quarters of 2010. Overall, 63 percent of respondents expect the average size of new homes in 2015 to be somewhere between 2,000 square feet and 2,399 square feet, 22 percent expect it to be between 2,400 square feet and 2,999 square feet, while 13 percent expect it to only be 1,600 square feet to 1,999 square feet (Figure 2).

Figure 2. Average Home Size in 2015

Data from the Census Bureau indicates that the average size of single-family homes completed peaked in 2007, at 2,521 square feet, was virtually unchanged in 2008, and then declined in 2009 to 2,438 square feet. Preliminary data for 2010 shows a further decline, down to 2,377 square feet. Although part of the recent drop in average home size may indeed be temporary due to hard economic times, a number of factors lead building professionals to expect home size declines in the long-run: consumers are focused on lowering the cost of heating and cooling their homes; they no longer have sizeable equity in their current homes to finance a much larger one; diminished expectations for house price appreciation has reduced demand for extra square footage in order to achieve appreciation on a larger base; demographics, 29 percent of the US population will be 55+ in the year 2020, demanding smaller homes; and strict mortgage underwriting for the foreseeable future. Combined, these factors will weigh on the consumer to purchase homes based on need more than want.

My interpretation of this is that a majority of builders think new homes in 2015 will be slightly smaller than new homes of today. Additionally, 23% still believe new homes will be larger than 2,400 square feet. Interestingly, there is not reported evidence of whether building professionals think these smaller new homes of the future will be cheaper.

And here is where square footage will be dropped from these future houses:

To save on square footage, the living room is high on the endangered list – 52 percent of builders expect it to be merged with other spaces in the home by 2015 and 30 percent said it will vanish entirely.

“As an overall share of total floor space, 54 percent of builders said the family room is likely to increase,” said Rose Quint, NAHB’s assistant vice president for survey research. “That makes it the only area of the home likely to get bigger.”

In addition, the relative size of the entry foyer and dining room are likely to be diminished by 2015. However, opinions were fairly evenly divided on the fate of the kitchen, master bedroom and bath and mudroom, she said.

The survey methodology is also worth noting – it was sent to a lot of interested parties but the response rate was under 10%:

NAHB’s The New Home in 2015 survey was sent electronically to 3,019 builders, designers, architects, manufacturers, and marketing specialists. The sample was stratified by region of the country (to be proportional to housing starts in each of the four Census regions) and, among builders, by their number of units started.

A total of 238 responses were received, of which 30 percent came from single-family builders, 19 percent from architects, 26 percent from designers, 7 percent from manufacturers, and 18 percent from “other” building industry professionals.

At first glance, this suggests to me that the findings are quite untrustworthy.

Highlights from the “Illinois’s 33%” poverty report

Posted on by

A new report from the Social Impact Research Center, “Illinois’s 33%,”  looks at poverty in Illinois. Here are a few highlights:

1. Something I did not realize: the preamble to the Illinois Constitution mentions “eliminat[ing] poverty” (p.1).

“We, the People of the State of Illinois…in order to provide for the health, safety and welfare of the people; maintain a representative and orderly government; eliminate poverty and inequality; assure legal, social and economic justice; provide opportunity for the fullest development of the individual; ensure domestic tranquility; provide for the common defense; and secure the blessings of freedom and liberty to ourselves and our posterity—do ordain and establish this Constitution for the State of Illinois.”

2. The report is not just about poverty; it is also about people in near-poverty. The income thresholds for this are here (p.5):

This methodology of measuring people with low incomes or near poverty seems to be growing. The Census reports the median household income in Illinois is $56,576.

3. There is definitely some geographic disparity in these figures. Here are the numbers for the Chicago region which clearly shows wealthier and less wealthy counties and Chicago neighborhoods (p.7):

I did not see any calls for metropolitan approaches to poverty. In the Chicago region, it would be difficult to deal with a particular problem, say affordable housing, in just Chicago or a few of its neighborhoods without cooperation and input from others in the region.

4. The report has more figures and possible solutions in five areas that could help people move out of poverty: employment, education, housing, health & nutrition, and assets (p.3-4, 15-17).

Confessions of researchers: #overlyhonestmethods

Posted on by

Here is a collection of 17 post under the Twitter hashtag #overlyhonestmethods. My favorite: “We assume 50 Ivy League kids represent the general population, b/c ‘real people’ can be sketchy or expensive.” This doesn’t surprise me considering the number of undergraduates used in psychology studies…

I wonder how many researchers could tell similar stories about research methods. These admissions don’t necessarily invalidate any of the findings but rather hint at the very human dimension present in conducting research studies.

(Disclaimer: of course it is difficult to know how many of these research method confessions are true.)

Call for more social science modeling for Social Security

Posted on by

An op-ed in the New York Times explains how poorly financial forecasts for Social Security are made and suggests social scientists can help:

Remarkably, since Social Security was created in 1935, the government’s forecasting methods have barely changed, even as a revolution in big data and statistics has transformed everything from baseball to retailing.

This omission can be explained by the fact that the Office of the Chief Actuary, the branch of the Social Security Administration that is responsible for the forecasts, is almost exclusively composed of, well, actuaries — without any serious representation of statisticians or social science methodologists. While these actuaries are highly responsible and careful and do excellent work curating and describing the data that go into the forecasts, their job is not to make statistical predictions. Yet the agency badly needs such expertise.

With considerable help from the actuaries and other officials at the Social Security Administration, we unearthed how the agency makes mortality forecasts and uses them to predict the program’s solvency. We learned that the methods are antiquated, subjective and needlessly complicated — and, as a result, are prone to error and to potential interference from political appointees. This may explain why the agency’s forecasts have, at times, changed significantly from year to year, even when there was little change in the underlying data.

We have made our methods, calculations and software available online at j.mp/SSecurity so that others can replicate or improve our forecasts. The implications of our findings go beyond social science. As the wave of retirement by the baby boomers continues, doing nothing to shore up Social Security’s solvency is irresponsible. If the amount of money coming in through payroll taxes does not increase and if the amount of money going out as benefits remains the same, the trust funds will become insolvent less than 20 years from now.

Sociologists seem to be looking for ways to get involved in major policy issues so perhaps this is one way to do that. It is also interesting to note this op-ed is based on a 2012 article in Demography titled “Statistical Security for Social Security.” Not too many articles can make such a claim…

Also, I’m sure this doesn’t inspire confidence among some for the government’s ability to keep track of all of its data. Does the federal government have the ability to hire and train the kind of people it needs? Can it compete with the private sector or political campaigns (think of what the lauded 2012 Obama campaign big data workers might be able to do)?

Patterns in “the most cited works in sociology, 2012 edition”

Posted on by

According to Neal at Scatterplot, here are the most cited sociological books and articles of 2012:

top25_2012

This is an interesting list. Three of the patterns in the data:

So, one in 33 articles cites Distinction. The majority at the top of the list are books along with a pair each from AJS, ASR and the Annual Review, along with one article from Social Forces. The authors and titles are truncated by Web of Science, so don’t blame me. Remember that the lists only counts citations in this group of sociology journals, so being famous in other worlds doesn’t get you on the list.

Fun fact: 2/3 of things that were cited last year were only cited once, and 95% of things cited were cited less than five times. And, unless one of your articles was cited nine or more times in one of these journals last year, you can consider yourself, like me, one of the 99%.

One thing that struck me was how old everything  on this top list was. The median publication year in the top 100 was 1992. Of the top 100, only one piece was published in the last five years.

A few other things stuck out to me from this list:

1. The list involves a number of big name sociologists. I assume they became big names because of the quality of their work, such as in the pieces cited here, but how much could it be that the works are cited more because they came from big names? There is some interesting work that could be done here with individual pieces to look at patterns of citations and how works become well-known.

2. There are several more methodological pieces on the list. The Raudenbush and Bryk 2002 book involves hierarchical linear modeling, a technique that uses multiple equations to nest individual cases within larger groups (like students within schools in the sociology of education). The Strauss and Glaser 1967 book is about the basics of grounded theory, a technique that has been adopted across a variety of qualitative studies. The Steensland et al. 2000 piece is about developing the measure RELTRAD which more effectively categorizes Americans into religious traditions. These methodological works have wide applications and were influential across a variety of subfields.

3. Could we interpret a list like this as one that tell us the “classic works” of sociology today? Could we hand a list like this to undergraduate majors or graduate students and tell them that this is what they need to know to understand the broader field? One way to check on this would be to compare the top cited works year to year to see how much the list changes and how consistently important these works are. Presumably, new works will be added to the list over time but this may not happen quickly.