Tag Archives: methodology

Why Public Policy Polling (PPP) should not conduct “goofy polls”

Posted on by

Here is an explanation why the polling firm Public Policy Polling (PPP) conducts “goofy polls”:

But over the past year, PPP has been regularly releasing goofy, sometimes pointless polls about every other month. In early January, one such survey showed that Congress was less popular than traffic jams, France and used-car salesmen. According to their food-centric surveys released this week, Americans clearly prefer Ronald McDonald over Burger King for President; Democrats are more likely to get their chicken at KFC than Chick-fil-A, and Republicans are more apt to order pancakes than waffles. “We’re obviously doing a lot of polling on the key 2014 races,” says Jensen. “That kind of polling is important. We also like to do some fun polls.”

PPP, which has a left-leaning reputation, releases fun polls in part because they’re entertaining but mostly in an attempt to set themselves apart as an approachable polling company. Questions for polls are sometimes crowd-sourced via Twitter. The outfit does informal on-site surveys about what state they should survey next. And when the results of offbeat polls come out, the tidbits have potential to go viral. “We’re not trying to be the next Gallup or trying to be the next Pew,” Jensen says. “We’re really following a completely different model where we’re known for being willing to poll on stuff other people aren’t willing to poll on.” Like whether Republicans are willing to eat sushi (a solid 64% are certainly not).

Which means polls about “Mexican food favorability” are a publicity stunt on some level. Jensen says PPP, which has about 150 clients, gets more business from silly surveys and the ethos it implies than they do cold-calling. One such client was outspoken liberal Bill Maher, who hired PPP to poll for numbers he could use on his HBO show Real Time. That survey, released during the 2012 Republican primaries, found that Republicans were more likely to vote for a gay candidate than an atheist candidate—and that conservative virgins preferred Mitt Romney, while Republicans with 20 or more sexual partners strongly favored Ron Paul.

Jensen argues that the offbeat polls do provide some useful information. One query from the food survey, for instance, asks respondents whether they consider themselves obese: about 20% of men and women said yes, well under the actual American obesity rate of 35.7%.  Information like that could give health crusaders some fodder for, say, crafting public education PSAs. Still, the vast majority of people are only going to use these polls to procrastinate at work: goodness knows it’s hard to resist a “scientific” analysis of partisans’ favorite pizza toppings (Republicans like olives twice as much!).

Here is my problem with this strategy: it is short-sighted and privileges PPP. While polling firms do need to market themselves as there are a number of organizations that conduct national polls, this strategy can harm the whole field. When the average American sees the results of “goofy polls,” is it likely to improve their view of the polling in general? I argue there is already enough suspicion in America about polls and their validity without throwing in polls that don’t tell us as much. This suspicion contributes to lower response rates across the board, a problem for all survey researchers.

In the end, the scientific nature of polling takes a hit when any firm is willing to reduce polling to marketing.

How can Lake County, Illinois be #9 on the list of “America’s Most Miserable Cities”?

Posted on by

Forbes just put out their 2013 list of “America’s Most Miserable Cities.” Out of the top 20, there is one that is not like the others: Lake County, Illinois at #9. Here is the short description of why Lake County made the list:

The Chicago suburb is one of the richest counties in the U.S., as measured by per capita income. But home prices are down 29% over the past 5 years. Other drawbacks: long commutes and lousy weather.

There are numerous problems with this:

1. Calling an entire county a suburb is strange. Lake County is made up of dozens of suburbs which are quite varied. For example, look at quick overviews of Deerfield versus Grayslake versus Waukegan. Lumping them all together is silly and is one of the traps many people make when looking at the suburbs: they are not all the same kind of places.

2. How does a county end up on this list when the rest of the top 20 are cities? In terms of categories, a suburban county is not in the same category as a city. While there might be some identity in saying one is from “Lake County,” it is nowhere close to being a singular city.

3. Just glancing at this description and the top 20 cities on the list, I have to wonder how Lake County could even make the list. According to this list, Lake County is the 56th wealthiest county in the United States with a median household income of $74,266. Here is a bit more on the methodology:

We looked at the 200 largest metropolitan statistical areas and divisions in the U.S. to determine America’s Most Miserable Cities. The minimum population to be eligible was 259,000. We ranked each area on 9 factors, including average unemployment rate between 2010 and 2012; median commute times to work for 2011 based on U.S. Census data; violent crimes per capita from the FBI’s 2011 Uniform Crime Report.

We included three housing metrics: the change in median home prices between 2009 and 2012; foreclosure rates in 2012, as compiled by RealtyTrac; and property tax rates based on median real estate taxes paid and median home values in 2011 per the U.S. Census. We factored in income tax rates and the weather in each metro on factors relating to temperature, precipitation and humidity. The data metrics are weighted equally in the final scoring.

We tweaked the methodology in this year’s list in response to feedback from readers, dropping our rankings of both pro sports team success and political corruption, since both were based on regional, rather than city-specific data. We also added a new measure—net migration—which we see as a clear gauge of whether or not residents feel a community is worth living in.

If this methodology puts Lake County at #9, Forbes may want to revisit their criteria.

Reassessing Mead versus Freeman in their studies of Samoa

Posted on by

A new look at anthropologist Derek Freeman’s critique of Margaret Mead’s famous study of sex in Samoa suggests Freeman may have manipulated data:

But Shankman’s new analysis — following his excellent 2009 book, The Trashing of Margaret Mead: Anatomy of an Anthropological Controversy — shows that Freeman manipulated “data” in ways so egregious that it might be time for Freeman’s publishers to issue formal retractions…Now Shankman has delved even deeper into the sources; in 2011, he obtained from Freeman’s archives the first key interview with one of the supposed “joshing” informants, a woman named Fa’apua’a. This interview, conducted in 1987, allegedly bolstered Freeman’s contention that Mead had based her “erroneous” portrait of Samoan sexuality on what Fa’apua’a and her friend Fofoa had jokingly told Mead back in the 1920s.

But Shankman shows that the interview was conducted and then represented in deeply problematic ways. The 1987 interview with Fa’apua’a was arranged and carried out by Fofoa’s son, a Samoan Christian of high rank who was convinced that Mead had besmirched the reputation of Samoans by portraying his mother, her friend Fa’apua’a, and other Samoans as sexually licentious…

But why did Freeman get it so wrong? Shankman’s book suggests Freeman was obsessed with Mead and with what he saw as her dangerous stories about the flexibility of human cultures. He saw himself as a brave “heretic,” a man saving true science from Mead’s mere ideology.

I wonder if Shankman’s work is the start to a solution to this debate. If two anthropologists disagree so much, wouldn’t bringing in other anthropologists to review the data or conduct their own fieldwork a possible answer to adjudicating who got it more right? There is a time factor here that makes the issue more complicated but people in addition to Shankman could review the notes and comparisons could be made to other societies which might be similar and offer insights.

More broadly, I wonder how much incentive there is for researchers to follow up on famous studies. Freeman made a name for himself by arguing against Mead’s famous findings but what if he had gone through the trouble and then found Mead was right? He likely would not have gotten very far.

 

Trying to ensure more accountability in US News & World Report college ranking data

Posted on by

The US News & World Report college rankings are big business but also a big headache in data collection. The company is looking into ways to ensure more trustworthy data:

A new report from The Washington Post‘s Nick Anderson explores the increasingly common problem, in which universities submit inflated standardized test scores and class rankings for members of their incoming classes to U.S. News, which doesn’t independently verify the information. Tulane University, Bucknell University, Claremont McKenna College, Emory University, and George Washington University have all been implicated in the past year alone. And those are just the schools that got caught:

A survey of 576 college admissions officers conducted by Gallup last summer for the online news outlet Inside Higher Ed found that 91 percent believe other colleges had falsely reported standardized test scores and other admissions data. A few said their own college had done so.

For such a trusted report, the U.S. News rankings don’t have many safeguards ensuring that their data is accurate. Schools self-report these statistics on the honor system, essentially. U.S. News editor Brian Kelly told Inside Higher Ed’s Scott Jaschik, “The integrity of data is important to everybody … I find it incredible to contemplate that institutions based on ethical behavior would be doing this.” But plenty of institutions are doing this, as we noted back in November 2012 when GWU was unranked after being caught submitting juiced stats. 

At this point, U.S. News shouldn’t be surprised by acknowledgment like those from Tulane and Bucknell. It turns out that if you let schools misreport the numbers — especially in a field of fierce academic competition and increasingly budgetary hardship — they’ll take you up on the offer. Kelly could’ve learned that by reading U.S. News‘ own blog, Morse Code. Written by data researcher Bob Morse, almost half of the recent posts have been about fraud. To keep schools more honest, the magazine is considering requiring university officials outside of enrollment offices to sign a statement vouching for submitted numbers. But still, no third party accountability would be in place, and many higher ed experts are already saying that the credibility of the U.S. News college rankings is shot.

Three quick thoughts:

1. With the amount of money involved in the entire process, this should not be a surprise. Colleges want to project the best image they can so having a weakly regulated system (and also a suspect methodology and set of factors to start with) can lead to abuses.

2. If the USNWR rankings can’t be trusted, isn’t there someone who could provide a more honest system? This sounds like an opportunity for someone.

3. I wonder if there are parallels to PED use in baseball. To some degree, it doesn’t matter if lots of schools are gaming the system as long as the perception among schools is that everyone else is doing it. With this perception, it is easier to justify one’s own cheating because colleges need to catch up or compete with each other.

Chicago traffic bad and, perhaps worse, unpredictable

Posted on by

Having heavy traffic is bad enough but Chicago also has unpredictable traffic, according to a new report.

Residents of the Chicago area are accommodating that increasing uncertainty by setting aside more time each day — just in case — for the commute, new research shows.

For the most important trips, such as going to work, medical appointments, the airport or making a 5:30 p.m. pickup at the child care center to avoid late fees, drivers in northeastern Illinois and northwest Indiana should count on allotting four times as much time as it would take to travel in free-flowing traffic, according to the “Urban Mobility Report” to be released Tuesday by the Texas A&M Transportation Institute. The analysis is based on 2011 data, which are the most recent available.

It is the first time that travel reliability was measured in the 30-year history of the annual report. The researchers created a Planning Time Index geared toward helping commuters reach their destinations on time in more than 95 percent of the trips. A second index, requiring less padding of travel time, would get an employee to work on time four out of five days a week…

The Chicago region ranked No. 7 among very large urban areas and 13th among 498 U.S. cities on a scale of the most unreliable highway travel times. The Washington area was the worst. A driver using the freeway system in the nation’s capital and surrounding suburbs should budget almost three hours to complete a high-priority trip that would take only 30 minutes in light traffic, the study said.

This sounds like an interesting new way to measure traffic. The absolute amount of time spent in traffic is interesting in itself but this study gives us a sort of confidence interval for time spent in traffic. This suggests that traffic is not just an issue of getting stuck but it is the threat of getting stuck that would affect a lot of behavior. Just the threat could lead to a lot more lost time and productivity.

It would also be interesting to look at how often the average driver gives themselves this time cushion. Could traffic be improved if people planned to take more time to get to their destination?

Mapping NFL fandom by county with Facebook likes

Posted on by

Facebook has put their massive data trove to use and examined the geographies of NFL fandom. Here is what they came up with:

The National Football League is one of the most popular sports in America with some incredibly devoted fans. At Facebook we have about 35 million account holders in the United States who have Liked a page for one of the 32 teams in the league, representing one of the most comprehensive samples of sports fanship ever collected. Put another way, more than 1 in 10 Americans have declared their support for an NFL team on Facebook…

While winning seems to matter, NFL teams have local followings that are probably heavily influenced by family ties and/or where a person grew up,  so we were obviously curious to see where the fans for various teams live now. By considering the physical locations of NFL fans, we can construct a map of the top team for each county in the US. It tells an interesting story about the ways that football rivalries and allegiances alternately divide and unite the country, and sometimes even individual states.

In some cases, whole states and even entire regions of the country uniformly support a single team.  For instance the Vikings are easily the only game in town in Minnesota, while New England appears to be comprised of entirely Patriots fans except for a small portion of Connecticut.

There are some states which are divided into regions by teams.  Florida has three teams–the Tampa Bay Bucs, Miami Dolphins, and the Jacksonville Jaguars–and Facebook users there seems fractured in their support, with some counties even defecting to teams from the North. Ohio is another interesting story, with the Cleveland Browns in the North, Cincinatti Bengals in the South, and Pittsburgh Steelers fans occupying the middle of the state.

Some teams, like the Steelers, Cowboys, and Packers, seem to transcend geography, with pockets of fans all over the country. On the other end of the spectrum, the Jets have to share New York with the Giants and are only the most popular team for a single stronghold county in Long Island.

Five quick thoughts:

1. There are few other organizations that could put together such a map without undertaking a major survey (since this is measured at the county level).

2. The best part for Facebook: users voluntarily provided this data.

3. Could Facebook end up being the most important future source for telling us about American society? There are still difficulties: users have to opt in (in this particular case, they had to “like” a NFL team), not everyone is involved (though it seems like pretty close), and not all users are putting everything in their profiles.

4. Is there a way to weight this map with population density? For example, the Cowboys may have a really broad geographic reach but many of those counties have fewer people. In contrast, teams like the Jets or Eagles have smaller reaches yet more people live in those areas.

5. Is there a way to show the percentage of county respondents who liked the dominant team? I imagine there are plenty of counties where one team does not have a strong majority, let alone even much of a plurality. For example, Jets fans barely show up on the map because they are only the top team in one county. Yet, there are plenty of Jets fans.

Bill Gates: we can make progress with goals, data, and a feedback loop

Posted on by

Bill Gates argues in the Wall Street Journal that significant progress can be made around the world if organizations and residents participate in a particular process:

In the past year, I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal—in a feedback loop similar to the one Mr. Rosen describes.

This may seem basic, but it is amazing how often it is not done and how hard it is to get right. Historically, foreign aid has been measured in terms of the total amount of money invested—and during the Cold War, by whether a country stayed on our side—but not by how well it performed in actually helping people. Closer to home, despite innovation in measuring teacher performance world-wide, more than 90% of educators in the U.S. still get zero feedback on how to improve.

An innovation—whether it’s a new vaccine or an improved seed—can’t have an impact unless it reaches the people who will benefit from it. We need innovations in measurement to find new, effective ways to deliver those tools and services to the clinics, family farms and classrooms that need them.

I’ve found many examples of how measurement is making a difference over the past year—from a school in Colorado to a health post in rural Ethiopia. Our foundation is supporting these efforts. But we and others need to do more. As budgets tighten for governments and foundations world-wide, we all need to take the lesson of the steam engine to heart and adapt it to solving the world’s biggest problems.

Gates doesn’t use this term but this sounds like a practical application of the scientific method. Instead of responding to a social problem by going out and trying to “do something,” the process should be more rigorous, involve setting goals, collecting good data, interpreting the data, and then adjusting the process from the beginning. This is related to other points about this process:

1. It is one thing to be able to collect data (and this is often its own complicated process) but it is another to know what to do with it once you have it. Compared to the past, data is relatively easy to obtain today but using it well is another matter.

2. Another broad issue in this kind of feedback loop is developing the measurements and what counts as “success.” Some of this is fairly easy; when Gates praises the UN Millennium Goals, reducing occurrences of disease or boosting incomes has face validity for getting at what matters. But, measuring teacher’s performances or what makes a quality college are a little trickier to define in the first place. Gates calls this developing goals but this could be a lengthy process in itself.

It is interesting that Gates mentions the need for such loops in colleges so that students “could know where they would get the most for their tuition money.” The Gates Foundation has put money into studying public schools and just a few weeks ago released some of their findings:

After a three-year, $45 million research project, the Bill and Melinda Gates Foundation believes it has some answers.

The most reliable way to evaluate teachers is to use a three-pronged approach built on student test scores, classroom observations by multiple reviewers and teacher evaluations from students themselves, the foundation found…

The findings released Tuesday involved an analysis of about 3,000 teachers and their students in Charlotte; Dallas; Denver; Memphis; New York; Pittsburgh; and Hillsborough County, Fla., which includes Tampa. Researchers were drawn from the Educational Testing Service and several universities, including Harvard, Stanford and the University of Virginia…

Researchers videotaped 3,000 participating teachers and experts analyzed their classroom performance. They also ranked the teachers using a statistical model known as value-added modeling, which calculates how much an educator has helped students learn based on their academic performance over time. And finally, the researchers surveyed the students, who turned out to be reliable judges of their teacher’s abilities, Kane said.

All this takes quite a few resources and time. For those interested in quick action, this is not the process to follow. Hopefully, however, the resources and time pay off with better solutions.

Average new house size expected to drop to 2,150 square feet by 2015

Posted on by

A short report on McMansions links to a National Association of Home Builders survey that suggests building professionals believe the square footage of the average new American house will fall by 2015:

Respondents expect the average, new single-family detached home in 2015 to be about 2,152 square feet, 10 percent smaller than the average size of single-family homes started in the first three quarters of 2010. Overall, 63 percent of respondents expect the average size of new homes in 2015 to be somewhere between 2,000 square feet and 2,399 square feet, 22 percent expect it to be between 2,400 square feet and 2,999 square feet, while 13 percent expect it to only be 1,600 square feet to 1,999 square feet (Figure 2).

Figure 2. Average Home Size in 2015

Data from the Census Bureau indicates that the average size of single-family homes completed peaked in 2007, at 2,521 square feet, was virtually unchanged in 2008, and then declined in 2009 to 2,438 square feet. Preliminary data for 2010 shows a further decline, down to 2,377 square feet. Although part of the recent drop in average home size may indeed be temporary due to hard economic times, a number of factors lead building professionals to expect home size declines in the long-run: consumers are focused on lowering the cost of heating and cooling their homes; they no longer have sizeable equity in their current homes to finance a much larger one; diminished expectations for house price appreciation has reduced demand for extra square footage in order to achieve appreciation on a larger base; demographics, 29 percent of the US population will be 55+ in the year 2020, demanding smaller homes; and strict mortgage underwriting for the foreseeable future. Combined, these factors will weigh on the consumer to purchase homes based on need more than want.

My interpretation of this is that a majority of builders think new homes in 2015 will be slightly smaller than new homes of today. Additionally, 23% still believe new homes will be larger than 2,400 square feet. Interestingly, there is not reported evidence of whether building professionals think these smaller new homes of the future will be cheaper.

And here is where square footage will be dropped from these future houses:

To save on square footage, the living room is high on the endangered list – 52 percent of builders expect it to be merged with other spaces in the home by 2015 and 30 percent said it will vanish entirely.

“As an overall share of total floor space, 54 percent of builders said the family room is likely to increase,” said Rose Quint, NAHB’s assistant vice president for survey research. “That makes it the only area of the home likely to get bigger.”

In addition, the relative size of the entry foyer and dining room are likely to be diminished by 2015. However, opinions were fairly evenly divided on the fate of the kitchen, master bedroom and bath and mudroom, she said.

The survey methodology is also worth noting – it was sent to a lot of interested parties but the response rate was under 10%:

NAHB’s The New Home in 2015 survey was sent electronically to 3,019 builders, designers, architects, manufacturers, and marketing specialists. The sample was stratified by region of the country (to be proportional to housing starts in each of the four Census regions) and, among builders, by their number of units started.

A total of 238 responses were received, of which 30 percent came from single-family builders, 19 percent from architects, 26 percent from designers, 7 percent from manufacturers, and 18 percent from “other” building industry professionals.

At first glance, this suggests to me that the findings are quite untrustworthy.

Highlights from the “Illinois’s 33%” poverty report

Posted on by

A new report from the Social Impact Research Center, “Illinois’s 33%,”  looks at poverty in Illinois. Here are a few highlights:

1. Something I did not realize: the preamble to the Illinois Constitution mentions “eliminat[ing] poverty” (p.1).

“We, the People of the State of Illinois…in order to provide for the health, safety and welfare of the people; maintain a representative and orderly government; eliminate poverty and inequality; assure legal, social and economic justice; provide opportunity for the fullest development of the individual; ensure domestic tranquility; provide for the common defense; and secure the blessings of freedom and liberty to ourselves and our posterity—do ordain and establish this Constitution for the State of Illinois.”

2. The report is not just about poverty; it is also about people in near-poverty. The income thresholds for this are here (p.5):

This methodology of measuring people with low incomes or near poverty seems to be growing. The Census reports the median household income in Illinois is $56,576.

3. There is definitely some geographic disparity in these figures. Here are the numbers for the Chicago region which clearly shows wealthier and less wealthy counties and Chicago neighborhoods (p.7):

I did not see any calls for metropolitan approaches to poverty. In the Chicago region, it would be difficult to deal with a particular problem, say affordable housing, in just Chicago or a few of its neighborhoods without cooperation and input from others in the region.

4. The report has more figures and possible solutions in five areas that could help people move out of poverty: employment, education, housing, health & nutrition, and assets (p.3-4, 15-17).

Confessions of researchers: #overlyhonestmethods

Posted on by

Here is a collection of 17 post under the Twitter hashtag #overlyhonestmethods. My favorite: “We assume 50 Ivy League kids represent the general population, b/c ‘real people’ can be sketchy or expensive.” This doesn’t surprise me considering the number of undergraduates used in psychology studies…

I wonder how many researchers could tell similar stories about research methods. These admissions don’t necessarily invalidate any of the findings but rather hint at the very human dimension present in conducting research studies.

(Disclaimer: of course it is difficult to know how many of these research method confessions are true.)