Tag Archives: methodology

Disagreement about unemployment figures between government and Gallup

Posted on by

Gallup suggests that the unemployment figures to be released by the federal government at the end of this week are underestimates. While the government figures are expected to be around 9.6-9.8%, Gallup says the unemployment is really closer to 10.1%.

The main issue seems to be that Gallup is measuring through the end of September while the government figures are based on data that ended in mid-September. And Gallup found that unemployment increased quite a bit in the last few weeks of September.

More on the study of happiness: the role of priorities

Posted on by

Measuring happiness is a small industry among researchers. A new study suggests another important factor: the priorities that people set for their life affects long-term happiness.

Most of us have thought, ‘If only I could win the lottery, then I’d be happy forever.’ But according to one of the first studies to look at long-term happiness, major life events, like a sudden cash windfall, are not what make us happy, rather, it’s the priorities we set in life.

“The main thing that’s surprising about these results is that it challenges this whole field,” said lead author Melbourne University sociologist Bruce Headey. “This study goes against the prevailing wisdom that happiness is fixed.” The study was published Monday in the Proceedings of the National Academy of Sciences.

Previous studies suggest that happiness is predetermined by genetics and early upbringing, and that we eventually revert back to the same level of happiness regardless of changes in our lives. Looking at data from about 60,000 Germans for up to 25 years, however, Headey found that the more people decided to prioritize goals such as good relationships and good health, the happier they were, regardless of major life events.

While there are some critiques of this study (for example, it measures long-term happiness rather than daily satisfaction), it suggests again that the topic of happiness is a complex one to research with many possible factors influencing outcomes.

So should people set easier-to-reach priorities to be happy? What happens to people who set good priorities but aren’t able to reach them?

This also seems to be an interesting dataset with 60,000 people being tracked over the last two and a half decades.

(I’m also curious about the lead author saying that the study challenges “the prevailing wisdom that happiness is fixed.”)

Measuring the economy by looking at midnight Walmart shoppers

Posted on by

There are all sorts of figures and statistics that are used to measure how the economy is doing. This NPR story introduces a new metric: looking at midnight sales at Walmart on the first day of the month.

Wal-Mart noticed that sales were spiking on the first of every month. In a recent conference call with investment analysts, Wal-Mart executive Bill Simon said these midnight shoppers provide a snapshot of the American economy today.

“And if you really think about it,” Simon said, “the only reason somebody gets out and buys baby formula is they need it and they’ve been waiting for it. Otherwise, we’re open 24 hours, come at 5 a.m., come at 7 a.m., come at 10 a.m. But if you’re there at midnight you’re there for a reason.”

And so Wal-Mart has changed its stocking pattern. It brings out larger packs of items in the beginning of the month, and smaller sizes toward the end. It makes sure shelves have plenty of diapers and formula.

This is a creative data source – but we would need more information before making broad conclusions about the American economy. Do other stores experience similar spikes? How big of a spike is this? What Walmart locations have seen the biggest jumps?

It strikes me that Walmart probably possesses a treasure trove of data that would be very interesting to look at.

Making the case for reputational rankings

Posted on by

A statistician argues that the National Research Council’s study of doctoral programs released earlier this week should have included reputational rankings:

Mr. Stigler says that it was a mistake for the NRC to so thoroughly abandon the reputational measures it used in its previous doctoral studies, in 1982 and 1995. Reputational surveys are widely criticized, he says, but they do provide a check on certain kinds of qualitative measures. When the new NRC counts faculty publication rates, it does not offer any information about whether scholars in the field believe those publications are any good. (That’s especially true in humanities fields, where the NRC report does not include citation counts.)

“Everybody involved in this was trying hard, and with good intentions and high integrity,” Mr. Stigler says. “But once they decided to rule out reputation, they cut off what I consider to be the most useful measure from all past surveys.”

In an e-mail message to The Chronicle this week, Mr. Ostriker declined to reply to Mr. Stigler’s specific statistical criticisms. But he pointed out that the National Academies explicitly instructed his committee not to use reputational measures.

I was curious about this when I looked at the list of sociology doctoral programs. Perhaps several of the schools that were lower than I expected, such as the University of California – Berkeley, were lower because of this.

Stigler defends reputational measures but I’ve seen others argue that they prohibit “true” rankings within fields because certain schools retain a reputation even without the necessary output (research, good grad students, etc.). This particular discussion is part of a larger one where it will need to be decided whether reputational rankings should be used or not.

Another reason for IRBs and ethical guidelines for research

Posted on by

There is a body of known research from around the mid 20th century that led to the formation of ethical guidelines for research and the establishment of Institutional Review Boards (IRBs). Here is another study in the news that shows why these guidelines are necessary:

The United States apologized on Friday for an experiment conducted in the 1940s in which U.S. government researchers deliberately infected Guatemalan prison inmates, women and mental patients with syphilis.

In the experiment, aimed at testing the then-new drug penicillin, inmates were infected by prostitutes and later treated with the antibiotic.

“The sexually transmitted disease inoculation study conducted from 1946-1948 in Guatemala was clearly unethical,” Secretary of State Hillary Clinton and Health and Human Services Secretary Kathleen Sebelius said in a statement.

A researcher discovered this case while doing research that followed up on the Tuskegee experiments of the 1960s.

Good news about Chicago traffic and congestion – but due to new criteria

Posted on by

A new report from a group named CEOs for Cities claims that Chicagoans spend the least amount of time in rush-hour traffic compared to other major cities:

The report’s ranking of mobility in 51 cities found that Chicago-area residents spend the least time in rush-hour travel. In Chicago and some of the other best-performing cities — including New Orleans, New York, Portland, Ore., and Sacramento, Calif. — commuters typically spend 40 fewer hours a year in peak-hour travel than the average American, the report said.

In metro areas with the worst urban sprawl — including Nashville, Detroit, Indianapolis and Raleigh, N.C. — residents spend as much as 240 hours per year in rush-period travel on average because commuting distances are much longer, said the report, which was produced with the support of the Rockefeller Foundation.

This seems to be contrary to other studies I’ve seen that suggest Chicago is quite congested. One reason this study might have different results is a new criteria in the methodology.

The report’s author criticized other mobility studies that focus on the amount of traffic congestion in a region without factoring in travel distance.

The Urban Mobility Report, issued every two years by the Texas Transportation Institute, is regarded by many experts as the authoritative voice on traffic congestion issues. The report consistently ranks the Chicago region as the second or third most-congested area of the nation. It does not account for travel distance.

I am left wondering whether travel distance an important factor to include…

The Infrastructurist comments on the disparities in the two sets of rankings.

Vested interests in the telling of history

Posted on by

In a recent class, we were discussing an article that talked about a number of issues of doing research after an event has happened. One of the main concerns was the vested interests of the respondents. Particularly, if the respondent were socially near to the event or phenomenon, their retelling is filtered through their own personal interests.

A classic example of this came across the news wires today: some new information that sheds light on why the Titantic hit an iceberg and then sank so quickly. A brief part of the story:

The Titanic hit an iceberg in 1912 because of a basic steering error, and only sank as fast as it did because an official persuaded the captain to continue sailing, an author said in an interview published on Wednesday.

Louise Patten, a writer and granddaughter of Titanic second officer Charles Lightoller, said the truth about what happened nearly 100 years ago had been hidden for fear of tarnishing the reputation of her grandfather, who later became a war hero.

Lightoller, the most senior officer to have survived the disaster, covered up the error in two inquiries on both sides of the Atlantic because he was worried it would bankrupt the ill-fated liner’s owners and put his colleagues out of a job.

I’ll be curious to see how quickly this new information can (or cannot) be corroborated.

The trolley problem, race, and making decisions

Posted on by

The trolley problem  is a classic vignette used in research studies and it asks under what conditions is it permissible to sacrifice one life for the lives of others (see an explanation here). Psychologist David Pizarro tweaked the trolley problem to include racial dimensions by using characters named Chip and Tyrone. Pizarro found that people’s opinions about race influenced which character they were more willing to sacrifice:

What did this say about people’s morals? Not that they don’t have any. It suggests that they had more than one set of morals, one more consequentialist than another, and choose to fit the situation…

Or as Pizarro told me on the phone, “The idea is not that people are or are not utilitarian; it’s that they will cite being utilitarian when it behooves them. People are aren’t using these principles and then applying them. They arrive at a judgment and seek a principle.”

So we’ll tell a child on one day, as Pizarro’s parents told him, that ends should never justify means, then explain the next day that while it was horrible to bomb Hiroshima, it was morally acceptable because it shortened the war. We act — and then cite whichever moral system fits best, the relative or the absolute.

Some interesting findings from a different take on a classic research tool. This is always an interesting question to ask regarding many social issues: when does the end justify the means and when does it not?

A sizable but smaller gap in happiness between blacks and whites

Posted on by

The Freakonomics blog reports on a new study showing that there is a narrowing gap in happiness between black and whites. The reason for this may be the lessening of “day-to-day racism” – but there needs to be more research so that scholars can figure out “How best to get a handle on the evolution of day-to-day racism.”

Ranking universities around the world

Posted on by

Ranking American colleges and universities has become a lucrative industry with a number of publications developing rankings. This quest has been extended to ranking universities around the world but there is tension between two of the important rankings and their methodologies:

Times Higher Education produced rankings for the first time this year without the collaboration of Quacquarelli Symonds Limited. Along with the Shanghai Jiao Tong University rankings, the World University Rankings that Times Higher Education and QS published together from 2004 until last year have become the most closely watched and influential university rankings in the world.

Quacquarelli Symonds has continued to produce those rankings, now called the QS World University Rankings, and is partnering with U.S. News and World Report for their publication in the United States.

The relationship between the former collaborators has deteriorated into barely veiled animosity.

While there may be some issues between those who produce the two sets of rankings, the Times Higher Education claims it has an improved methodology which emphasizes different features of a school:

Foremost among the criticisms of the previous compilation was that it relied too heavily on a reputational survey of academics, based on fewer than 4,000 responses in 2009. THE’s new methodology is based on 13 indicators in five broad performance categories—teaching (weighted 30 percent); research influence as measured in citations (32.5 percent); research, based on volume, income, and reputation (30 percent); internationalization, based on student and staff ratios (5 percent); and knowledge transfer and innovation based on industry income (2.5 percent).

Times Higher Education said that the new system was the only global ranking to devote a section to teaching. The new methodology is much more evidence-based and relies far less on subjective criteria than the old tables, said Mr. Baty. But whereas teaching was previously measured based solely on student-staff ratio, the new rankings incorporate a reputational survey.

Which rankings system is “better” will be decided by a variety of actors. Of course, the rankings will likely differ because of the different weights placed on different university traits. The newest rankings from Times Higher Education are favorable toward American schools and UC-Berkeley made quite a jump up.

Overall, these rankings can be very powerful, not just for prestige purposes but also for helping to determining the global flow of students and money. Additionally, the companies producing and publishing the rankings have a financial stake in how the rankings are perceived both by schools and the public.