Tag Archives: methodology

Confronting the problems with p-values

Posted on by

Nature provides an overview of concerns about how much scientists rely on p-values “which is neither as reliable nor as objective as most scientists assume”:

One result is an abundance of confusion about what the P value means4. Consider Motyl’s study about political extremists. Most scientists would look at his original P value of 0.01 and say that there was just a 1% chance of his result being a false alarm. But they would be wrong. The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place. To ignore this would be like waking up with a headache and concluding that you have a rare brain tumour — possible, but so unlikely that it requires a lot more evidence to supersede an everyday explanation such as an allergic reaction. The more implausible the hypothesis — telepathy, aliens, homeopathy — the greater the chance that an exciting finding is a false alarm, no matter what the P value is…

These are sticky concepts, but some statisticians have tried to provide general rule-of-thumb conversions (see ‘Probable cause’). According to one widely used calculation5, a P value of 0.01 corresponds to a false-alarm probability of at least 11%, depending on the underlying probability that there is a true effect; a P value of 0.05 raises that chance to at least 29%. So Motyl’s finding had a greater than one in ten chance of being a false alarm. Likewise, the probability of replicating his original result was not 99%, as most would assume, but something closer to 73% — or only 50%, if he wanted another ‘very significant’ result6, 7. In other words, his inability to replicate the result was about as surprising as if he had called heads on a coin toss and it had come up tails…

Critics also bemoan the way that P values can encourage muddled thinking. A prime example is their tendency to deflect attention from the actual size of an effect. Last year, for example, a study of more than 19,000 people showed8 that those who meet their spouses online are less likely to divorce (p < 0.002) and more likely to have high marital satisfaction (p < 0.001) than those who meet offline (see Nature http://doi.org/rcg; 2013). That might have sounded impressive, but the effects were actually tiny: meeting online nudged the divorce rate from 7.67% down to 5.96%, and barely budged happiness from 5.48 to 5.64 on a 7-point scale. To pounce on tiny P values and ignore the larger question is to fall prey to the “seductive certainty of significance”, says Geoff Cumming, an emeritus psychologist at La Trobe University in Melbourne, Australia. But significance is no indicator of practical relevance, he says: “We should be asking, ‘How much of an effect is there?’, not ‘Is there an effect?’”

Perhaps the worst fallacy is the kind of self-deception for which psychologist Uri Simonsohn of the University of Pennsylvania and his colleagues have popularized the term P-hacking; it is also known as data-dredging, snooping, fishing, significance-chasing and double-dipping. “P-hacking,” says Simonsohn, “is trying multiple things until you get the desired result” — even unconsciously. It may be the first statistical term to rate a definition in the online Urban Dictionary, where the usage examples are telling: “That finding seems to have been obtained through p-hacking, the authors dropped one of the conditions so that the overall p-value would be less than .05”, and “She is a p-hacker, she always monitors data while it is being collected.”

As the article then goes on to note, alternatives haven’t quite caught on. It seems the most basic defense is one that statisticians should adopt anyhow: always recognizing the chance that their statistics could be wrong. It also highlights the need for replicating studies with different datasets to confirm results.

At a relatively basic level, if p-levels are so problematic, how does this change the basic statistics courses so many undergraduates take?

21st century methodology problem: 4 ways to measure online readership

Posted on by

While websites can collect lots of information about readers, how exactly this should all be measured is still unclear. Here are four options:

Uniques: Unique visitors is a good metric, because it measures monthly readers, not just meaningless clicks. It’s bad because it measures people rather than meaningful engagement. For example, Facebook viral hits now account for a large share of traffic at many sites. There are one-and-done nibblers on the Web and there are loyal readers. Monthly unique visitors can’t tell you the difference.

Page Views: They’re good because they measure clicks, which is an indication of engagement that unique visitors doesn’t capture (e.g.: a blog with loyal readers will have a higher ratio of page views-to-visitors, since the same people keep coming back). They’re bad for the same reason that they can be corrupted. A 25-page slideshow of the best cities for college graduates will have up to 25X more views than a one-page article with all the same information. The PV metric says the slideshow is 25X more valuable if ads are reloaded on each page of the slideshow. But that’s ludicrous.

Time Spent/Attention Minutes: Page views and uniques tell you an important but incomplete fact: The article page loaded. It doesn’t tell you what happens after the page loads. Did the reader click away? Did he stay for 20 minutes? Did he open the browser tab and never read the story? These would be nice things to know. And measures like attention minutes can begin to tell us. But, as Salmon points out, they still don’t paint a complete picture. Watching a 5 minute video and deciding it was stupid seems less valuable than watching a one minute video that you share with friends and praise. Page views matter, and time spent matters, but reaction matters, too. This suggests two more metrics …

Shares and Mentions: “Shares” (on Facebook, Twitter, LinkedIn, or Google+) ostensibly tell you something that neither PVs, nor uniques, nor attention minutes can tell you: They tell you that visitors aren’t just visiting. They’re taking action. But what sort of action? A bad column will get passed around on Twitter for a round of mockery. An embarrassing article can go viral on Facebook. Shares and mentions can communicate the magnitude of an article’s attention, but they can’t always tell you the direction of the share vector: Did people share it because they loved it, or because they loved hating it?

Here are some potential options for sorting this all out:

1. Developing a scale or index that combines all of these factors. It could be as easy as each of these four counts for 25% or the components could be weighted differently.

2. Heavyweights in the industry – whether particular companies or advertisers or analytical leaders – make a decision about which of these is most important. For example, comments after this story note the problems with Nielsen television ratings over the decades but Nielsen had a stranglehold on this area.

3. Researchers outside the industry could “objectively” develop a measure. This may be unlikely as outside actors have less financial incentive but perhaps someone sees an opportunity here.

In the meantime, there is plenty of information on online readership to look at, websites and companies can claim various things with different metrics, and websites and advertisers will continue to have a strong financial interest in all of this.

Argument: businesses should use scientific method in studying big data

Posted on by

Sociologist Duncan Watts explains how businesses should go about analyzing big data:

A scientific mind-set takes as its inspiration the scientific method, which at its core is a recipe for learning about the world in a systematic, replicable way: start with some general question based on your experience; form a hypothesis that would resolve the puzzle and that also generates a testable prediction; gather data to test your prediction; and finally, evaluate your hypothesis relative to competing hypotheses.

The scientific method is largely responsible for the astonishing increase in our understanding of the natural world over the past few centuries. Yet it has been slow to enter the worlds of politics, business, policy, and marketing, where our prodigious intuition for human behavior can always generate explanations for why people do what they do or how to make them do something different. Because these explanations are so plausible, our natural tendency is to want to act on them without further ado. But if we have learned one thing from science, it is that the most plausible explanation is not necessarily correct. Adopting a scientific approach to decision making requires us to test our hypotheses with data.

While data is essential for scientific decision making, theory, intuition, and imagination remain important as well—to generate hypotheses in the first place, to devise creative tests of the hypotheses that we have, and to interpret the data that we collect. Data and theory, in other words, are the yin and yang of the scientific method—theory frames the right questions, while data answers the questions that have been asked. Emphasizing either at the expense of the other can lead to serious mistakes…

Even here, though, the scientific method is instructive, not for eliciting answers but rather for highlighting the limits of what can be known. We can’t help asking why Apple became so successful, or what caused the last financial crisis, or why “Gangnam Style” was the most viral video of all time. Nor can we stop ourselves from coming up with plausible answers. But in cases where we cannot test our hypothesis many times, the scientific method teaches us not to infer too much from any one outcome. Sometimes the only true answer is that we just do not know.

To summarize: the scientific method provides ways to ask questions and receive data regarding answering these questions. It is not perfect – it doesn’t always produce the answer or the answers people are looking for, it may only be as good as the questions asked, it requires a rigorous methodology – but it can help push forward the development of knowledge.

While there are businesses and policymakers using such approaches, it strikes me that such an argument for the scientific method is especially needed in the midst of big data and gobs of information. In today’s world, getting information is not a problem. Individuals and companies can quickly find or measure lots of data. However, it still requires work, interpretation, and proper methodology to interpret that data.

Cell phones are not an impediment to public social interaction

Posted on by

Recent research from a sociologist analyzing video footage of public spaces shows cell phones don’t limit public interaction:

Between 2008 and 2010, his team accrued enough footage to begin a comparison with the P.P.S. films — together the two collections totaled more than 38 hours. “Films were sampled at 15-second intervals for a total of 9,173 observation periods,” he writes in his article, which reads like a study in scholarly masochism. Hampton and a team of 11 graduate and undergraduate students from Penn spent a total of 2,000 hours looking at the films, coding the individuals they observed for four characteristics: sex, group size, “loitering” and phone use…

First off, mobile-phone use, which Hampton defined to include texting and using apps, was much lower than he expected. On the steps of the Met, only 3 percent of adults captured in all the samples were on their phones. It was highest at the northwest corner of Bryant Park, where the figure was 10 percent. More important, according to Hampton, was the fact that mobile-phone users tended to be alone, not in groups. People on the phone were not ignoring lunch partners or interrupting strolls with their lovers; rather, phone use seemed to be a way to pass the time while waiting to meet up with someone, or unwinding during a solo lunch break. Of course, there’s still the psychic toll, which we all know, of feeling tethered to your phone — even while relaxing at the park. But that’s a personal cost. From what Hampton could tell, the phones weren’t nearly as hard on our relationships as many suspect…

According to Hampton, our tendency to interact with others in public has, if anything, improved since the ‘70s. The P.P.S. films showed that in 1979 about 32 percent of those visited the steps of the Met were alone; in 2010, only 24 percent were alone in the same spot. When I mentioned these results to Sherry Turkle, she said that Hampton could be right about these specific public spaces, but that technology may still have corrosive effects in the home: what it does to families at the dinner table, or in the den. Rich Ling, a mobile-phone researcher in Denmark, also noted the limitations of Hampton’s sample. “He was capturing the middle of the business day,” said Ling, who generally admires Hampton’s work. For businesspeople, “there might be a quick check, do I have an email or a text message, then get on with life.” Fourteen-year-olds might be an entirely different story…

In fact, this was Hampton’s most surprising finding: Today there are just a lot more women in public, proportional to men. It’s not just on Chestnut Street in Philadelphia. On the steps of the Met, the proportion of women increased by 33 percent, and in Bryant Park by 18 percent. The only place women decreased proportionally was in Boston’s Downtown Crossing — a major shopping area. “The decline of women within this setting could be interpreted as a shift in gender roles,” Hampton writes. Men seem to be “taking on an activity that was traditionally regarded as feminine.”

Perhaps there is such a reaction to people using phones in public because (1) they are a new technology and people still aren’t used to them – smartphones are only less than a decade old and/or (2) phones are less noticeable or personally intrusive in wide open settings like the steps of the Met but very noticeable in more confined settings where conversations can be heard.

I think there is also a lot sociologists could build on here with Hampton’s methodology. Video may seem archaic when you can utilize big data but it can still provide unique insights into social behavior. While the coding of the video was rather simple (they looked at four categories: “sex, group size, “loitering” and phone use”), it took a lot of time to go through the video and compare it to Whyte’s earlier film. This comparative element is also quite useful: we can then compare patterns over time. All together, think how much video footage is collected in public these days and how it might lend itself to research…

Chicago innovation #14: Consumer preference research

Posted on by

A cousin of social science research, consumer preference research, got its start in Chicago:

It was 1928. Benton was working at Chicago’s Lord & Thomas advertising agency when owner Albert Lasker told him to land Colgate-Palmolive by impressing the outsized toiletry powerhouse with market research. Benton worked night and day for two months to record housewives’ preferences for the products of each company.

The firm used the pioneering survey in its initial Colgate-Palmolive campaign and landed the account before the survey was completed.

This drew criticism from an early sociologist:

Sociologist Gordon Hancock hated the idea. It was tantamount to cheating.

In a statement that must have brought grins to the faces of that up-and-coming generation of ad men, Hancock decried in 1926: “Excessive scientific advertising takes undue advantage of the public.”

This was, of course, the point.

This tension between marketing and sociology still exists today. The two areas use similar methods of collecting data such as surveys, focus groups, interviews, and ethnographies or participant observation. However, they have very different purposes: marketing is intended to sell products while sociologists are trying to uncover how social life works. The tension reminds me of Morgan Spurlock’s documentary The Greatest Movie Ever Sold that questions the marketing complex.

This comes in at #14 on a list of the top 20 innovations from Chicago; I highlighted the #5 innovation, balloon frame housing, in an earlier post.

 

Ethnography = reporting = paying attention, right?

Posted on by

A look at psychologist Sherry Turkle’s latest study and promotion of normal conversation includes an interesting description of ethnography:

Turkle is at work on a new book, aspirationally titled Reclaiming Conversation, which will be a continuation of her thinking in Alone Together. In it, she will out herself again, this time as “a partisan of conversation.” Her research for the book has involved hours upon hours of talking with people about conversation as well as eavesdropping on conversations: the kind of low-grade spying that in academia is known as “ethnography,” that in journalism is known as “reporting,” and that everywhere else is known as “paying attention.”

Considering Turkle’s years of studying human interaction with machines and technology (including the fascinating book Alone Together), I suspect she would not describe ethnography this way. But, it is not hard to find similar descriptions of ethnography. Isn’t it just observation and paying attention? Not quite. Journalists tend to equate ethnography with good reporting but this is not the case. Here are some key differences:

1. Ethnography does not involve intentionally interviewing key informants in a story. It involves much more discussion, observation, and time.

2. Ethnography is sometimes known as participant observation. Ethnographers don’t just interview; they often participate with the people or groups they are studying so they can get an insider view (key: while still retaining their outside, analytical perspective).

3. There is a rigorous process to ethnography that typically involves months of participant observation, copious note taking (both on the spot as well at the end of each day – I’ve seen recommendations for 2-3 hours of note-taking for each 1 hour in the field), returning from the field and coding and analyzing the notes, writing a study that interacts with and adds to existing theories.

4. This is not just “low-grade spying.” Ethnography is often an intense, draining experience that involves a lot of human interaction.

In other words, ethnography is not just about showing up and eavesdropping. Some people may be pretty good at this but this does not automatically make a good study. This, in my mind, is often the difference between academic and journalistic approaches to topics and social issues: the methodology employed by journalists tends to be scattered and there is little discussion of the trade-offs involved in their methodological choices.

Gallup to start surveying college graduates to find if their college degree led to “a great life”

Posted on by

Gallup is working on a new initiative to measure a wider range of life outcomes for college graduates:

As the old saying goes, money can’t buy happiness. And yet, in measuring alumni success and satisfaction, colleges – often prodded by those seeking to hold them accountable – typically look at two things: whether their former students are gainfully employed, and whether they’re making a decent salary.

A new project announced today, led by Gallup and debuting at Purdue University, aims to change that. Focusing on a set of factors that are shown to correlate with “a great life,” the survey of 30,000 graduates annually will provide data on how alumni of groups of colleges (public or private institutions in certain states, for instance, or athletic conferences) are faring and how they compare to national averages. The final product will be a benchmark for student success against which any campus can measure its own graduates, if it works with Gallup individually…

The survey’s line of questioning goes beyond job placement and salary, also inquiring about work place and community engagement, personal relationships, physical fitness, sense of purpose and happiness, and economic management and stress…

“No one is going to suggest that colleges and universities are responsible for 100 percent of your great job and great life,” Busteed admitted, “but obviously, if you go to college and you get a degree, the odds are you increase the probability of having a good outcome.”

Given the arguments about the cost of college, I’m not surprised efforts like this are quickly moving forward. And, as the article notes, there are lots of methodological questions in play: what exactly is “a great life”? How many years after college should people be asked these questions? How can the effects of college be separated out from other life experiences (though people’s perceptions about whether college mattered is important as well)?

At the same time, I’m not opposed to trying to get at these life outcomes after college. Colleges often make the argument they improve the lives of their graduates from earning more to training for careers to giving students room to start living to critical thinking to a broader understanding of the world. Is some of the concern about measuring these things that colleges might not be able to live up to lofty claims? For example, given the findings of Academically Adrift from a few years ago, not all college students are benefiting. Once findings start trickling out, it will then be imperative to see what gets counted as “success” for colleges.

Using randomized controlled trials to test methods for addressing global poverty

Posted on by

Here is a relatively new way to test options for addressing poverty: use randomized controlled trials.

What Kremer was suggesting is a scientific technique that has long been considered the gold standard in medical research: the randomized controlled trial. At the time, though, such trials were used almost exclusively in medicine—and were conducted by large, well-funded institutions with the necessary infrastructure and staff to manage such an operation. A randomized controlled trial was certainly not the domain of a recent PhD, partnering with a tiny NGO, out in the chaos of the developing world…

The study wound up taking four years, but eventually Kremer had a result: The free textbooks didn’t work. Standardized tests given to all students in the study showed no evidence of improvement on average. The disappointing conclusion launched ICS and Kremer on a quest to discover why the giveaway wasn’t helping students learn, and what programs might be a better investment.

As Kremer was realizing, the campaign for free textbooks was just one of countless development initiatives that spend money in a near-total absence of real-world data. Over the past 50 years, developed countries have spent something like $6.5 trillion on assistance to the developing world, most of those outlays guided by little more than macroeconomic theories, anecdotal evidence, and good intentions. But if it were possible to measure the effects of initiatives, governments and nonprofits could determine which programs actually made the biggest difference. Kremer began collaborating with other economists and NGOs in Kenya and India to test more strategies for bolstering health and education…

In the decade since their founding, J-PAL and IPA have helped 150 researchers conduct more than 425 randomized controlled trials in 55 countries, testing hypotheses on subjects ranging from education to agriculture, microfinance to malaria prevention, with new uses cropping up every year (see “Randomize Everything,” below). Economists trained on randomized controlled trials now work in the faculties of top programs, and some universities have set up their own centers to support their growing rosters of experiments in the social sciences.

If this is indeed a relatively new approach, what took so long? Perhaps the trick was thinking that experiments, typically associated with very controlled laboratory or medical settings, could be preformed in less controlled settings. As the article notes, they are not easy to set up. One of the biggest issues might be randomizing enough people into the different groups to wash out all of the possible factors that might influence the results.

This also seems related to the uptick in interest in natural experiments where social scientists take advantage of “natural” occurrences, perhaps a policy change or a natural disaster, to compare results across groups. Again, laboratories offer controlled settings but there are only so many things that can be addressed and the number of people in the studies tend to be pretty small.

Sociologist on how studying an extremist group led to a loss of objectivity

Posted on by

A retired sociologist who studied the Aryan Nation discusses how his research led to a loss of objectivity and a change of research topics:

Aho began his research in the mid-1980s with a focus on the most notorious group in Idaho, the Aryan Nation Church near Coeur d’Alene and Hayden Lake. Annual conferences were held there with people from all around the world to fight what they called the “race war.” The group, originally formed in California, was forced to relocate to Idaho due to pressure from authorities. Aho was able to interview members of the group face to face, conduct phone interviews and correspond with prison inmates who were part of the organization.

“These individuals were genuinely good, congenial folks,” said Aho. “They were very independent, married, church-going people with deep beliefs. It was only when they gathered in groups and reaffirmed each other’s prejudices that things became dangerous…

In his research, Aho tried to place himself in his subjects’ shoes. He expressed how it is important to see yourself in the other person to find mutual ground and truths that can only be obtained by using this research methodology. However, after nearly a decade of research, he felt that he was losing objectivity and only adding to the problem.

“I spent years trying to understand the people who are attracted to violence, but I began to feel like my fascination with violence made me partly responsible for it,” Aho said. “I think I lost my sociological objectivity, and thought it was time to end my efforts of trying to understanding it, and move on to other scholarly activities.”

Some candor about researching a difficult topic. Given statements by some recently that we should not “commit sociology” and refrain from looking for explanations for violence, we could just ignore such groups. But, looking for explanations is not the same as excusing or condoning behavior and may help limit violence in the future. At the same time, spending lots of time with people, whether they are good or bad, can lead to relationships and a humanizing the research subject. This may provide better data for a while as well as dignity for the research subject but can lead to the “going native” issue that anthropologists sometimes discuss. A sociologist wants to be able to remain an observer and analyst, even as they try to put themselves in the shoes of others.

It would be interesting to hear the opinions of sociologists regarding studying clearly unpopular groups like white supremacists/terrorists. Sociologists are often interested in studying disadvantaged or voiceless groups but what about groups with which they profoundly disagree?

What exactly is the ethnographic line between entertaining stories and academic content?

Posted on by

One sociologist expresses his dislike of Sudhir Venkatesh’s latest book Floating City and raises an interesting question about the line between entertaining stories and sociological analysis:

 

Early in my career I had a book reviewed by a wise sage of Fleet Street who mocked my sociological language and academic conceits. Some years later, a literary agent explained that if I were to abandon academic conventions but retain the stories of “low life”, untold riches would surely follow. In exchange for academic orthodoxy I should situate myself firmly in the narrative, charting my no doubt dangerous but hilarious adventures through the life-world of the lower orders. Although I rejected this suggestion, I always wondered what an ethnography eviscerated of academic content would look like.

Floating City has answered this question and much more. A self-styled “maverick sociologist” and experienced urban ethnographer who holds a named chair at an Ivy League university, Sudhir Venkatesh drains his study of illegal entrepreneurship in New York of most of the academic conventions that will come naturally to someone with almost a quarter of a century of experience in the dangerous enclaves of US higher education. On arriving in New York from Chicago, Venkatesh finds life in a “World City” unfamiliar (but isn’t Chicago a World City?), with old standbys such as neighbourhood and community apparently made redundant by globalisation…

The populist format of Floating City is insistent that the Homeric author is a pioneer, the first in the field, an ingénue intent on emphasising that the unique demands of researching the global city require a rejection of academic tradition. But his outsider stance and maverick posturing are an irritant; they get in the way of his street-savvy case studies and vignettes of urban life. For at the heart of this book is a conventional ethnography that challenges the author’s ingénue facade by using the full arsenal of ethnographic orthodoxy, and along the way highlighting the enduring importance of community and neighbourhood.

The best of this book is precisely the result of the “traditional sociology” that Venkatesh derides. I will not forget in a hurry the poignancy of his descriptions of Manjun and his family, who run a shop selling pornographic DVDs, or his fine-grained interaction with a group of aspirational prostitute women struggling on streets paved with something less than gold. His interaction with coke-addled socialites I found irritating and somewhat pointless, and his aim to make connections between the so-called upper- and underworlds fizzles out. For while the city may be more fluid, the rich and the poor remain separated, and the dots are not joined.

This gets at some basic issues sociologists face:

1. What audience should they aim for when writing a book? If they write for the public, they may be derided by fellow academics for not having enough academic content. If they write a more academic book, few in the public will touch it. It is rare to find such sociological works that can easily transcend these audience boundaries. Just the other day, I was explaining to someone how the book Bowling Alone was one of these rare texts that contained an academic approach to a problem that caught people’s attention.

2. How exactly should ethnographic research be conveyed? Yes, there are stories to relate but it also needs to contain some sociological analysis. Different ethnographers rely on different mixes of narrative and analysis but the sociological element still has to come through to some degree.

3. If the sociologist is studying “low life,” how can these lives be explored without being exploitative or cheapening? Salaciousness might sell but it does not tend to grant dignity and respect to those being described. Sociologists also often develop personal relationships with those they are studying and these complicates the retelling.

These are common concerns of ethnographers and there are not easy answers. Amongst themselves, sociologists disagree how to do these things better.