Tag Archives: methodology

Argument: businesses should use scientific method in studying big data

Posted on by

Sociologist Duncan Watts explains how businesses should go about analyzing big data:

A scientific mind-set takes as its inspiration the scientific method, which at its core is a recipe for learning about the world in a systematic, replicable way: start with some general question based on your experience; form a hypothesis that would resolve the puzzle and that also generates a testable prediction; gather data to test your prediction; and finally, evaluate your hypothesis relative to competing hypotheses.

The scientific method is largely responsible for the astonishing increase in our understanding of the natural world over the past few centuries. Yet it has been slow to enter the worlds of politics, business, policy, and marketing, where our prodigious intuition for human behavior can always generate explanations for why people do what they do or how to make them do something different. Because these explanations are so plausible, our natural tendency is to want to act on them without further ado. But if we have learned one thing from science, it is that the most plausible explanation is not necessarily correct. Adopting a scientific approach to decision making requires us to test our hypotheses with data.

While data is essential for scientific decision making, theory, intuition, and imagination remain important as well—to generate hypotheses in the first place, to devise creative tests of the hypotheses that we have, and to interpret the data that we collect. Data and theory, in other words, are the yin and yang of the scientific method—theory frames the right questions, while data answers the questions that have been asked. Emphasizing either at the expense of the other can lead to serious mistakes…

Even here, though, the scientific method is instructive, not for eliciting answers but rather for highlighting the limits of what can be known. We can’t help asking why Apple became so successful, or what caused the last financial crisis, or why “Gangnam Style” was the most viral video of all time. Nor can we stop ourselves from coming up with plausible answers. But in cases where we cannot test our hypothesis many times, the scientific method teaches us not to infer too much from any one outcome. Sometimes the only true answer is that we just do not know.

To summarize: the scientific method provides ways to ask questions and receive data regarding answering these questions. It is not perfect – it doesn’t always produce the answer or the answers people are looking for, it may only be as good as the questions asked, it requires a rigorous methodology – but it can help push forward the development of knowledge.

While there are businesses and policymakers using such approaches, it strikes me that such an argument for the scientific method is especially needed in the midst of big data and gobs of information. In today’s world, getting information is not a problem. Individuals and companies can quickly find or measure lots of data. However, it still requires work, interpretation, and proper methodology to interpret that data.

Cell phones are not an impediment to public social interaction

Posted on by

Recent research from a sociologist analyzing video footage of public spaces shows cell phones don’t limit public interaction:

Between 2008 and 2010, his team accrued enough footage to begin a comparison with the P.P.S. films — together the two collections totaled more than 38 hours. “Films were sampled at 15-second intervals for a total of 9,173 observation periods,” he writes in his article, which reads like a study in scholarly masochism. Hampton and a team of 11 graduate and undergraduate students from Penn spent a total of 2,000 hours looking at the films, coding the individuals they observed for four characteristics: sex, group size, “loitering” and phone use…

First off, mobile-phone use, which Hampton defined to include texting and using apps, was much lower than he expected. On the steps of the Met, only 3 percent of adults captured in all the samples were on their phones. It was highest at the northwest corner of Bryant Park, where the figure was 10 percent. More important, according to Hampton, was the fact that mobile-phone users tended to be alone, not in groups. People on the phone were not ignoring lunch partners or interrupting strolls with their lovers; rather, phone use seemed to be a way to pass the time while waiting to meet up with someone, or unwinding during a solo lunch break. Of course, there’s still the psychic toll, which we all know, of feeling tethered to your phone — even while relaxing at the park. But that’s a personal cost. From what Hampton could tell, the phones weren’t nearly as hard on our relationships as many suspect…

According to Hampton, our tendency to interact with others in public has, if anything, improved since the ‘70s. The P.P.S. films showed that in 1979 about 32 percent of those visited the steps of the Met were alone; in 2010, only 24 percent were alone in the same spot. When I mentioned these results to Sherry Turkle, she said that Hampton could be right about these specific public spaces, but that technology may still have corrosive effects in the home: what it does to families at the dinner table, or in the den. Rich Ling, a mobile-phone researcher in Denmark, also noted the limitations of Hampton’s sample. “He was capturing the middle of the business day,” said Ling, who generally admires Hampton’s work. For businesspeople, “there might be a quick check, do I have an email or a text message, then get on with life.” Fourteen-year-olds might be an entirely different story…

In fact, this was Hampton’s most surprising finding: Today there are just a lot more women in public, proportional to men. It’s not just on Chestnut Street in Philadelphia. On the steps of the Met, the proportion of women increased by 33 percent, and in Bryant Park by 18 percent. The only place women decreased proportionally was in Boston’s Downtown Crossing — a major shopping area. “The decline of women within this setting could be interpreted as a shift in gender roles,” Hampton writes. Men seem to be “taking on an activity that was traditionally regarded as feminine.”

Perhaps there is such a reaction to people using phones in public because (1) they are a new technology and people still aren’t used to them – smartphones are only less than a decade old and/or (2) phones are less noticeable or personally intrusive in wide open settings like the steps of the Met but very noticeable in more confined settings where conversations can be heard.

I think there is also a lot sociologists could build on here with Hampton’s methodology. Video may seem archaic when you can utilize big data but it can still provide unique insights into social behavior. While the coding of the video was rather simple (they looked at four categories: “sex, group size, “loitering” and phone use”), it took a lot of time to go through the video and compare it to Whyte’s earlier film. This comparative element is also quite useful: we can then compare patterns over time. All together, think how much video footage is collected in public these days and how it might lend itself to research…

Chicago innovation #14: Consumer preference research

Posted on by

A cousin of social science research, consumer preference research, got its start in Chicago:

It was 1928. Benton was working at Chicago’s Lord & Thomas advertising agency when owner Albert Lasker told him to land Colgate-Palmolive by impressing the outsized toiletry powerhouse with market research. Benton worked night and day for two months to record housewives’ preferences for the products of each company.

The firm used the pioneering survey in its initial Colgate-Palmolive campaign and landed the account before the survey was completed.

This drew criticism from an early sociologist:

Sociologist Gordon Hancock hated the idea. It was tantamount to cheating.

In a statement that must have brought grins to the faces of that up-and-coming generation of ad men, Hancock decried in 1926: “Excessive scientific advertising takes undue advantage of the public.”

This was, of course, the point.

This tension between marketing and sociology still exists today. The two areas use similar methods of collecting data such as surveys, focus groups, interviews, and ethnographies or participant observation. However, they have very different purposes: marketing is intended to sell products while sociologists are trying to uncover how social life works. The tension reminds me of Morgan Spurlock’s documentary The Greatest Movie Ever Sold that questions the marketing complex.

This comes in at #14 on a list of the top 20 innovations from Chicago; I highlighted the #5 innovation, balloon frame housing, in an earlier post.

 

Ethnography = reporting = paying attention, right?

Posted on by

A look at psychologist Sherry Turkle’s latest study and promotion of normal conversation includes an interesting description of ethnography:

Turkle is at work on a new book, aspirationally titled Reclaiming Conversation, which will be a continuation of her thinking in Alone Together. In it, she will out herself again, this time as “a partisan of conversation.” Her research for the book has involved hours upon hours of talking with people about conversation as well as eavesdropping on conversations: the kind of low-grade spying that in academia is known as “ethnography,” that in journalism is known as “reporting,” and that everywhere else is known as “paying attention.”

Considering Turkle’s years of studying human interaction with machines and technology (including the fascinating book Alone Together), I suspect she would not describe ethnography this way. But, it is not hard to find similar descriptions of ethnography. Isn’t it just observation and paying attention? Not quite. Journalists tend to equate ethnography with good reporting but this is not the case. Here are some key differences:

1. Ethnography does not involve intentionally interviewing key informants in a story. It involves much more discussion, observation, and time.

2. Ethnography is sometimes known as participant observation. Ethnographers don’t just interview; they often participate with the people or groups they are studying so they can get an insider view (key: while still retaining their outside, analytical perspective).

3. There is a rigorous process to ethnography that typically involves months of participant observation, copious note taking (both on the spot as well at the end of each day – I’ve seen recommendations for 2-3 hours of note-taking for each 1 hour in the field), returning from the field and coding and analyzing the notes, writing a study that interacts with and adds to existing theories.

4. This is not just “low-grade spying.” Ethnography is often an intense, draining experience that involves a lot of human interaction.

In other words, ethnography is not just about showing up and eavesdropping. Some people may be pretty good at this but this does not automatically make a good study. This, in my mind, is often the difference between academic and journalistic approaches to topics and social issues: the methodology employed by journalists tends to be scattered and there is little discussion of the trade-offs involved in their methodological choices.

Gallup to start surveying college graduates to find if their college degree led to “a great life”

Posted on by

Gallup is working on a new initiative to measure a wider range of life outcomes for college graduates:

As the old saying goes, money can’t buy happiness. And yet, in measuring alumni success and satisfaction, colleges – often prodded by those seeking to hold them accountable – typically look at two things: whether their former students are gainfully employed, and whether they’re making a decent salary.

A new project announced today, led by Gallup and debuting at Purdue University, aims to change that. Focusing on a set of factors that are shown to correlate with “a great life,” the survey of 30,000 graduates annually will provide data on how alumni of groups of colleges (public or private institutions in certain states, for instance, or athletic conferences) are faring and how they compare to national averages. The final product will be a benchmark for student success against which any campus can measure its own graduates, if it works with Gallup individually…

The survey’s line of questioning goes beyond job placement and salary, also inquiring about work place and community engagement, personal relationships, physical fitness, sense of purpose and happiness, and economic management and stress…

“No one is going to suggest that colleges and universities are responsible for 100 percent of your great job and great life,” Busteed admitted, “but obviously, if you go to college and you get a degree, the odds are you increase the probability of having a good outcome.”

Given the arguments about the cost of college, I’m not surprised efforts like this are quickly moving forward. And, as the article notes, there are lots of methodological questions in play: what exactly is “a great life”? How many years after college should people be asked these questions? How can the effects of college be separated out from other life experiences (though people’s perceptions about whether college mattered is important as well)?

At the same time, I’m not opposed to trying to get at these life outcomes after college. Colleges often make the argument they improve the lives of their graduates from earning more to training for careers to giving students room to start living to critical thinking to a broader understanding of the world. Is some of the concern about measuring these things that colleges might not be able to live up to lofty claims? For example, given the findings of Academically Adrift from a few years ago, not all college students are benefiting. Once findings start trickling out, it will then be imperative to see what gets counted as “success” for colleges.

Using randomized controlled trials to test methods for addressing global poverty

Posted on by

Here is a relatively new way to test options for addressing poverty: use randomized controlled trials.

What Kremer was suggesting is a scientific technique that has long been considered the gold standard in medical research: the randomized controlled trial. At the time, though, such trials were used almost exclusively in medicine—and were conducted by large, well-funded institutions with the necessary infrastructure and staff to manage such an operation. A randomized controlled trial was certainly not the domain of a recent PhD, partnering with a tiny NGO, out in the chaos of the developing world…

The study wound up taking four years, but eventually Kremer had a result: The free textbooks didn’t work. Standardized tests given to all students in the study showed no evidence of improvement on average. The disappointing conclusion launched ICS and Kremer on a quest to discover why the giveaway wasn’t helping students learn, and what programs might be a better investment.

As Kremer was realizing, the campaign for free textbooks was just one of countless development initiatives that spend money in a near-total absence of real-world data. Over the past 50 years, developed countries have spent something like $6.5 trillion on assistance to the developing world, most of those outlays guided by little more than macroeconomic theories, anecdotal evidence, and good intentions. But if it were possible to measure the effects of initiatives, governments and nonprofits could determine which programs actually made the biggest difference. Kremer began collaborating with other economists and NGOs in Kenya and India to test more strategies for bolstering health and education…

In the decade since their founding, J-PAL and IPA have helped 150 researchers conduct more than 425 randomized controlled trials in 55 countries, testing hypotheses on subjects ranging from education to agriculture, microfinance to malaria prevention, with new uses cropping up every year (see “Randomize Everything,” below). Economists trained on randomized controlled trials now work in the faculties of top programs, and some universities have set up their own centers to support their growing rosters of experiments in the social sciences.

If this is indeed a relatively new approach, what took so long? Perhaps the trick was thinking that experiments, typically associated with very controlled laboratory or medical settings, could be preformed in less controlled settings. As the article notes, they are not easy to set up. One of the biggest issues might be randomizing enough people into the different groups to wash out all of the possible factors that might influence the results.

This also seems related to the uptick in interest in natural experiments where social scientists take advantage of “natural” occurrences, perhaps a policy change or a natural disaster, to compare results across groups. Again, laboratories offer controlled settings but there are only so many things that can be addressed and the number of people in the studies tend to be pretty small.

Sociologist on how studying an extremist group led to a loss of objectivity

Posted on by

A retired sociologist who studied the Aryan Nation discusses how his research led to a loss of objectivity and a change of research topics:

Aho began his research in the mid-1980s with a focus on the most notorious group in Idaho, the Aryan Nation Church near Coeur d’Alene and Hayden Lake. Annual conferences were held there with people from all around the world to fight what they called the “race war.” The group, originally formed in California, was forced to relocate to Idaho due to pressure from authorities. Aho was able to interview members of the group face to face, conduct phone interviews and correspond with prison inmates who were part of the organization.

“These individuals were genuinely good, congenial folks,” said Aho. “They were very independent, married, church-going people with deep beliefs. It was only when they gathered in groups and reaffirmed each other’s prejudices that things became dangerous…

In his research, Aho tried to place himself in his subjects’ shoes. He expressed how it is important to see yourself in the other person to find mutual ground and truths that can only be obtained by using this research methodology. However, after nearly a decade of research, he felt that he was losing objectivity and only adding to the problem.

“I spent years trying to understand the people who are attracted to violence, but I began to feel like my fascination with violence made me partly responsible for it,” Aho said. “I think I lost my sociological objectivity, and thought it was time to end my efforts of trying to understanding it, and move on to other scholarly activities.”

Some candor about researching a difficult topic. Given statements by some recently that we should not “commit sociology” and refrain from looking for explanations for violence, we could just ignore such groups. But, looking for explanations is not the same as excusing or condoning behavior and may help limit violence in the future. At the same time, spending lots of time with people, whether they are good or bad, can lead to relationships and a humanizing the research subject. This may provide better data for a while as well as dignity for the research subject but can lead to the “going native” issue that anthropologists sometimes discuss. A sociologist wants to be able to remain an observer and analyst, even as they try to put themselves in the shoes of others.

It would be interesting to hear the opinions of sociologists regarding studying clearly unpopular groups like white supremacists/terrorists. Sociologists are often interested in studying disadvantaged or voiceless groups but what about groups with which they profoundly disagree?

What exactly is the ethnographic line between entertaining stories and academic content?

Posted on by

One sociologist expresses his dislike of Sudhir Venkatesh’s latest book Floating City and raises an interesting question about the line between entertaining stories and sociological analysis:

 

Early in my career I had a book reviewed by a wise sage of Fleet Street who mocked my sociological language and academic conceits. Some years later, a literary agent explained that if I were to abandon academic conventions but retain the stories of “low life”, untold riches would surely follow. In exchange for academic orthodoxy I should situate myself firmly in the narrative, charting my no doubt dangerous but hilarious adventures through the life-world of the lower orders. Although I rejected this suggestion, I always wondered what an ethnography eviscerated of academic content would look like.

Floating City has answered this question and much more. A self-styled “maverick sociologist” and experienced urban ethnographer who holds a named chair at an Ivy League university, Sudhir Venkatesh drains his study of illegal entrepreneurship in New York of most of the academic conventions that will come naturally to someone with almost a quarter of a century of experience in the dangerous enclaves of US higher education. On arriving in New York from Chicago, Venkatesh finds life in a “World City” unfamiliar (but isn’t Chicago a World City?), with old standbys such as neighbourhood and community apparently made redundant by globalisation…

The populist format of Floating City is insistent that the Homeric author is a pioneer, the first in the field, an ingénue intent on emphasising that the unique demands of researching the global city require a rejection of academic tradition. But his outsider stance and maverick posturing are an irritant; they get in the way of his street-savvy case studies and vignettes of urban life. For at the heart of this book is a conventional ethnography that challenges the author’s ingénue facade by using the full arsenal of ethnographic orthodoxy, and along the way highlighting the enduring importance of community and neighbourhood.

The best of this book is precisely the result of the “traditional sociology” that Venkatesh derides. I will not forget in a hurry the poignancy of his descriptions of Manjun and his family, who run a shop selling pornographic DVDs, or his fine-grained interaction with a group of aspirational prostitute women struggling on streets paved with something less than gold. His interaction with coke-addled socialites I found irritating and somewhat pointless, and his aim to make connections between the so-called upper- and underworlds fizzles out. For while the city may be more fluid, the rich and the poor remain separated, and the dots are not joined.

This gets at some basic issues sociologists face:

1. What audience should they aim for when writing a book? If they write for the public, they may be derided by fellow academics for not having enough academic content. If they write a more academic book, few in the public will touch it. It is rare to find such sociological works that can easily transcend these audience boundaries. Just the other day, I was explaining to someone how the book Bowling Alone was one of these rare texts that contained an academic approach to a problem that caught people’s attention.

2. How exactly should ethnographic research be conveyed? Yes, there are stories to relate but it also needs to contain some sociological analysis. Different ethnographers rely on different mixes of narrative and analysis but the sociological element still has to come through to some degree.

3. If the sociologist is studying “low life,” how can these lives be explored without being exploitative or cheapening? Salaciousness might sell but it does not tend to grant dignity and respect to those being described. Sociologists also often develop personal relationships with those they are studying and these complicates the retelling.

These are common concerns of ethnographers and there are not easy answers. Amongst themselves, sociologists disagree how to do these things better.

ESPN pushing unscientific “NFL Hazing Survey”

Posted on by

On Sportscenter last night as well as on their website, ESPN last night was pushing a survey of 72 NFL players regarding the recent locker room troubles involving the Miami Dolphins. The problem: the survey is unscientific, something they mentioned at the beginning of the TV reports. The online story includes a similar disclaimer at the beginning of the third paragraph:

But in an unscientific survey conducted by team reporters for ESPN.com’s NFL Nation over two days this week, Incognito does not have the same level of support from some of his peers. Three players participated from each team surveyed, with 72 players in all asked three questions. The players taking part were granted anonymity.

If the survey is unscientific, why do they then spend time discussing the results? If they admit upfront that it is unscientific, what exactly could the viewer/reader learn from the data? It is good that they mentioned the unscientific sample but their own statement suggests we shouldn’t put much stock in what they say next.

Studying religiosity by text messages and three minute surveys

Posted on by

A new study of religiosity utilizes text messages and short surveys:

After signing up on soulpulse.org, users receive text messages twice a day for 14 days that direct them to a 15 to 20-question survey. These questions gather data on daily spiritual attitudes and physical influences at points during the day, such as quality of sleep, amount of exercise and alcohol consumption. The average length of time required to complete the survey is around three minutes and is designed with the ideas of simplicity and ease of use.

At the end of the two-week testing period, the reward for participants is a comprehensive review of their data that allows them to see and learn more about their spiritual mindsets. In return, the research team is given the opportunity to analyze the information that they have collected. Wright said they have already found that people report the greatest feelings of spirituality on Sundays and the least amount on Wednesdays.

A collection of three-minute surveys however, took months of collaboration across the country to complete. 18 months of planning and 10 trips to Silicon Valley were necessary, as well as a team of people who each contributed a unique skillset to the group. The Soulpulse team consists of four computer programmers, three public engagers and six academic advisors – including UConn professors Crystal Park and Jeremy Pais.

Measuring religiosity is well established in sociology but it often relies on people reporting on their past behavior. For example, some sociologists suggest church attendance figures are regularly inflated. Using text messages would allow more up-to-date data as the goal is to quickly interrupt people’s activity and get their more accurate take on their religious behavior.

Generally, I would guess sociology and other social science fields are headed in this direction for data collection: less formal and more minute to minute. In the past, some of this was done with time diaries or logs. But, even these posed problems as at the end of the day a person might misremember or reinterpret their earlier actions. Utilizing text messages or pop-up Internet surveys or other means could yield more better data, utilize newer technologies respondents are regularly engaging, and perhaps even take less time in the long run.