Tag Archives: experiments

Facebook’s Data Science Team running experiments

Posted on by

Facebook’s Data Science Team of 12 researchers is working with all of its data (900 million users worth) and running experiments:

“Recently the Data Science Team has begun to use its unique position to experiment with the way Facebook works, tweaking the site-the way scientists might prod an ant’s nest-to see how users react… So [Eytan Bakshy] messed with how Facebook operated for a quarter of a billion users. Over a seven-week period, the 76 million links that those users shared with each other were logged. Then, on 219 million randomly chosen occasions, Facebook prevented someone from seeing a link shared by a friend. Hiding links this way created a control group so that Bakshy could assess how often people end up promoting the same links because they have similar information sources and interests.

“He found that our close friends strongly sway which information we share, but overall their impact is dwarfed by the collective influence of numerous more distant contacts-what sociologists call “weak ties.” It is our diverse collection of weak ties that most powerfully determines what information we’re exposed to.”

But if that sounds a little creepy, it shouldn’t. Well, not too creepy, because these kinds of experiments aren’t designed to influence us, but rather understand us. The piece continues:

“Marlow says his team wants to divine the rules of online social life to understand what’s going on inside Facebook, not to develop ways to manipulate it. “Our goal is not to change the pattern of communication in society,” he says. “Our goal is to understand it so we can adapt our platform to give people the experience that they want.” But some of his team’s work and the attitudes of Facebook’s leaders show that the company is not above using its platform to tweak users’ behavior. Unlike academic social scientists, Facebook’s employees have a short path from an idea to an experiment on hundreds of millions of people.”

I think there is a lot of room to explore the world of weak ties on Facebook and similar websites. Just how much do friends of friends affect us? What is the impact of people a few ties along in our network? For example, the book Connected shows that traits like obesity and happiness are tied to network behavior which could be examined on Facebook.

I would guess some people may not like hearing this but there are at least three points in Facebook’s favor here:

1. They are not the only online company running such experiments. Google has been doing such things with search results for quite a while. Theoretically, these experiments could help create a better user experience.

2. People are voluntarily giving their data. I don’t think these companies have to explain that user’s data might be used in experiments…but perhaps I am wrong?

3. This is “Big Data” writ large. Facebook and others would love to be able to run randomized trials with this large group and with all of the information available to researchers.

Sociology experiment: mixing strong academics and athletics at Northwestern

Posted on by

Chicago Tribune columnist David Haugh suggested yesterday that Northwestern University is facing a sociology experiment by wanting strong academics and athletics:

So continued America’s fascinating sports sociology experiment in Evanston: Can a major-college sports program thrive in an environment in which winning clearly isn’t the No. 1 determinant of success? As Final Four week begins, it would behoove every basketball campus to reconsider its definitions of thrive, winning and success…

So I reached a different conclusion about Carmody but loved the way Phillips defended his. I loved the idea of a Big Ten school espousing ideals more typically found in Division III programs, of an AD taking an unpopular route by taking a stand for something noble. I can applaud a decision I wouldn’t have made because of what it symbolizes.

On one hand, Northwestern shows it recognizes the Big Ten basketball arms race by working on plans for $250 million worth of necessary facility upgrades. On the other, it stayed true to an underlying mission colleges usually ignore by keeping a coach who does things the right way…

Roll your eyes and look up Pollyannaish if you wish. But ultimately Phillips’ decision embodied the mandate for college sports programs Secretary of Education Arne Duncan outlined in a news conference on the eve of the NCAA tournament intended to remind schools of their priorities. Theoretically, Northwestern’s stance also reflected the emphasis more Big Ten and BCS-conference universities must consider in light of the NCAA linking academic progress rate with tournament eligibility beginning in 2013.

Haugh defends Northwestern’s actions in trying to do both: have high academic standards and have competitive sports programs. A few thoughts about this:

1. I’ve heard a lot of this argument at both Notre Dame and Northwestern. The situations are slightly different (Northwestern doesn’t have the past football glory of Notre Dame) but the argument generally go like this: the schools need to lower their academic standards in football and basketball if they really hope to compete for national championships. Perhaps this is right – neither school is the kind of powerhouse that brings athletes in and spits them out. But, as Haugh suggests, the schools have some different priorities.

2. These different priorities are not just tertiary concerns: Northwestern is a serious academic school (as is Notre Dame). According to the US News and World Report rankings, Northwestern is the #12 undergraduate school (Notre Dame is #19), #4 among business graduate schools, and #9 among education graduate schools (among other high rankings). So this isn’t quite a high-ranking Division III school; Northwestern is a strong academic university where there are many things going on besides athletics.

3. In other sports, Northwestern and Notre Dame can do just fine. Let’s be honest here: what is really driving these arguments is football (and maybe a little basketball). Interestingly, both Northwestern and Notre Dame are not bad at these sports but also not great. Northwestern football has been improved since the mid 1990s but they are not going to compete for a national championship. Northwestern basketball just missed the NCAA tournament but they played in perhaps the toughest conference this year and had a number of chances to make their season really memorable.

3a. If you look at the Director’s Cup rankings which account for all sports, some more academic schools do just fine. For example, look at the most recent March 22 rankings: Stanford is #1, Duke is #28, Notre Dame is #34, and Northwestern is #63. Granted, the big public schools seem to do well in these rankings across the sports but it’s not like academic schools can’t compete in other sports. For example, Northwestern has been known in recent years for two other sports: fencing and women’s lacrosse. While these are not high profile, the athletes have proven can be champions as well as high-performing athletes.

3b. I wonder at times if Northwestern isn’t lucky on this front to be located in Chicago. Since Chicago doesn’t care much about college sports, schools like Northwestern and the University of Chicago (who used to be in the Big 10 but now competes at the Division III level) don’t have to go the athletic route.

In the end, I think Northwestern will be just fine. This is a sociology experiment that doesn’t have to happen – not all colleges need to be athletic powerhouses.

Quick Review: The Better Angels of Our Nature

Posted on by

I hadn’t looked at much from psychologist Stephen Pinker for a while but I was intrigued by his latest book The Better Angels of Our Nature: Why Violence Has Declined. Here are a few comments about this thought-provoking work:

1. Here is Pinker’s argument: we must just be living in the safest era in human history as violent crime is down and wars affect fewer people. If you adjust for the population on earth at the time, World War II barely makes the top 10 (while typically lists put it at #1). Since World War II, fewer people are affected by violence and most people don’t know this.

2. Best argument of this book: this remarkable peacefulness is almost completely under-the-radar and people need to recognize how much safer the world has become. (I’ve noted before the incorrect perceptions regarding crime.)

2a. Pinker marshals a lot of evidence to show the declining trends in violence. In fact, Pinker talks about this for dozens upon dozens of pages. In fact, if you went by the percentage of the book devoted to each topic, you might think Pinker is more of a social scientist who studies violence and who is most interested in how societies and cultures have changed in such a way as to deincentivize violence. Overall, the number of wars have decreased, the number of wars involving great powers has decreased, the number of soldier and civilian deaths has decreased, and the length of wars have decreased. Pinker is, of course, building upon the work of many others but there are a lot of charts and figures here that I find quite convincing.

2b. Several periods were key to this change: the Enlightenment which didn’t necessarily limit violence but brought about ideas and values that eventually contributed and the post-World War II era when the world responded to the horror by promoting international peace and human rights.

3. The catch: Pinker is committed to going beyond a social explanation in the decrease in violence and wants to argue that this has trickled down to individuals. On one hand, you could imagine a number of sociologists making this argument: changes in society and culture influence the choices available to and made by individuals. On the other hand, Pinker wants to go further and even suggest that humans have evolved away from violence. Making this connection between social and individual change is tougher to do and Pinker relies a lot on social psychology experiments such as Prisoner’s Dilemma and the Ultimatum Game. The social and cultural change arguments are convincing but taking this next step to the individual level is more problematic. Part of the problem here might be that Pinker is so committed to his own perspective that he is determined to push his points about rationality further than they can go.

4. An interesting issue: Pinker argues that one way in which violence can get out of hand is that it requires a powerful ideology. One type of ideology that Pinker makes clear he does not like is religion which he argues is false and generally contributes to violence. In his historical overviews, Pinker makes clear that religion only contributes to and legitimizes violence and may not do any good. Additionally, the revolutions in values happened solely in the secular sphere and humans today are much more able to be rational (and religion is not that).

Overall, this is an interesting, long book that presents several intriguing arguments. Pinker provides a service in helping to fight the narrative that violence is spiraling out of control and yet has more difficulty in showing how humans have evolved into more rational beings.

Don’t dismiss social science research just because of one fradulent scientist

Posted on by

Andrew Ferguson argued in early December that journalists fall too easily for bad academic research. However, he seems to base much of his argument on the actions of one fraudulent scientist:

Lots of cultural writing these days, in books and magazines and newspapers, relies on the so-called Chump Effect. The Effect is defined by its discoverer, me, as the eagerness of laymen and journalists to swallow whole the claims made by social scientists. Entire journalistic enterprises, whole books from cover to cover, would simply collapse into dust if even a smidgen of skepticism were summoned whenever we read that “scientists say” or “a new study finds” or “research shows” or “data suggest.” Most such claims of social science, we would soon find, fall into one of three categories: the trivial, the dubious, or the flatly untrue.

A rather extreme example of this third option emerged last month when an internationally renowned social psychologist, Diederik Stapel of Tilburg University in the Netherlands, was proved to be a fraud. No jokes, please: This social psychologist is a fraud in the literal, perhaps criminal, and not merely figurative, sense. An investigative committee concluded that Stapel had falsified data in at least “several dozen” of the nearly 150 papers he had published in his extremely prolific career…

But it hardly seems to matter, does it? The silliness of social psychology doesn’t lie in its questionable research practices but in the research practices that no one thinks to question. The most common working premise of social-psychology research is far-fetched all by itself: The behavior of a statistically insignificant, self-selected number of college students or high schoolers filling out questionnaires and role-playing in a psych lab can reveal scientifically valid truths about human behavior…

Who cares? The experiments are preposterous. You’d have to be a highly trained social psychologist, or a journalist, to think otherwise. Just for starters, the experiments can never be repeated or their results tested under controlled conditions. The influence of a hundred different variables is impossible to record. The first group of passengers may have little in common with the second group. The groups were too small to yield statistically significant results. The questionnaire is hopelessly imprecise, and so are the measures of racism and homophobia. The notions of “disorder” and “stereotype” are arbitrary—and so on and so on.

Yet the allure of “science” is too strong for our journalists to resist: all those numbers, those equations, those fancy names (say it twice: the Self-Activation Effect), all those experts with Ph.D.’s!

I was afraid that the actions of one scientist might taint the work of many others.

But there are a couple of issues here and several are worth pursuing:

1. The fact that Stapel committed fraud doesn’t mean that all scientists do bad work. Ferguson seems to want to blame other scientists for not knowing Stapel was committing fraud – how exactly would they have known?

2. Ferguson doesn’t seem to like social psychology. He does point to some valid methodological concerns: many studies involve small groups of undergraduates. Drawing large conclusions from these studies is difficult and indeed, perhaps dangerous. But this isn’t all social psychology is about.

2a. More generally, Ferguson could be writing about a lot of disciplines. Medical research tends to start with small groups and then decisions are made. Lots of research, particularly in the social sciences, could be invalidated if Ferguson was completely right. Ferguson really would suggest “Most such claims of social science…fall into one of three categories: the trivial, the dubious, or the flatly untrue.”?

3. I’ve said it before and I’ll say it again: journalists need more training in order to understand what scientific studies mean. Science doesn’t work in the way that journalists suggests where there is a steady stream of big findings. Rather, scientists find something and then others try to replicate the findings in different settings with different populations. Science is more like an accumulation of evidence than a lot of sudden lightning strikes of new facts. One small study of undergraduates may not tell us much but dozens of such experiments among different groups might.

4. I can’t help but wonder if there is a political slant to this: what if scientists were reporting positive things about conservative viewpoints? Ferguson complains that measuring things like racism and homophobia are difficult but this is the nature of studying humans and society. Ferguson just wants to say that it is all “arbitrary” – this is simply throwing up our hands and saying the world is too difficult to comprehend so we might as well quit. If there isn’t a political edge here, perhaps Ferguson is simply anti-science? What science does Ferguson suggest is credible and valid?

In the end, you can’t dismiss all of social psychology because of the actions of one scientist or because journalists are ill-prepared to report on scientific findings.

h/t Instapundit

The “Trolley Problem” in virtual 3-D

Posted on by

A psychologist has taken a classic experiment, the “Trolley Problem,” to the virtual realm:

Virtual reality, however, is emerging as intriguing new tool because it enables researchers, to some degree, test a subject’s claim by simulating just about any given situation. Recently, Carlos David Navarrete, a evolutionary psychologist at Michigan State University, applied the technology to shed light on how people might respond when faced with an ethical conundrum.The Catch-22 bind he chose to put his subjects in is a popular philosophical thought experiment known as the “Trolley Problem.” The classical version of it goes a little something like this: You’re a train worker who observes a runaway train moving down a track where five people are about to get run over. You can pull a switch that diverts the train onto another track but you would end up killing one person who happens to be walking on the alternative track unaware. What would you do? Think about the situation in a pragmatic sense (saving more lives) and pull the switch? Or do nothing, which can be viewed as a wash-your-hands-of-any-responsibility decision.

The virtual simulation version was devised by wiring up test subjects with eyewear that generated a 3-D re-enactment of such a scenario. Attached to their fingertips were electrodes that measured their heartbeat and other indicators of their emotional state as they were forced to make a difficult decision. Using a joystick, users can either re-route the runaway boxcar, killing a lone hiker, or do nothing and let the it kill the group of five hikers…

While the findings corroborated with the results of a previous study that relied on self-reported methods, the experiment also showed that participants who did not pull the switch were more emotionally aroused. This means that their inaction might not be so much a conscious choice but a result of freezing up during highly anxious moments, which is akin to a solider failing to fire his weapon in battle, Navarrete said. Perhaps if they had remained calm enough to process what was happening, the percentage of people who would have pulled the switch to save five and let one die might have actually been greater.

Several thoughts:

1. The finding about emotional arousal reminds me of Malcolm Gladwell’s summarizing in Blink: humans make different decisions in emotionally charged situations.

2. The article is set up by suggesting that this kind of virtual research helps get around the issue facing the social sciences that people say one thing and then do another. While I don’t disagree that this is a problem, people’s stated beliefs and attitudes are still consequential. Take a story I blogged about a few days ago: people may be getting married at lower rates but majorities still aspire to get married, reinforcing a social norm. We need data on both actions and beliefs.

3. I’m tempted to ask whether people’s responses in this virtual world are different than if they were in the real situation. Is there a larger body of research that suggests these virtual experiments are truly better than typical research experiments?

4. Whenever I have presented this experiment to students, they tend to find to find it interesting. Also, since this one is fairly straightforward (one life for five), introducing variants to it such as having to push a large man onto the tracks to stop the train, thus giving the bystander more culpability, can change people’s responses.

Social science findings can help you find the perfect gift

Posted on by

As you shop this holiday season, some recent social science research can help you find just the gift you need. Keep these principles in mind:

1. You don’t have to spend any time looking for “thoughtful” gifts.

2. You don’t have to spend much money, either.

3. Actually, you may not have to spend any money.

Yes, I know this sounds too good to be true. I was skeptical, too, if only because it contradicts a previous holiday column of mine. After looking at anthropological research into the potlatch, and talking with a Kwakwaka’wakw Indian chief who carries on this gift-giving ritual in British Columbia, I concluded that lavish presents are essential to social harmony.

Read on to find out the results of experimental studies and how this compares to the ritual of potlatches. In this whole process, you may be best off using your “generalized other” and trying to anticipate what the receiver might want rather than doing what you would want.

It strikes me that all of this is inherently cultural. As gift givers and receivers, we have ideas about what the social norms are for each of these positions. When the norms (and our expectations) are not met, we feel hurt. It seems like these experiments are suggesting that the norms about gifting are changing and “manners” and “polite” behavior hasn’t quite caught up yet. If my cultural idea is correct, then experiments done in different cultural settings or perhaps even among different American generations would show differences.

An alternative takeaway: the Amazon wish list will (or already is?) taking over the whole ritual of gifting in American culture.

Higher taxes might push companies to leave but not necessarily wealthy residents?

Posted on by

Many municipalities and states are looking for ways to raise additional tax revenue and this has led to conflict with companies that either have had or want tax breaks to stay where they are (a prominent Illinois example here). But we could also consider whether higher tax rates prompt wealthier residents to move elsewhere. Some evidence from New York City suggests this did not happen:

According to the Census Bureau’s latest American Community Survey, the average household income of those who left the state in 2010 was $44,739. The average for those who came was $55,419 — the largest differential in at least five years…

A separate analysis of census data found that the number of households making more than $250,000 who lived in New York a year earlier but left peaked in 2004 and has generally declined since 2007. About 14,000 households in 2009 and the about the same number in 2010 reported having left New York within the past year, the lowest numbers in that category since 2003.

That analysis did not take into account inflation, and could reflect lower migration rates in general across the country.

As this short piece suggests, we may not want to run and apply this to all wealthy residents in the United States. Additionally, if this can be done with American Community Survey data for New York City, why not do it with other areas of the country in order to make comparisons? Then we could find out whether this data is more reflective of New York City and its relative wealth and importance as a finance and cultural center than of larger trends about wealthy people.

I do wonder about the value of using short-term migration data to prove points about new legislation and revised taxes. People could move for a lot of reasons beyond just one change and I don’t think the ACS data tells us why people move. This could be a clever way to examine a “natural experiment” but there needs to be care taken in interpreting the results.

Why cases of scientific fraud can affect everyone in sociology

Posted on by

The recent case of a Dutch social psychologist admitting to working with fraudulent data can lead some to paint social psychology or the broader discipline of sociology as problematic:

At the Weekly Standard, Andrew Ferguson looks at the “Chump Effect” that prompts reporters to write up dubious studies uncritically:

The silliness of social psychology doesn’t lie in its questionable research practices but in the research practices that no one thinks to question. The most common working premise of social-psychology research is far-fetched all by itself: The behavior of a statistically insignificant, self-selected number of college students or high schoolers filling out questionnaires and role-playing in a psych lab can reveal scientifically valid truths about human behavior.

And when the research reaches beyond the classroom, it becomes sillier still…

Described in this way, it does seem like there could be real journalistic interest in this study – as a human interest story like the three-legged rooster or the world’s largest rubber band collection. It just doesn’t have any value as a study of abstract truths about human behavior. The telling thing is that the dullest part of Stapel’s work – its ideologically motivated and false claims about sociology – got all the attention, while the spectacle of a lunatic digging up paving stones and giving apples to unlucky commuters at a trash-strewn train station was considered normal.

A good moment for reaction from a conservative perspective: two favorite whipping boys, liberal (and fraudulent!) social scientists plus journalists/the media (uncritical and biased!), can be tackled at once.

Seriously, though: the answer here is not to paint entire academic disciplines as problematic because of one case of fraud. Granted, some of the questions raised are good ones that social scientists themselves have raised recently: how much about human activity can you discover through relatively small sample tests of American undergraduates? But good science is not based on one study anyway. An interesting finding should be corroborated by similar studies done in different places at different times with different people. These multiple tests and observations help establish the reliability and validity of findings. This can be a slow process, another issue in a media landscape where new stories are needed all the time.

This reminds me of Joel Best’s recommendations regarding dealing with statistics. One common option is to simply trust all statistics. Numbers look authoritative, often come from experts, and they can be overwhelming. Just accepting them can be easy. At the other pole is the common option of saying that all statistics are simply interpretation and are manipulated so we can’t trust any of them. No numbers are trustworthy. Neither approaches are good options but they are relatively easy options. The better route to go when dealing with scientific studies is to have the basic skills necessary to understand whether they are good studies or not and how the process of science works. In this case, this would be a great time to call for better training among journalists about scientific studies so they can provide better interpretations for the public.

In the end, when one prominent social psychologist admits to massive fraud, the repercussions might be felt by others in the field for quite a while.

What is “The Big Data Boom”on the Internet good for?

Posted on by

The Internet is a giant source of ready-to-use data:

Today businesses can measure their activities and customer relationships with unprecedented precision. As a result, they are awash with data. This is particularly evident in the digital economy, where clickstream data give precisely targeted and real-time insights into consumer behavior…

Much of this information is generated for free, by computers, and sits unused, at least initially. A few years after installing a large enterprise resource planning system, it is common for companies to purchase a “business intelligence” module to try to make use of the flood of data that they now have on their operations. As Ron Kohavi at Microsoft memorably put it, objective, fine-grained data are replacing HiPPOs (Highest Paid Person’s Opinions) as the basis for decision-making at more and more companies.

The wealth of data also makes it easy to run experiments:

Consider two “born-digital” companies, Amazon and Google. A central part of Amazon’s research strategy is a program of “A-B” experiments where it develops two versions of its website and offers them to matched samples of customers. Using this method, Amazon might test a new recommendation engine for books, a new service feature, a different check-out process, or simply a different layout or design. Amazon sometimes gets sufficient data within just a few hours to see a statistically significant difference…

According to Google economist Hal Varian, his company is running on the order of 100-200 experiments on any given day, as they test new products and services, new algorithms and alternative designs. An iterative review process aggregates findings and frequently leads to further rounds of more targeted experimentation.

This sounds like a social scientist’s dream – if we could get our hands on the data.

My big question about all of this data is this: what should be done with it? This article, and others I’ve seen, have said that it will transform business. If this is just a way for businesses to become more knowledgeable, more efficient, and ultimately, more profitable, is this enough? Occasionally, we hear of things like discovering and/or tracking epidemics by looking at search queries or tools like the “mechanical turk” to crowdsource small but needed work. On the whole, does the data from the Internet advance human flourishing or concentrate some benefits in the hands of a few or even hinder flourishing? Does this data give us insights into health and medicine, international relations, and social interactions or does it primarily give entrepreneurs and established companies the chance to make more money? Are these questions that anyone really asks or cares about?

Brit Derren Brown to test four sociology (?) theories on TV

Posted on by

If you search for YouTube videos of the famous Milgram Experiment, you’ll run into an interesting recreation on the BBC hosted by Derren Brown (see part one of three here). When I’ve showed this to students, they tend to ask why a TV performer gets to perform this experiment but no university IRB would likely allow this. I don’t know the answer to this. But, Brown is back with a new show where he is going to test four more sociological theories:

His new show, The Experiment, will see Brown trying out four sociological theories on unsuspecting citizens.

The performer said: “Three of them are relatively dark, looking into the darker side of human behaviour, and one of them is rather positive and jolly. The first one is called The Assassin.”…

He explained: “It’s whether or not it’s possible to hypnotise somebody to kill, to carry out an assassination. This is based on the testimonies given by political assassins who say they were brainwashed by the CIA.”

Some of the theories have their origins in academia, while some of them are developed by Brown himself. Which is even more concerning.

So perhaps this isn’t terribly sociological and is more entertainment/conspiracy theory. What would it take to get an American host to replicate some famous or intriguing sociological experiments on TV? What about things like the Ultimatum Game and how the results can differ across groups and cultures? Instead, we are stuck with weaker shows like What Would You Do. A show that could demonstrate that sociological studies are both intriguing and beneficial for society could go a long way toward boosting the image of the discipline.