Teaching how science and research actually works

As a regular instructor of Statistics and Social Research classes, I took note at this paragraph in a recent profile of Bruno Latour:

Latour believes that if scientists were transparent about how science really functions — as a process in which people, politics, institutions, peer review and so forth all play their parts — they would be in a stronger position to convince people of their claims. Climatologists, he says, must recognize that, as nature’s designated representatives, they have always been political actors, and that they are now combatants in a war whose outcome will have planetary ramifications. We would be in a much better situation, he has told scientists, if they stopped pretending that “the others” — the climate-change deniers — “are the ones engaged in politics and that you are engaged ‘only in science.’ ” In certain respects, new efforts like the March for Science, which has sought to underscore the indispensable role that science plays (or ought to play) in policy decisions, and groups like 314 Action, which are supporting the campaigns of scientists and engineers running for public office, represent an important if belated acknowledgment from today’s scientists that they need, as one of the March’s slogans put it, to step out of the lab and into the streets. (To this Latour might add that the lab has never been truly separate from the streets; that it seems to be is merely a result of scientific culture’s attempt to pass itself off as above the fray.)

Textbooks on Statistics and Social Research say there are right ways and wrong ways to do the work. There are steps to follow, guidelines to adhere to, clear cut answers on how to do the work right. It is all presented in a logical and consistent format.

There are hints that this may not happen all the time. Certain known factors as well as unknown issues can push a researcher off track a bit. But, to do a good job, to do work that is scientifically interesting and acceptable to the scientific community, you would want to stick to the guidelines as much as possible.

This provides a Weberian ideal type of how science should operate. Or, perhaps the opposite ideal type occasionally provides a contrast. The researcher who committed outright fraud. The scholar who stepped way over ethical boundaries.

I see one of my jobs of teaching these classes as providing how these steps work out in actuality. You want to follow those guidelines but here is what can often happen. I regularly talk about the constraints of time and money: researchers often want to answer big questions with ideal data and that does not always happen. You make mistakes, such as in collecting data or analyzing results. You send the manuscript off for review and people offer all sorts of suggestions of how to fix it. The focus of the project and the hypothesis changes, perhaps even multiple times. It takes years to see everything through to publication.

On one hand, students often want the black and white presentation because it offers clear guidelines. If this happens, do this. On the other hand, presenting the cleaner version is an incomplete education into how research works. Students need to know how to respond when the process does not go as planned and know that this does not necessarily mean their work is doomed.

Scientific research is not easy nor is it always clear cut. Coming back to the ideal type concept, perhaps we should present it as we aspire to certain standards and particular matters may be non-negotiable but there are parts of the process, sometimes small and sometimes large, that are more flexible depending on circumstances.

Get back to the actual behavior in the science of behavior

An interesting look at the replicability of the concept of ego depletion includes this bit toward the end about doing experiments:

If the replication showed us anything, Baumeister says, it’s that the field has gotten hung up on computer-based investigations. “In the olden days there was a craft to running an experiment. You worked with people, and got them into the right psychological state and then measured the consequences. There’s a wish now to have everything be automated so it can be done quickly and easily online.” These days, he continues, there’s less and less actual behavior in the science of behavior. “It’s just sitting at a computer and doing readings.”

Perhaps, just like with the reliance on smartphones in daily life, researchers are also becoming overly dependent on the Internet and computers to help them do the work. On one hand, it certainly speeds up the work, both in data collection and analysis. Speed is very important in academia where the stakes for publishing quickly and often continue to rise. On the other hand, the suggestion here is that we miss something by sitting at a computer too much and not actually analyzing behavior. We might take mental shortcuts, not ask the same kind and number of questions, and perform different analyses compared to direct observation and doing some work by hand.

This reminds me of a reading I had my social research students do last week. The reading involved the different types of notes one should take when doing fieldwork. When it came to doing the analysis, the researcher suggested nothing beat spreading out all the paper notes on the floor and immersing oneself in them. This doesn’t seem very efficient these days; whether one is searching for words in a text document or using qualitative data analysis software, putting paper all over the floor and wading through it seems time consuming and unnecessary. But, I do think the author was right: the physical practice of immersing oneself in data and observations is simply a unique experience that yields rich data.

11 recommendations from social scientists to journalists reporting scientific findings

Twenty social scientists were asked to give advice to journalists covering scientific research; here are a few of the recommendations.

1) Journalists often want clear answers to life and social problems. Individual studies rarely deliver that…

3) Journalists are obsessed with what’s new. But it’s better to focus on what’s old…

6) There’s a difference between real-world significance and statistical significance

10) Always direct readers back to the original research

And yes, not confusing correlation and causation is on the list. This would indeed be a good list for journalists and the media to keep in mind; the typical social science study produces pretty modest findings. Occasionally, there are studies that truly challenge existing theories and findings or these shifts might happen across a short amount of time or within a few studies.

At the same time, this would be a good list for the general public as well or starting students in a social science statistics or research methods course. For example, students sometimes equate using statistics or numbers with “proof” but that is not really what social science studies provide. Instead, studies tend to provide probabilities – people are more or less likely to have a future behavior or attitude (and this is covered specifically in #5 in the list). Or, we may have to explain in class how studies add up over time and lead to a consensus within a discipline rather than having a single study provide all the evidence (#s 1, 2, 3 on the list).

Using behavioral science to improve interaction with government

President Obama signed an executive order yesterday that promotes using behavioral science to make the government more user-friendly and efficient:

The report features the Social and Behavioral Sciences Team’s first year of projects, which have made government programs easier to access and more user-friendly, and have boosted program efficiency and integrity. As a result of these projects, more Servicemembers are saving for retirement, more students are going to college, more Veterans are accessing their benefits, more farmers are obtaining credit, and more families are gaining healthcare coverage.

The Federal Government administers a wide array of programs on behalf of the American people, such as financial aid to assist with college access and workplace savings plans to promote retirement security. Americans are best served when these programs are easy to access and when program choices and information are presented clearly. When programs are designed without these considerations in mind, Americans can incur real consequences. One behavioral science study found, for example, that a complex application process for college financial aid not only decreased applications for aid, but also led some students to delay or forgo going to college altogether.

Behavioral science insights—research insights about how people make decisions—not only identify aspects of programs that can act as barriers to engagement, but also provide policymakers with insight into how those barriers can be removed through commonsense steps, such as simplifying communications and making choices more clear. That same study on financial aid found that streamlining the process of applying—by providing families with assistance and enabling families to automatically fill parts of the application using information from their tax return—increased the rates of both aid applications and college enrollment.

On one hand, the administration suggests this improves efficiency and helps people make use of the help available to them. On the other hand, there are predictable responses from the other side: “Obama issues Orwellian executive order.”

These are not new ideas. Richard Thaler and Cass Sunstein (who tweeted the news of the executive order) wrote the 2008 book titled Nudge that makes policy recommendations based on such science. For example, instead of having people opt-in to programs like setting aside matched retirement savings or organ donor programs, change the default to opting out rather than opting in and see participation rates rise.

I imagine both parties might want to use this to their advantage (though it might might rile up the conservative base a bit more if it was made public) when promoting their own policies.

Should you study love in a sociology course?

I’ve seen multiple stories about a new sociology course on love at a university in India. Here is one such story:

The battle of superiority between natural and social sciences is being played out at one of India‘s oldest universities and good old Love may just become a casualty.

Among general education courses to familiarise humanities and science students with each others’ disciplines, Kolkata’s Presidency University is offering unique optional papers like “Digital Humanities”, “The Physics of Everyday Life”, and “Love” – likely to be option number 1 for most undergraduate students!

The subject of Love, hitherto the premise of departments of English and Philosophy, will be addressed for the first time by a department of sociology in an Indian university. The only other known precedent is the Sociology of Love undergraduate course offered at the University of Massachusetts in the US…

Roy hopes to cover several elements of Love – from Love-as-romance to Love-as-industry. He is hoping to bank on Love theorists like Anthony Giddens, Zygmunt Bauman and Eric Fromm, who have enriched sociological discourses with “The Transformation of Intimacy; Sexuality, Love and Eroticism in Modern Societies” , “Liquid Love” and “The Art of Loving” respectively.

My first thought is a line that I have provided to Introduction to Sociology students: if humans are involved in any way, sociologists can and will study it. Considering there is not a shortage of writing and commentary about love, sociologists should study it.

But, several articles, including this one, seem to hint at a different sort of issue: by applying social science methods to love, do social scientists change what love is? If it is shown to be influenced by social forces and norms, does this demean love? This sounds a bit silly to me: we know there is a more individual component to love (emotions, though this individualistic idea ), we know there is a physical dimension (the response of the body), and we know there is a social dimension (what love is and how it is expressed differs). Sociologists often “pull back the curtain” on social behavior but this doesn’t necessarily mean it ruins the experience of love. On the contrary, it may just enlighten people about the social dimensions of love.

Another idea. A number of social scientists have been behind the creation of popular dating websites (great phrase: “algorithms of love”): a psychologist is behind eHarmony.com, a sociologist is behind perfectmatch.com, and an anthropologist developed the algorithm behind chemistry.com. These social scientists have helped develop the idea that love can be scientific, that there are patterns that can be applied in a waiting market where plenty of people want such “Scientific” matching.

Spreadsheet errors, austerity, ideology, and social science

The graduate student who found some spreadsheet errors in an influential anti-austerity paper discusses what happened. Here is part of the conversation about the process of finding this error:

Q. You say, don’t you, that their use of data was faulty?

A. Yes. The terms we used about their data—”selective” and “unconventional”—are appropriate ones. The reasons for the choices they made needed to be given, and there was nowhere where they were.

Q. And how about their claim that your findings support their thesis that growth slows as debt rises?

A. That is not our interpretation of our paper, at all. If you read their paper, it’s interesting how they handle causality. They waffle between strong and weak claims. The weak claim is that it’s just a negative association. If that’s all they claim, then it’s not really relevant for policy. But they also make a strong claim, more in public than in the paper, that there’s causality going from high debt to drops in growth. They haven’t been obvious about that…

Q. Paul Krugman wrote in The New York Times that your work confirms what many economists have long intuitively thought. Was that your intuition?

A. Yes. I just thought it was counterintuitive when I first saw their claim. It wasn’t plausible.

Q. This is more than a spreadsheet error, then?

A. Yes. The Excel error wasn’t the biggest error. It just got everyone talking about this. It was an emperor-has-no-clothes moment.

This would make for a good case study in a methodology class in the social sciences: how much of this is about actual data errors versus different interpretations? You have people who are clearly staking out space on either side of a policy discussion and it is a bit unclear how much does this color their interpretation of “facts”/data. I suspect some time will help sort this out – if the spreadsheet was indeed wrong, shouldn’t this lead to a correction or a retraction?

I do like the fact that the original authors were willing to share their data – this is something that could happen more often in the social sciences and give people the ability to look at the data for themselves.

Getting the data to model society like we model the natural world

A recent session at the American Association for the Advancement of Science included a discussion of how to model the social world:

Dirk Helbing was speaking at a session entitled “Predictability: from physical to data sciences”. This was an opportunity for participating scientists to share ways in which they have applied statistical methodologies they usually use in the physical sciences to issues which are more ‘societal’ in nature. Examples stretched from use of Twitter data to accurately predict where a person is at any moment of each day, to use of social network data in identifying the tipping point at which opinions held by a minority of committed individuals influence the majority view (essentially looking at how new social movements develop) through to reducing travel time across an entire road system by analysing mobile phone and GIS (Geographical Information Systems) data…

With their eye on the big picture, Dr Helbing and multidisciplinary colleagues are collaborating on FuturICT, a 10-year, 1 billion EUR programme which, starting in 2013, is set to explore social and economic life on earth to create a huge computer simulation intended to simulate the interactions of all aspects of social and physical processes on the planet. This open resource will be available to us all and particularly targeted at policy and decision makers. The simulation will make clear the conditions and mechanisms underpinning systemic instabilities in areas as diverse as finance, security, health, the environment and crime. It is hoped that knowing why and being able to see how global crises and social breakdown happen, will mean that we will be able to prevent or mitigate them.

Modelling so many complex matters will take time but in the future, we should be able to use tools to predict collective social phenomena as confidently as we predict physical pheno[men]a such as the weather now.

This will require a tremendous amount of data. It may also require asking for a lot more data from individual members of society in a way that has not happened yet. To this point, individuals have been willing to volunteer information in places like Facebook and Twitter but we will need much more consistent information than that to truly develop models like are suggested here. Additionally, once that minute to minute information is collected, it needs to be put in a central dataset or location to see all the possible connections. Who is going to keep and police this information? People might be convinced to participate if they could see the payoff. A social model will be able to do what exactly – limit or stop crime or wars? Help reduce discrimination? Thus, getting the data from people might be as much of a problem as knowing what to do with it once it is obtained.

Quick Review: The Better Angels of Our Nature

I hadn’t looked at much from psychologist Stephen Pinker for a while but I was intrigued by his latest book The Better Angels of Our Nature: Why Violence Has Declined. Here are a few comments about this thought-provoking work:

1. Here is Pinker’s argument: we must just be living in the safest era in human history as violent crime is down and wars affect fewer people. If you adjust for the population on earth at the time, World War II barely makes the top 10 (while typically lists put it at #1). Since World War II, fewer people are affected by violence and most people don’t know this.

2. Best argument of this book: this remarkable peacefulness is almost completely under-the-radar and people need to recognize how much safer the world has become. (I’ve noted before the incorrect perceptions regarding crime.)

2a. Pinker marshals a lot of evidence to show the declining trends in violence. In fact, Pinker talks about this for dozens upon dozens of pages. In fact, if you went by the percentage of the book devoted to each topic, you might think Pinker is more of a social scientist who studies violence and who is most interested in how societies and cultures have changed in such a way as to deincentivize violence. Overall, the number of wars have decreased, the number of wars involving great powers has decreased, the number of soldier and civilian deaths has decreased, and the length of wars have decreased. Pinker is, of course, building upon the work of many others but there are a lot of charts and figures here that I find quite convincing.

2b. Several periods were key to this change: the Enlightenment which didn’t necessarily limit violence but brought about ideas and values that eventually contributed and the post-World War II era when the world responded to the horror by promoting international peace and human rights.

3. The catch: Pinker is committed to going beyond a social explanation in the decrease in violence and wants to argue that this has trickled down to individuals. On one hand, you could imagine a number of sociologists making this argument: changes in society and culture influence the choices available to and made by individuals. On the other hand, Pinker wants to go further and even suggest that humans have evolved away from violence. Making this connection between social and individual change is tougher to do and Pinker relies a lot on social psychology experiments such as Prisoner’s Dilemma and the Ultimatum Game. The social and cultural change arguments are convincing but taking this next step to the individual level is more problematic. Part of the problem here might be that Pinker is so committed to his own perspective that he is determined to push his points about rationality further than they can go.

4. An interesting issue: Pinker argues that one way in which violence can get out of hand is that it requires a powerful ideology. One type of ideology that Pinker makes clear he does not like is religion which he argues is false and generally contributes to violence. In his historical overviews, Pinker makes clear that religion only contributes to and legitimizes violence and may not do any good. Additionally, the revolutions in values happened solely in the secular sphere and humans today are much more able to be rational (and religion is not that).

Overall, this is an interesting, long book that presents several intriguing arguments. Pinker provides a service in helping to fight the narrative that violence is spiraling out of control and yet has more difficulty in showing how humans have evolved into more rational beings.

Cultural differences regarding the “accordion family”

A new sociology book highlights the phenomenon of the accordion family by contrasting different cultural approaches to the issue:

The global economic recession is a big driver of this phenomenon but hardly the only one. Cultural attitudes about “boomerang kids’’ vary widely. In Japan, which has been in recession for two decades, both parents and their adult children are filled with shame, and turn inward. For the Japanese, writes Newman, “personal character takes center stage,’’ not abstract explanations about diminishing economic opportunity. The Japanese “retain a strong normative sense of what is appropriate and what is deviant in the evolution from youth to adult,’’ Newman writes, and boomerang kids represent deviance (the Japanese often refer to boomerangs as “parasites’’), bringing social stigma on the entire family.

Italy is a completely different story. Italians, especially Italian men, have for centuries remained in the family home until they get married, which may find them there into their 30s or beyond. Newman interviews various 30-something Italian men living at home who quite simply don’t see a problem. The parents Newman interviews also don’t consider it dysfunctional, generally enjoying the company of their adult children. There is no social stigma attached, writes Newman, since “37 percent of [Italian] men age thirty have never lived away from home.’’

In the United States, we are somewhere in between Japanese-level stigma and Italian-style acceptance. “American attitudes are more conditional than other cultures,’’ explains Newman. Parents will support a boomerang adult child who has a plan, a way forward to improve life (e.g., through additional education, training, or an internship), but will object if their adult child is using the family home as an escape from the world.

These are some interesting contrasts across these countries. The American case in the middle here has me thinking about moral symbolic boundaries. The idea here would be that young adults living at home are fine as long as they can justify this move and reassure their parents that this is a step toward their eventual success and moving out. If they can’t make this case, this is seen as mooching. This fits with a larger American idea that we are willing to help people who also seem willing to help themselves.

I wonder if Newman also tracks these attitudes over time as perhaps these are relatively recent developments to adjust to a changing industrialized, globalized world. What aspects of a society or culture directly lead to these rules about who can live at home?

Another note from this review. Here is a paragraph that sums up the work:

Newman interviewed hundreds of boomerang adults and their parents for this accessible book, which effectively, even entertainingly, combines rigorous, statistics-driven social science with personal accounts to provide a vivid portrait of what’s happening globally.

Here is my translation of this paragraph: “It is an academic book that doesn’t read like one, meaning that you will be convinced by the data (hundreds of interviews!) but it has plenty of personal accounts to keep you entertained.” Perhaps that is too cynical. But this does offer some insights into how the general public tends to read social science research. Data, numbers in particular, can’t be too overwhelming. The book still has to be entertaining in the end, even if it is making an important point. Stories, whether they are personal accounts or good examples, are very helpful. None of these things are necessarily bad things to do yet one wonders whether the larger point of the work is muted by having to meet these requirements.

Don’t dismiss social science research just because of one fradulent scientist

Andrew Ferguson argued in early December that journalists fall too easily for bad academic research. However, he seems to base much of his argument on the actions of one fraudulent scientist:

Lots of cultural writing these days, in books and magazines and newspapers, relies on the so-called Chump Effect. The Effect is defined by its discoverer, me, as the eagerness of laymen and journalists to swallow whole the claims made by social scientists. Entire journalistic enterprises, whole books from cover to cover, would simply collapse into dust if even a smidgen of skepticism were summoned whenever we read that “scientists say” or “a new study finds” or “research shows” or “data suggest.” Most such claims of social science, we would soon find, fall into one of three categories: the trivial, the dubious, or the flatly untrue.

A rather extreme example of this third option emerged last month when an internationally renowned social psychologist, Diederik Stapel of Tilburg University in the Netherlands, was proved to be a fraud. No jokes, please: This social psychologist is a fraud in the literal, perhaps criminal, and not merely figurative, sense. An investigative committee concluded that Stapel had falsified data in at least “several dozen” of the nearly 150 papers he had published in his extremely prolific career…

But it hardly seems to matter, does it? The silliness of social psychology doesn’t lie in its questionable research practices but in the research practices that no one thinks to question. The most common working premise of social-psychology research is far-fetched all by itself: The behavior of a statistically insignificant, self-selected number of college students or high schoolers filling out questionnaires and role-playing in a psych lab can reveal scientifically valid truths about human behavior…

Who cares? The experiments are preposterous. You’d have to be a highly trained social psychologist, or a journalist, to think otherwise. Just for starters, the experiments can never be repeated or their results tested under controlled conditions. The influence of a hundred different variables is impossible to record. The first group of passengers may have little in common with the second group. The groups were too small to yield statistically significant results. The questionnaire is hopelessly imprecise, and so are the measures of racism and homophobia. The notions of “disorder” and “stereotype” are arbitrary—and so on and so on.

Yet the allure of “science” is too strong for our journalists to resist: all those numbers, those equations, those fancy names (say it twice: the Self-Activation Effect), all those experts with Ph.D.’s!

I was afraid that the actions of one scientist might taint the work of many others.

But there are a couple of issues here and several are worth pursuing:

1. The fact that Stapel committed fraud doesn’t mean that all scientists do bad work. Ferguson seems to want to blame other scientists for not knowing Stapel was committing fraud – how exactly would they have known?

2. Ferguson doesn’t seem to like social psychology. He does point to some valid methodological concerns: many studies involve small groups of undergraduates. Drawing large conclusions from these studies is difficult and indeed, perhaps dangerous. But this isn’t all social psychology is about.

2a. More generally, Ferguson could be writing about a lot of disciplines. Medical research tends to start with small groups and then decisions are made. Lots of research, particularly in the social sciences, could be invalidated if Ferguson was completely right. Ferguson really would suggest “Most such claims of social science…fall into one of three categories: the trivial, the dubious, or the flatly untrue.”?

3. I’ve said it before and I’ll say it again: journalists need more training in order to understand what scientific studies mean. Science doesn’t work in the way that journalists suggests where there is a steady stream of big findings. Rather, scientists find something and then others try to replicate the findings in different settings with different populations. Science is more like an accumulation of evidence than a lot of sudden lightning strikes of new facts. One small study of undergraduates may not tell us much but dozens of such experiments among different groups might.

4. I can’t help but wonder if there is a political slant to this: what if scientists were reporting positive things about conservative viewpoints? Ferguson complains that measuring things like racism and homophobia are difficult but this is the nature of studying humans and society. Ferguson just wants to say that it is all “arbitrary” – this is simply throwing up our hands and saying the world is too difficult to comprehend so we might as well quit. If there isn’t a political edge here, perhaps Ferguson is simply anti-science? What science does Ferguson suggest is credible and valid?

In the end, you can’t dismiss all of social psychology because of the actions of one scientist or because journalists are ill-prepared to report on scientific findings.

h/t Instapundit