Teaching how science and research actually works

As a regular instructor of Statistics and Social Research classes, I took note at this paragraph in a recent profile of Bruno Latour:

Latour believes that if scientists were transparent about how science really functions — as a process in which people, politics, institutions, peer review and so forth all play their parts — they would be in a stronger position to convince people of their claims. Climatologists, he says, must recognize that, as nature’s designated representatives, they have always been political actors, and that they are now combatants in a war whose outcome will have planetary ramifications. We would be in a much better situation, he has told scientists, if they stopped pretending that “the others” — the climate-change deniers — “are the ones engaged in politics and that you are engaged ‘only in science.’ ” In certain respects, new efforts like the March for Science, which has sought to underscore the indispensable role that science plays (or ought to play) in policy decisions, and groups like 314 Action, which are supporting the campaigns of scientists and engineers running for public office, represent an important if belated acknowledgment from today’s scientists that they need, as one of the March’s slogans put it, to step out of the lab and into the streets. (To this Latour might add that the lab has never been truly separate from the streets; that it seems to be is merely a result of scientific culture’s attempt to pass itself off as above the fray.)

Textbooks on Statistics and Social Research say there are right ways and wrong ways to do the work. There are steps to follow, guidelines to adhere to, clear cut answers on how to do the work right. It is all presented in a logical and consistent format.

There are hints that this may not happen all the time. Certain known factors as well as unknown issues can push a researcher off track a bit. But, to do a good job, to do work that is scientifically interesting and acceptable to the scientific community, you would want to stick to the guidelines as much as possible.

This provides a Weberian ideal type of how science should operate. Or, perhaps the opposite ideal type occasionally provides a contrast. The researcher who committed outright fraud. The scholar who stepped way over ethical boundaries.

I see one of my jobs of teaching these classes as providing how these steps work out in actuality. You want to follow those guidelines but here is what can often happen. I regularly talk about the constraints of time and money: researchers often want to answer big questions with ideal data and that does not always happen. You make mistakes, such as in collecting data or analyzing results. You send the manuscript off for review and people offer all sorts of suggestions of how to fix it. The focus of the project and the hypothesis changes, perhaps even multiple times. It takes years to see everything through to publication.

On one hand, students often want the black and white presentation because it offers clear guidelines. If this happens, do this. On the other hand, presenting the cleaner version is an incomplete education into how research works. Students need to know how to respond when the process does not go as planned and know that this does not necessarily mean their work is doomed.

Scientific research is not easy nor is it always clear cut. Coming back to the ideal type concept, perhaps we should present it as we aspire to certain standards and particular matters may be non-negotiable but there are parts of the process, sometimes small and sometimes large, that are more flexible depending on circumstances.

Get back to the actual behavior in the science of behavior

An interesting look at the replicability of the concept of ego depletion includes this bit toward the end about doing experiments:

If the replication showed us anything, Baumeister says, it’s that the field has gotten hung up on computer-based investigations. “In the olden days there was a craft to running an experiment. You worked with people, and got them into the right psychological state and then measured the consequences. There’s a wish now to have everything be automated so it can be done quickly and easily online.” These days, he continues, there’s less and less actual behavior in the science of behavior. “It’s just sitting at a computer and doing readings.”

Perhaps, just like with the reliance on smartphones in daily life, researchers are also becoming overly dependent on the Internet and computers to help them do the work. On one hand, it certainly speeds up the work, both in data collection and analysis. Speed is very important in academia where the stakes for publishing quickly and often continue to rise. On the other hand, the suggestion here is that we miss something by sitting at a computer too much and not actually analyzing behavior. We might take mental shortcuts, not ask the same kind and number of questions, and perform different analyses compared to direct observation and doing some work by hand.

This reminds me of a reading I had my social research students do last week. The reading involved the different types of notes one should take when doing fieldwork. When it came to doing the analysis, the researcher suggested nothing beat spreading out all the paper notes on the floor and immersing oneself in them. This doesn’t seem very efficient these days; whether one is searching for words in a text document or using qualitative data analysis software, putting paper all over the floor and wading through it seems time consuming and unnecessary. But, I do think the author was right: the physical practice of immersing oneself in data and observations is simply a unique experience that yields rich data.

11 recommendations from social scientists to journalists reporting scientific findings

Twenty social scientists were asked to give advice to journalists covering scientific research; here are a few of the recommendations.

1) Journalists often want clear answers to life and social problems. Individual studies rarely deliver that…

3) Journalists are obsessed with what’s new. But it’s better to focus on what’s old…

6) There’s a difference between real-world significance and statistical significance

10) Always direct readers back to the original research

And yes, not confusing correlation and causation is on the list. This would indeed be a good list for journalists and the media to keep in mind; the typical social science study produces pretty modest findings. Occasionally, there are studies that truly challenge existing theories and findings or these shifts might happen across a short amount of time or within a few studies.

At the same time, this would be a good list for the general public as well or starting students in a social science statistics or research methods course. For example, students sometimes equate using statistics or numbers with “proof” but that is not really what social science studies provide. Instead, studies tend to provide probabilities – people are more or less likely to have a future behavior or attitude (and this is covered specifically in #5 in the list). Or, we may have to explain in class how studies add up over time and lead to a consensus within a discipline rather than having a single study provide all the evidence (#s 1, 2, 3 on the list).

Using behavioral science to improve interaction with government

President Obama signed an executive order yesterday that promotes using behavioral science to make the government more user-friendly and efficient:

The report features the Social and Behavioral Sciences Team’s first year of projects, which have made government programs easier to access and more user-friendly, and have boosted program efficiency and integrity. As a result of these projects, more Servicemembers are saving for retirement, more students are going to college, more Veterans are accessing their benefits, more farmers are obtaining credit, and more families are gaining healthcare coverage.

The Federal Government administers a wide array of programs on behalf of the American people, such as financial aid to assist with college access and workplace savings plans to promote retirement security. Americans are best served when these programs are easy to access and when program choices and information are presented clearly. When programs are designed without these considerations in mind, Americans can incur real consequences. One behavioral science study found, for example, that a complex application process for college financial aid not only decreased applications for aid, but also led some students to delay or forgo going to college altogether.

Behavioral science insights—research insights about how people make decisions—not only identify aspects of programs that can act as barriers to engagement, but also provide policymakers with insight into how those barriers can be removed through commonsense steps, such as simplifying communications and making choices more clear. That same study on financial aid found that streamlining the process of applying—by providing families with assistance and enabling families to automatically fill parts of the application using information from their tax return—increased the rates of both aid applications and college enrollment.

On one hand, the administration suggests this improves efficiency and helps people make use of the help available to them. On the other hand, there are predictable responses from the other side: “Obama issues Orwellian executive order.”

These are not new ideas. Richard Thaler and Cass Sunstein (who tweeted the news of the executive order) wrote the 2008 book titled Nudge that makes policy recommendations based on such science. For example, instead of having people opt-in to programs like setting aside matched retirement savings or organ donor programs, change the default to opting out rather than opting in and see participation rates rise.

I imagine both parties might want to use this to their advantage (though it might might rile up the conservative base a bit more if it was made public) when promoting their own policies.

Should you study love in a sociology course?

I’ve seen multiple stories about a new sociology course on love at a university in India. Here is one such story:

The battle of superiority between natural and social sciences is being played out at one of India‘s oldest universities and good old Love may just become a casualty.

Among general education courses to familiarise humanities and science students with each others’ disciplines, Kolkata’s Presidency University is offering unique optional papers like “Digital Humanities”, “The Physics of Everyday Life”, and “Love” – likely to be option number 1 for most undergraduate students!

The subject of Love, hitherto the premise of departments of English and Philosophy, will be addressed for the first time by a department of sociology in an Indian university. The only other known precedent is the Sociology of Love undergraduate course offered at the University of Massachusetts in the US…

Roy hopes to cover several elements of Love – from Love-as-romance to Love-as-industry. He is hoping to bank on Love theorists like Anthony Giddens, Zygmunt Bauman and Eric Fromm, who have enriched sociological discourses with “The Transformation of Intimacy; Sexuality, Love and Eroticism in Modern Societies” , “Liquid Love” and “The Art of Loving” respectively.

My first thought is a line that I have provided to Introduction to Sociology students: if humans are involved in any way, sociologists can and will study it. Considering there is not a shortage of writing and commentary about love, sociologists should study it.

But, several articles, including this one, seem to hint at a different sort of issue: by applying social science methods to love, do social scientists change what love is? If it is shown to be influenced by social forces and norms, does this demean love? This sounds a bit silly to me: we know there is a more individual component to love (emotions, though this individualistic idea ), we know there is a physical dimension (the response of the body), and we know there is a social dimension (what love is and how it is expressed differs). Sociologists often “pull back the curtain” on social behavior but this doesn’t necessarily mean it ruins the experience of love. On the contrary, it may just enlighten people about the social dimensions of love.

Another idea. A number of social scientists have been behind the creation of popular dating websites (great phrase: “algorithms of love”): a psychologist is behind eHarmony.com, a sociologist is behind perfectmatch.com, and an anthropologist developed the algorithm behind chemistry.com. These social scientists have helped develop the idea that love can be scientific, that there are patterns that can be applied in a waiting market where plenty of people want such “Scientific” matching.

Spreadsheet errors, austerity, ideology, and social science

The graduate student who found some spreadsheet errors in an influential anti-austerity paper discusses what happened. Here is part of the conversation about the process of finding this error:

Q. You say, don’t you, that their use of data was faulty?

A. Yes. The terms we used about their data—”selective” and “unconventional”—are appropriate ones. The reasons for the choices they made needed to be given, and there was nowhere where they were.

Q. And how about their claim that your findings support their thesis that growth slows as debt rises?

A. That is not our interpretation of our paper, at all. If you read their paper, it’s interesting how they handle causality. They waffle between strong and weak claims. The weak claim is that it’s just a negative association. If that’s all they claim, then it’s not really relevant for policy. But they also make a strong claim, more in public than in the paper, that there’s causality going from high debt to drops in growth. They haven’t been obvious about that…

Q. Paul Krugman wrote in The New York Times that your work confirms what many economists have long intuitively thought. Was that your intuition?

A. Yes. I just thought it was counterintuitive when I first saw their claim. It wasn’t plausible.

Q. This is more than a spreadsheet error, then?

A. Yes. The Excel error wasn’t the biggest error. It just got everyone talking about this. It was an emperor-has-no-clothes moment.

This would make for a good case study in a methodology class in the social sciences: how much of this is about actual data errors versus different interpretations? You have people who are clearly staking out space on either side of a policy discussion and it is a bit unclear how much does this color their interpretation of “facts”/data. I suspect some time will help sort this out – if the spreadsheet was indeed wrong, shouldn’t this lead to a correction or a retraction?

I do like the fact that the original authors were willing to share their data – this is something that could happen more often in the social sciences and give people the ability to look at the data for themselves.

Getting the data to model society like we model the natural world

A recent session at the American Association for the Advancement of Science included a discussion of how to model the social world:

Dirk Helbing was speaking at a session entitled “Predictability: from physical to data sciences”. This was an opportunity for participating scientists to share ways in which they have applied statistical methodologies they usually use in the physical sciences to issues which are more ‘societal’ in nature. Examples stretched from use of Twitter data to accurately predict where a person is at any moment of each day, to use of social network data in identifying the tipping point at which opinions held by a minority of committed individuals influence the majority view (essentially looking at how new social movements develop) through to reducing travel time across an entire road system by analysing mobile phone and GIS (Geographical Information Systems) data…

With their eye on the big picture, Dr Helbing and multidisciplinary colleagues are collaborating on FuturICT, a 10-year, 1 billion EUR programme which, starting in 2013, is set to explore social and economic life on earth to create a huge computer simulation intended to simulate the interactions of all aspects of social and physical processes on the planet. This open resource will be available to us all and particularly targeted at policy and decision makers. The simulation will make clear the conditions and mechanisms underpinning systemic instabilities in areas as diverse as finance, security, health, the environment and crime. It is hoped that knowing why and being able to see how global crises and social breakdown happen, will mean that we will be able to prevent or mitigate them.

Modelling so many complex matters will take time but in the future, we should be able to use tools to predict collective social phenomena as confidently as we predict physical pheno[men]a such as the weather now.

This will require a tremendous amount of data. It may also require asking for a lot more data from individual members of society in a way that has not happened yet. To this point, individuals have been willing to volunteer information in places like Facebook and Twitter but we will need much more consistent information than that to truly develop models like are suggested here. Additionally, once that minute to minute information is collected, it needs to be put in a central dataset or location to see all the possible connections. Who is going to keep and police this information? People might be convinced to participate if they could see the payoff. A social model will be able to do what exactly – limit or stop crime or wars? Help reduce discrimination? Thus, getting the data from people might be as much of a problem as knowing what to do with it once it is obtained.