Debating the reliability of social science research

A philosopher argues social science research is not that reliable and therefore should have a limited impact on public policy:

Without a strong track record of experiments leading to successful predictions, there is seldom a basis for taking social scientific results as definitive.  Jim Manzi, in his recent book, “Uncontrolled,” offers a careful and informed survey of the problems of research in the social sciences and concludes that “nonexperimental social science is not capable of making useful, reliable and nonobvious predictions for the effects of most proposed policy interventions.”

Even if social science were able to greatly increase their use of randomized controlled experiments, Manzi’s judgment is that “it will not be able to adjudicate most policy debates.” Because of the many interrelated causes at work in social systems, many questions are simply “impervious to experimentation.”   But even when we can get reliable experimental results, the causal complexity restricts us to “extremely conditional, statistical statements,” which severely limit the range of cases to which the results apply.

My conclusion is not that our policy discussions should simply ignore social scientific research.  We should, as Manzi himself proposes, find ways of injecting more experimental data into government decisions.  But above all, we need to develop a much better sense of the severely limited reliability of social scientific results.   Media reports of research should pay far more attention to these limitations, and scientists reporting the results need to emphasize what they don’t show as much as what they do.

Given the limited predictive success and the lack of consensus in social sciences, their conclusions can seldom be primary guides to setting policy.  At best, they can supplement the general knowledge, practical experience, good sense and critical intelligence that we can only hope our political leaders will have.

Several quick thoughts:

1. There seems to be some misunderstanding about the differences between the social and natural sciences. The social sciences don’t have laws in the same sense that the natural sciences do. People don’t operate like planets (to pick up on one of the examples). Social behaviors change over time in response to changing conditions and this makes study more difficult.

2. There is a heavy emphasis in this article on experiments. However, these are more difficult to conduct in the social realm: it is hard to control for all sorts of possible influential factors, have a sizable enough N to make generalizations, and experiments in the “harder sciences” like medicine have some of their own issues (see this critique of medical studies).

3. Saying the social sciences have some or a little predictive ability is different than saying they have none. Having some knowledge of social life is better than none when crafting policy, right?

4. Leaders should have “the general knowledge, practical experience, good sense and critical intelligence” to be able to make good decisions. Are these qualities simply individualistic or could social science help inform and create these abilities?

5. While there are limitations to doing social science research, there are also ways that researchers can increase the reliability and validity of studies. These techniques are not inconsequential; there are big differences between good research methods and bad research methods in what kind of data they produce. There is a need within social science to think about “big science” more often rather than pursuing smaller, limited studies but these studies than can speak to broader questions typically require more data and analysis which in turn requires more resources and time.

McMansion owners are bike haters

I know the arguments between drivers and bike riders can become quite heated but I haven’t seen this twist before regarding bike-hating McMansion owners:

It takes just five minutes on top of this bike for me to know I am good for the environment, healthy, frugal, smarter than all of y’all.

Whoa! Slow your roll, Virginia boy. Can’t you see that I’m busy saving the Earth on my bike? That SUV of yours takes up half the city. I bet you live on a huge cul-de-sac, in a McMansion with your own septic system and sad little saplings planted by the developer who chopped down all the mature trees to build that monument to yourself. I bet you don’t even recycle.

I roll my eyes at you, shake my helmet head at your obvious ignorance.

Headline of this column: “Bike lover, bike hater: Depends on whether you’re on four wheels or two.”

I’m always intrigued by the propensity in our culture to label people based on one consumer item, whether it is a McMansion or a bicycle. Here we get a classic description of a McMansion owner: because a person lives in a McMansion, they hate the earth, drive an SUV, and are generally self-centered. Granted, buying a McMansion is a large monetary choice, a home probably the biggest single investment people will make in a lifetime, and large symbolic choice as Americans have long held that one’s home should reflect those who live in them. It would be interesting to see how these single choices, McMansion or bike, line up with other consumer choices: do bike riders live in the city, tend to drive a Prius (or even better, not own a car and utilize Zipcar), and shop at Whole Foods while McMansion owners are suburbanites who tend to drive SUVs and shop at Walmart?

This is a reminder that moral values are often attached to consumer goods. Buying items isn’t simply about functionality or desires but rather indicate how a consumer views the world and what they value. Additionally, certain items, such as McMansions, are clearly viewed as moral signals by others.

 

No, Ikea does not sell a prefabricated home

Two months ago, I noted a story about a prefab home whose interior was outfitted by Ikea. Apparently, lots of people thought Ikea was providing the whole home:

The story went nuts. Literally within an hour, we were getting emails from the Phillippines to South America to Australia to Iceland — it seemed to be real big in Iceland. We heard right away from Greece and Italy — people who were interested in knowing more about an “Ikea house.” We got 2 million Web hits.

It got mentioned on the “Tonight Show” and “Today.” The Huffington Post picked it up but later issued a correction.

We got 4,000 emails in just the first few days after the erroneous story broke. We tried to do our best to respond right away, explaining to people what we do and that the Aktiv house just contains some Ikea products.

Ikea put out a corporate press release explaining things, saying that despite erroneous reports, it is not selling prefabricated homes in the United States. We were told that people were walking into Ikea stores around the world and asking about getting a house.

As I noted two months ago, you can still get a prefabricated home from Menards!

Based on the popularity of this idea, I wonder if Ikea is looking into providing prefabricated houses. I’m sure there would be some issues of economies of scale and getting into home development but I imagine Ikea could tackle these issues if they wanted to.

Another question: are there are other brands that people would buy prefab houses from? What if Pottery Barn got into this?

David Brooks, “Boo-boos in Paradise,” and American public intellectuals

I like David Brooks’ pop sociology analysis of the suburbs in Bobos in Paradise but a piece in Philadelphia suggests Brooks got some of his facts wrong:

Brooks, an agile and engaging writer, was doing what he does best, bringing sweeping social movements to life by zeroing in on what Tom Wolfe called “status detail,” those telling symbols — the Weber Grill, the open-toed sandals with advanced polymer soles — that immediately fix a person in place, time and class. Through his articles, a best-selling book, and now a twice-a-week column in what is arguably journalism’s most prized locale, the New York Times op-ed page, Brooks has become a must-read, charming us into seeing events in the news through his worldview.

There’s just one problem: Many of his generalizations are false. According to Amazon.com sales data, one of Goodwin’s strongest markets has been deep-Red McAllen, Texas. That’s probably not, however, QVC country. “I would guess our audience would skew toward Blue areas of the country,” says Doug Rose, the network’s vice president of merchandising and brand development. “Generally our audience is female suburban baby boomers, and our business skews towards affluent areas.” Rose’s standard PowerPoint presentation of the QVC brand includes a map of one zip code — Beverly Hills, 90210 — covered in little red dots that each represent one QVC customer address, to debunk “the myth that they’re all little old ladies in trailer parks eating bonbons all day.”

But this isn’t the main complaint of this arguement: rather, the main problem is that Brooks is considered a public intellectual and his words have a lot of weight:

On the publication of Bobos, New York Times critic Walter Goodman lumped Brooks with William H. Whyte Jr., author of The Organization Man, and David Riesman, who wrote The Lonely Crowd, as a practitioner of “sociological journalism.” (In the introduction to Bobos, Brooks invoked Whyte — plus Jane Jacobs and John Kenneth Galbraith — as predecessors.) In 2001, the New School for Social Research, in Manhattan, held a panel discussion in which real-life scholars pondered the bobo. When, in 2001, Richard Posner ranked the 100 highest-profile public intellectuals, Brooks came in 85th, just behind Marshall McLuhan at 82nd, and ahead of Garry Wills, Isaiah Berlin and Margaret Mead.

Ironically, Richard Florida is granted the final academic say regarding needing more serious public intellectuals:

Richard Florida, a Carnegie Mellon demographer whose 2002 book The Rise of the Creative Class earned Bobos-like mainstream cachet, nostalgizes an era when readers looked to academia for such insights:

“You had Holly Whyte, who got Jane Jacobs started, Daniel Bell, David Riesman, Galbraith. This is what we’re missing; this is a gap,” Florida says. “Now you have David Brooks as your sociologist, and Al Franken and Michael Moore as your political scientists. Where is the serious public intellectualism of a previous era? It’s the failure of social science to be relevant enough to do it.”

Here is what I take away from this: this writer is worried that Brooks (and other New Journalists) are influencing public opinion and possibly public policy more through impressionistic writing than facts and correctly interpreting data.

This could make for an interesting discussion involving things like the role of columnists and opinion-makers (facts or zeitgeists?), why social scientists and sociologists aren’t seen as public intellectuals, and who should guide public policy anyway. It is interesting to note that the American Sociological Association (ASA) gave David Brooks the Excellence in Reporting of Social Issues Award in 2011. I assume the ASA didn’t just give the award because Brooks discusses sociological research or is of the same political/social persuasion as sociologists.

By the way, having read a lot of David Brooks and Tom Wolfe, I wonder how many commentators would suggest these two are engaging in similar techniques.

Demographic change: more minority birth than whites

A number of news outlets reported last week on another marker of demographic change in America: there are now more minority babies born than white babies.

“This is an important landmark,” said Roderick Harrison, a former chief of racial statistics at the Census Bureau who is now a sociologist at Howard University. “This generation is growing up much more accustomed to diversity than its elders.”…

As a whole, the nation’s minority population continues to rise, following a higher-than-expected Hispanic count in the 2010 census. Minorities increased 1.9 percent to 114.1 million, or 36.6 percent of the total U.S. population, lifted by prior waves of immigration that brought in young families and boosted the number of Hispanic women in their prime childbearing years…

Minorities made up roughly 2.02 million, or 50.4 percent of U.S. births in the 12-month period ending July 2011. That compares with 37 percent in 1990…

Births actually have been declining for both whites and minorities as many women postponed having children during the economic slump. But the drop since 2008 has been larger for whites, who have a median age of 42. The number of white births fell by 11.4 percent, compared with 3.2 percent for minorities, according to Kenneth Johnson, a sociologist at the University of New Hampshire.

I think the last paragraph above is particularly interesting. The story isn’t just that there is a large minority and immigrant population that is having lots of children. Rather, whites and minorities are having fewer children but whites particularly have chosen to have fewer children. How much is this tied to more American living alone?

Of course, it will take some time for all of this to move through the generations. For example, it will be roughly two decades before you have more minorities than whites turning 18 and exercising this at the polls.

Trend of some moving back to rural areas?

A sociologist  argues that while there may be a lot of talk (and data) about adults seeking out denser communities, there is a countertrend of some adults moving to rural areas.

Tolkkinen’s experience is similar to that of many people who move from the city to the country. They love the beauty and peace and security. But they tend to have a hard time finding decent paying jobs and don’t like to drive the long distances to work, school and shopping.

Winchester posits that while young people continue to leave rural areas for the cities, there is an ongoing countertrend of people in their 30s and 40s moving back. He calls the phenomenon the “brain gain.” We’ll have more coverage of the report this afternoon, but here’s a summary of what people told us…

Interestingly, Winchester has found that people who move or return to rural areas tend to have higher incomes and be more civically engaged than longtime locals. That’s definitely true of Ann Thompson, who returned to her hometown of Milan, in western Minnesota, seven years ago after living overseas for 18 years. “When I left, I didn’t necessarily think I would come back,” she said. “I just thought I wanted to see the world.”…

Cheap housing draws a lot of people to rural Minnesota, judging by Winchester’s research and responses to our PIN query. Hoglin wrote that her husband “was missing rural life and wanted to be able to hunt and fish more often. I was definitely not missing rural life, but eventually warmed to the idea of moving back when I realized we could afford to buy an acreage, while we couldn’t afford to buy anything in the Twin Cities area.”

Most of this isn’t too surprising; people who move to rural areas find both advantages and disadvantages. I did find it interesting that those who move have higher incomes and higher levels of civic engagement: are they moving because they have the option to do so (you have to have money to move and perhaps going to a rural area is just another choice to try out for a while) and/or they are seeking out some “authentic community” they haven’t found elsewhere?

This article reminds me of a foundational concept in urban sociology: place matters. Even in a connected world where people can use the Internet to communicate and work from a distance, where one lives still matters a lot for jobs, cultural amenities, and social life.

I wish there were actual numbers in this story: how many people are actually moving back to rural areas? How many are doing it because of economic reasons (cheaper), family reasons (caring for family), or looking for something in the rural environment they can’t find elsewhere?

Chicagoans should be grateful for NATO summit in their city

I realize the NATO summit may be a hassle for several days (and the weeks of media hype) but it reinforces a point that Chicagoans often worry about: Chicago is indeed a world-class city, #7 on a recent list. This ranking, meaning that Chicago is a top ten city in the world (!), has both advantages and disadvantages. Even though this ranking may include short periods of difficulty, millions of metropolitan-area residents have benefited in the past and will benefit in the future.

Tying subways to the concentric rings of the Chicago School

Joel recently noted an academic study that suggests subway systems converge on a similar form. Whet Moser of Chicago argues that understanding subway patterns requires considering how cities grow and the concentric rings model of the Chicago School of urban sociology.

This is where I get skeptical that subways converging towards a “common mathematical space may hint at universal principles of human self-organization.” The subway systems the authors study were built within a relatively narrow band: 1863 (London) to 1995 (Shanghai). But they’re all also very old cities. Shanghai has a dense central business district, dating back to its long history as a port town; Moscow’s rings radiate out from the Kremlin and Red Square, following old fortifications; Beijing grew out from a model of urbanism that way predates Burgess and Park:

Many researchers reached consensus on urban morphology of the Old Beijing from physical composition. It is agreed that the Old Beijing was laid out exactly according to the concept of the Chinese utopia capital city in the book Kao Gong Ji, Notes on Works, written more than 2,000 years ago. The ideal city form is ‘a walled square city of nine by nine li (4.5 kilometers) with nine north/south main streets and east/west main avenues, three gates on each side, the ancestral temple on the left and an altar on the right of the palace, municipal administration buildings in front of the palace and a marketplace behind it’ (Fu, 1998; Liu, 1986).

So: who cares? If it’s just a neat little mathematical model, what’s its relevance? It’s relevant when the model becomes prescriptive, as the authors of “World Subway Networks” write:

In the case of Beijing, Seoul and Shanghai, it seems that their relative ‘youth’ is why they have not yet reached their long time limit.

Translation: since the subways were started after 1971, they haven’t fully converged on that ideal “core and branch” shape and ratio…

In short, Beijing is stuck in Park and Burgess’s concentric zones, and wants to move towards Harris and Ullman’s multiple-nuclei model. At the very least, it’s neat to see these comparatively dated theories of urbanization at the forefront of 21st century development. But the Beijing subway system may be following a multiple-nuclei model…

In other words, urban sociologists started to figure out that the concentric rings model doesn’t seem to fit all cities (though it still seems to overlay nicely on Chicago, it doesn’t fit other places like Beijing or newer Sunbelt cities in the United States). First came the multiple nuclei model in the 1940s and then a whole new paradigm, the political economy approach, started to emerge in the 1960s. The political economy prescriptive relies less on prescriptive models and instead focus on a different mechanism: whereas the Chicago School emphasizes competition for land and cities growing as people seek out cheaper land, the political economy model focuses on the profit motives of developers, politicians, and business leaders.

So if we looked at subway growth and locations in the political economy perspective, we could examine why lines and stops were built in certain places. Using two other forms of mass transportation as examples, we know that a good number of railroad and streetcar owners in the mid to late 1800s built lines to their new real estate developments. In other words, these lines were not built to service existing residents but rather to spur new development. I bet you could find some scholars who would argue that subways may sometimes be built to wealthier neighborhoods rather than poorer neighborhoods because there is more money to be made in these connections.

Builders constructing denser, more urban developments in the suburbs

USA Today reports that more builders are constructing denser suburban subdivisions:

The nation’s development patterns may be at a historic juncture as builders begin to reverse 60-year-old trends. They’re shifting from giant communities on wide-open “greenfields” to compact “infill” housing in already-developed urban settings…

“It’s the kids (ages 18 to 32), the empty nesters (Baby Boomers with no kids at home),” says Chris Leinberger, president of Smart Growth America’s LOCUS (Latin for “place”), a national coalition of real estate developers and investors who support urban developments that encourage walking over driving. “These two generations combined are more than half of the American population.”…

Most major builders have created “urban” divisions in the past five years to scout for available land in already-developed parts of cities and closer suburbs — even if it means former industrial and commercial sites or land that may require environmental cleanup…

Even traditional communities built on greenfields are transforming. In Southern California’s Inland Empire, an area where housing prices are lower and appeal to first-time buyers, Brookfield is building Edenglen in Ontario. The homes are built on smaller lots — 4,500 square feet instead of the more conventional 7,200 square feet — and priced from $200,000 to $300,000.

This phenomenon has been noted by a number of commentators in recent years though I wonder if it will last.

A few other consequences of this for suburbs:

1. How will existing suburban residents respond to dense, infill projects? I would guess that a good number of suburbanites would object to these dense projects being built near them, spoiling their neighborhoods.

2. Related to the first question about NIMBYism, how will these new developments change the character of existing suburbs? If a community is used to wide suburban streets and big lots, narrow lots and denser housing could change things.

3. This article hints at this but this could also be a product of the age of many American suburbs. Outside of the suburban fringe or exurbs, many suburbs not have at least a few decades of history and perhaps little to no open land (reaching build-out). If these suburbs want to continue to grow (boosting revenues and fees as well as prestige), infill development might be the only choice.

4. This article makes a common claim: certain generations (emerging adults and baby boomers) desire more urban kinds of housing. However, I wonder if it less about generational differences and more about the changing structure of American households. Is the increasing number of single households (which might be located more in these generations) really driving this? If so, this would be have bigger effects as the American suburbs have traditionally been communities build around family life and child-rearing.

Why promote education and reading with stars who make lots of money?

As a kid, I remember seeing posters of Michael Jordan (see here) and other star athletes promoting reading. While watching NBA playoff games currently, you can see plenty of NBA Cares advertisements with NBA stars talking about the importance of school. But, amidst seeing several stories that 13-year NBA player Shareef Abdur-Rahim went back to UC-Berkeley to finish his undergraduate degree in sociology, why do these campaigns feature athletic stars and not feature athletes who thought they had a chance to be a star but then realized they needed their academic degree for the rest of their lives? For example, such campaigns could feature a college star who tried to make it in the pros but had a short career, didn’t make much money or got injured early on, and then realized that he needed his academic degree to work the rest of his adult life. Or going further, perhaps non-athletes with decent adult lives could promote the value of a degree. Or athletes could talk about or promote the valuable contributions to society made by people with high school and college degrees. Either way, the star who makes a lot of money, a dream a lot of kids hold but few can attain, doesn’t end up as the primary spokesperson for education.

(I assume that these reading and education campaigns have some data or studies that show using celebrities is the best way to reach children. However, perhaps this strategy of using celebrities doesn’t work, just as using celebrities to promote organ donations isn’t the only factor that increases donation rates. See the book Last Book Gifts.)