A philosopher argues social science research is not that reliable and therefore should have a limited impact on public policy:

Without a strong track record of experiments leading to successful predictions, there is seldom a basis for taking social scientific results as definitive.  Jim Manzi, in his recent book, “Uncontrolled,” offers a careful and informed survey of the problems of research in the social sciences and concludes that “nonexperimental social science is not capable of making useful, reliable and nonobvious predictions for the effects of most proposed policy interventions.”

Even if social science were able to greatly increase their use of randomized controlled experiments, Manzi’s judgment is that “it will not be able to adjudicate most policy debates.” Because of the many interrelated causes at work in social systems, many questions are simply “impervious to experimentation.”   But even when we can get reliable experimental results, the causal complexity restricts us to “extremely conditional, statistical statements,” which severely limit the range of cases to which the results apply.

My conclusion is not that our policy discussions should simply ignore social scientific research.  We should, as Manzi himself proposes, find ways of injecting more experimental data into government decisions.  But above all, we need to develop a much better sense of the severely limited reliability of social scientific results.   Media reports of research should pay far more attention to these limitations, and scientists reporting the results need to emphasize what they don’t show as much as what they do.

Given the limited predictive success and the lack of consensus in social sciences, their conclusions can seldom be primary guides to setting policy.  At best, they can supplement the general knowledge, practical experience, good sense and critical intelligence that we can only hope our political leaders will have.

Several quick thoughts:

1. There seems to be some misunderstanding about the differences between the social and natural sciences. The social sciences don’t have laws in the same sense that the natural sciences do. People don’t operate like planets (to pick up on one of the examples). Social behaviors change over time in response to changing conditions and this makes study more difficult.

2. There is a heavy emphasis in this article on experiments. However, these are more difficult to conduct in the social realm: it is hard to control for all sorts of possible influential factors, have a sizable enough N to make generalizations, and experiments in the “harder sciences” like medicine have some of their own issues (see this critique of medical studies).

3. Saying the social sciences have some or a little predictive ability is different than saying they have none. Having some knowledge of social life is better than none when crafting policy, right?

4. Leaders should have “the general knowledge, practical experience, good sense and critical intelligence” to be able to make good decisions. Are these qualities simply individualistic or could social science help inform and create these abilities?

5. While there are limitations to doing social science research, there are also ways that researchers can increase the reliability and validity of studies. These techniques are not inconsequential; there are big differences between good research methods and bad research methods in what kind of data they produce. There is a need within social science to think about “big science” more often rather than pursuing smaller, limited studies but these studies than can speak to broader questions typically require more data and analysis which in turn requires more resources and time.