Tag Archives: proof

Journalists: stop saying scientists “proved” something in studies

Posted on by

One comment after a story about a new study on innovation in American films over time reminds journalists that scientists do not “prove” things in studies.

The front page title is “Scientist Proves…”

I’m willing to bet the scientist said no such thing. Rather it was probably more along the lines of “the data gives an indication that…”

Terms in science have pretty specific meanings that differ from our day-to-day usage. “Prove” and “theory, among others, are such terms. Indeed, science tends to avoid “prove” or “proof.” To quote another article “Proof, then, is solely the realm of logic and mathematics (and whiskey).”

[end pedantry]

To go further, using the language of proof/prove tends to relay a particular meaning to the public: the scientist has shown without a doubt and that in 100% of cases that a causal relationship exists. This is not how science, natural or social, works. We tend to say outcomes are more or less likely. There can also be relationships that are not causal – correlation without causation is a common example. Similarly, a relationship can still be true even if it doesn’t apply to all or even most cases. When teaching statistics and research methods, I try to remind my students of this. Early on, I suggest we are into “proving” things but rather looking for relationships between things using methods, quantitative or qualitative, that still have some measure of error built-in. If we can’t have 100% proof, that doesn’t mean science is dead – it just means that done correctly, we can be more confident about our observations.

See an earlier post regarding how Internet commentors often fall into similar traps when responding to scientific studies.

 

Science more about consensus than proven facts

Posted on by

A new book titled The Half-Life of Facts looks at how science is more about consensus than canon. A book review in the Wall Street Journal summarizes the argument:

Knowledge, then, is less a canon than a consensus in a state of constant disruption. Part of the disruption has to do with error and its correction, but another part with simple newness—outright discoveries or new modes of classification and analysis, often enabled by technology. A single chapter in “The Half-Life of Facts” looking at the velocity of knowledge growth starts with the author’s first long computer download—a document containing Plato’s “Republic”—journeys through the rapid rise of the “@” symbol, introduces Moore’s Law describing the growth rate of computing power, and discusses the relevance of Clayton Christensen’s theory of disruptive innovation. Mr. Arbesman illustrates the speed of technological advancement with examples ranging from the magnetic properties of iron—it has become twice as magnetic every five years as purification techniques have improved—to the average distance of daily travel in France, which has exponentially increased over the past two centuries.

To cover so much ground in a scant 200 pages, Mr. Arbesman inevitably sacrifices detail and resolution. And to persuade us that facts change in mathematically predictable ways, he seems to overstate the predictive power of mathematical extrapolation. Still, he does show us convincingly that knowledge changes and that scientific facts are rarely as solid as they appear…

More commonly, however, changes in scientific facts reflect the way that science is done. Mr. Arbesman describes the “Decline Effect”—the tendency of an original scientific publication to present results that seem far more compelling than those of later studies. Such a tendency has been documented in the medical literature over the past decade by John Ioannidis, a researcher at Stanford, in areas as diverse as HIV therapy, angioplasty and stroke treatment. The cause of the decline may well be a potent combination of random chance (generating an excessively impressive result) and publication bias (leading positive results to get preferentially published)…

Science, Mr. Arbesman observes, is a “terribly human endeavor.” Knowledge grows but carries with it uncertainty and error; today’s scientific doctrine may become tomorrow’s cautionary tale. What is to be done? The right response, according to Mr. Arbesman, is to embrace change rather than fight it. “Far better than learning facts is learning how to adapt to changing facts,” he says. “Stop memorizing things . . . memories can be outsourced to the cloud.” In other words: In a world of information flux, it isn’t what you know that counts—it is how efficiently you can refresh.

To add to the conclusion of this review as cited above, it is less about the specific content of the scientific facts and more about the scientific method one uses to arrive at scientific conclusions. There is a reason the scientific process is taught starting in grade school: the process is supposed to help observers get around their own biases and truly observe reality in a reliable and valid way. Of course, whether our bias can actually be eliminated and how we go about observing both matter for our results but it is the process itself that remains intact.

This also gets to an issue some colleagues and I have noticed where college students talk about “proving” things about the world (natural or social). The language of “proof” implies that data collection and analysis can yield unchanging facts which cannot be disputed. But, as this book points out, this is not how science works. When a researcher finds something interesting, they report on their finding and then others go about retesting the findings or applying the findings to new areas. Over time, knowledge accumulates. To put it in the terms of this review, a consensus is eventually reached. But, new information can counteract this consensus and the paradigm building process starts over again (a la Thomas Kuhn in The Structure of Scientific Revolutions). This doesn’t mean science can’t tell us anything but it does mean that the theories and findings of science can change over time (and here is another interesting discussion point: what exactly is a law, theory, and a finding).

In the end, science requires a longer view. As I’ve noted before, the media tends to play up new scientific findings but we are better served looking at the big picture of scientific findings and waiting for a consensus to emerge.

Having to “prove” racism versus assuming that it is a common feature of American life

Posted on by

In defending some comments she made regarding white liberals and their support for President Obama, Melissa Harris-Perry looks at three common objections to conversations about race in the United States: “prove it,” “I have black friends,” and “who made you an expert?” While these are all familiar responses, the first one is a particularly sociological point that raises questions about how we view society and how this plays out in court:

The first is a common strategy of asking any person of color who identifies a racist practice or pattern to “prove” that racism is indeed the causal factor. This is typically demanded by those who are certain of their own purity of racial motivation. The implication is if one cannot produce irrefutable evidence of clear, blatant and intentional bias, then racism must be banned as a possibility. But this is both silly as an intellectual claim and dangerous as a policy standard.

In a nation with the racial history of the United States I am baffled by the idea that non-racism would be the presumption and that it is racial bias which must be proved beyond reasonable doubt. More than 100 years of philosophical, psychological and sociological research that begins, at least, with the work of W.E.B. Du Bois has mapped the deeply entrenched realities of racial bias on the American consciousness. If anything, racial bias, not racial innocence is the better presumption when approaching American political decision-making. Just fifty years ago, nearly all white Democrats in the US South shifted parties rather than continuing to affiliate with the party of civil rights. No one can prove that this decision was made on the basis of racial bias, but the historical trend is so clear as to require mental gymnastics to imagine this was a choice not motivated by race.

Progressives and liberals should be particularly careful when they demand proof of intentionality rather than evidence of disparate impact in conversations about racism. Recall that initially the 1964 Civil Rights Act made “disparate impact” a sufficient evidentiary claim for racial bias. In other words, a plaintiff did not need to prove that anyone was harboring racial animus in their hearts, they just needed to show that the effects of a supposedly race neutral policy actually had a discernible, disparate impact on people of color. The doctrine of disparate impact helped to clear many discriminatory housing and employment policies off the books.

Michelle Alexander brilliantly demonstrates in The New Jim Crow, the pernicious effect of the Supreme Court moving away from disparate impact as a standard to forcing plaintiffs to demonstrate racist intention. This new standard has encouraged the explosive growth of incarceration of African-Americans, turning a blind eye to disparate impact while it demands “proof” of racial bias.

I believe we must be careful and judicious in our conversations about racism. But I also believe that those who demand proof of interpersonal intention to create a racist outcome are missing the point about how racism works. Racism is not exclusively about hooded Klansmen; it is also about the structures of bias and culture of privilege that infect the left as well.

I like how Harris-Perry flips this objection: looking at the broad sweep of American history, from its days of more overt racism to more covert racism today, why don’t we assume that racism plays a role in everyday life in this society? Can we really assume, as many seem to do, that the issues with race ended at some point, either in the Civil Rights legislation of the 1960s or in the election of minority politicians or the ending of segregationist society in the South? With plenty of indicators of racial disparity today, from online comments from young adults to incarceration rates to homeownership to wealth to residential segregation, perhaps we should we see racism as a default feature of American society until proven otherwise.

Harris-Perry hints at one reason why it is difficult for Americans to see the effects of racism: the court system moving to the burden of proof shifting to “proving” “racist intention.” Without the proverbial smoking gun, it then becomes more difficult to develop arguments just from data and patterns, even if they are overwhelming. While the recent court case involving gender discrimination at Walmart and sociologists siding with the prosecution isn’t about race, it illustrates some of these principles. The data suggests discrimination may have taken place as more women did not receive promotions or pay raises. But without “proof” that this was a deliberate Walmart policy meant to harm women, the numbers may not be enough. The same holds true with race: “statistical discrimination,” stereotypes about large groups of people, may be okay because no individual or corporation can be held directly responsible for the outcome.