Tag Archives: statistical significance

Lancet editor suggests “much of the scientific literature, perhaps half, may be simply untrue”

Posted on by

The editor of The Lancet quickly summarizes several major issues regarding scientific studies:

The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”. The Academy of Medical Sciences, Medical Research Council, and Biotechnology and Biological Sciences Research Council have now put their reputational weight behind an investigation into these questionable research practices. The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data. Journal editors deserve their fair share of criticism too. We aid and abet the worst behaviours. Our acquiescence to the impact factor fuels an unhealthy competition to win a place in a select few journals. Our love of “significance” pollutes the literature
with many a statistical fairy-tale. We reject important confirmations. Journals are not the only miscreants. Universities are in a perpetual struggle for money and talent, endpoints that foster reductive metrics, such as high-impact publication. National assessment procedures, such as the Research Excellence Framework, incentivise bad practices. And individual scientists, including their most senior leaders, do little to alter a research culture that occasionally veers close to misconduct.

He goes to suggest some solutions such as different incentives, data review before publication, and a higher bar for statistical significance. Are there also some basic questions here about methodology such as whether randomized controlled experiments are the best way to go, particularly if the N is small? Dr. John Ioaniddis has argued for more rigorous methods in medical research, suggesting trials need to compare a new treatment to an existing treatment rather than a new option to a placebo. Perhaps we also need more metastudies that look across various studies to summarize findings rather than relying on a single study or a small group of studies to validate a finding.

At the least, this is a public relations issue for the natural and social sciences. The public tends to trust science but an increasing number of studies that are later retracted amongst breathless pronouncements of new findings will not go over well. Beyond the optics, this gets at a basic question for scientists: are we/they truly interested in finding reality? What is this scientific work intended to do anyway?

The American cities with the highest percentage of households without a car

Posted on by

As part of a look at the connection between education levels and car ownership, Derek Thompson includes this information about which American cities have lower rates of car ownership:

Here are the non-car household rates in 30 large U.S. cities (the national average is in RED):

Source: Michael Sivak, University of Michigan

What do NYC, DC, Boston, and Philadelphia have in common? For one, they’re old, crowded cities with good (okay, decent) public transit. “The five cities with the highest proportions of households without a vehicle were all among the top five cities in a recent ranking of the quality of public transportation,” Michael Sivak, director of Sustainable Worldwide Transportation at Michigan, told WSJ.

That might be the most important, variable, but it wasn’t the first thing this graph reminds me of. When I see New York, D.C., Boston, Philadelphia, San Francisco, the first thing I think is: These are all the classic, even cliche, magnets for elite college graduates. 

So I compared the cities’ non-car ownership rates to their share of bachelor’s-degree holders. And it turns out there is a statistically significant relationship between being college-dense and car-light.

Then follows a correlation chart – but no number or measure of the significance of the relationship! If one is going to claim a statistically significant relationship, more information needs to be provided like the correlation coefficient and the significance level.

That said, larger Sunbelt cities don’t come out well, nor do smaller Northern or Midwestern cities. All together, these cities are more likely to have sprawl and not have the kind of dense downtowns like Manhattan or the Loop that supports a lot of workers traveling to a single area each day. There was less historical incentive in these communities to build mass transit (outside of commuter rail) and such services, particularly subways or light rail, are quite expensive to build today in more sprawling conditions.