Non-fiction books can have limited fact-checking, no peer review

An example of a significant misinterpretation of survey data in a recent book provides a reminder of about reading “facts”:

There are a few major lessons here. The first is that books are not subject to peer review, and in the typical case not even subject to fact-checking by the publishers — often they put responsibility for fact-checking on the authors, who may vary in how thoroughly they conduct such fact-checks and in whether they have the expertise to notice errors in interpreting studies, like Wolf’s or Dolan’s.

The second, Kimbrough told me, is that in many respects we got lucky in the Dolan case. Dolan was using publicly available data, which meant that when Kimbrough doubted his claims, he could look up the original data himself and check Dolan’s work. “It’s good this work was done using public data,” Kimbrough told me, “so I’m able to go pull the data and look into it and see, ‘Oh, this is clearly wrong.’”…

Book-publishing culture similarly needs to change to address that first problem. Books often go to print with less fact-checking than an average Vox article, and at hundreds of pages long, that almost always means several errors. The recent high-profile cases where these errors have been serious, embarrassing, and highly public might create enough pressure to finally change that.

In the meantime, don’t trust shocking claims with a single source, even if they’re from a well-regarded expert. It’s all too easy to misread a study, and all too easy for those errors to make it all the way to print.

These are good steps, particularly the last paragraph above: shocking or even surprising statistics are worth checking against the data or against other sources to verify. After all, it is not that hard for a mutant statistic to spread.

Unfortunately, correctly interpreting data continues to get pushed down the chain to readers and consumers. When I read articles or books in 2019, I need to be fairly skeptical of what I am reading. This is hard to do with (1) the glut of information we all face (so many sources!) and (2) needing to know how to be skeptical of information. This is why it is easy to fall into filtering sources of information into camps of sources we trust versus ones we do not. At the same time, knowing how statistics and data works goes a long way in questioning information. In the main example in the story above, the interpretation issue came down to how the survey questions were asked. An average consumer of the book may have little idea to question the survey data collection process, let alone the veracity of the claim. It took an academic who works with the same dataset to question the interpretation.

To do this individual fact-checking better (and to do it better at a structural level before books are published), we need to combat innumeracy. Readers need to be able to understand data: how it is collected, how it is interpreted, and how it ends up in print or in the public arena. This usually does not require a deep knowledge of particular methods but it does require some familiarity with how data becomes data. Similarly, being cynical about all data and statistics is not the answer; readers need to know when data is good enough.

Mutant statistic: marketing, health, and 10,000 steps a day

A recent study suggests the 10,000 steps a day for better health advice may not be based in research:

I-Min Lee, a professor of epidemiology at the Harvard University T. H. Chan School of Public Health and the lead author of a new study published this week in the Journal of the American Medical Association, began looking into the step rule because she was curious about where it came from. “It turns out the original basis for this 10,000-step guideline was really a marketing strategy,” she explains. “In 1965, a Japanese company was selling pedometers, and they gave it a name that, in Japanese, means ‘the 10,000-step meter.’”

Based on conversations she’s had with Japanese researchers, Lee believes that name was chosen for the product because the character for “10,000” looks sort of like a man walking. As far as she knows, the actual health merits of that number have never been validated by research.

Scientific or not, this bit of branding ingenuity transmogrified into a pearl of wisdom that traveled around the globe over the next half century, and eventually found its way onto the wrists and into the pockets of millions of Americans. In her research, Lee put it to the test by observing the step totals and mortality rates of more than 16,000 elderly American women. The study’s results paint a more nuanced picture of the value of physical activity.

“The basic finding was that at 4,400 steps per day, these women had significantly lower mortality rates compared to the least active women,” Lee explains. If they did more, their mortality rates continued to drop, until they reached about 7,500 steps, at which point the rates leveled out. Ultimately, increasing daily physical activity by as little as 2,000 steps—less than a mile of walking—was associated with positive health outcomes for the elderly women.

This sounds like a “mutant statistic” like sociologist Joel Best describes. The study suggests the figure originally arose for marketing purposes and was less about the actual numeric quantity and more about a particular cultural reference. From there, the figure spread until it became a normal part of cultural life and organizational behavior as people and groups aimed to walk 10,000 steps. Few people likely stopped to think about whether 10,000 was an accurate figure or an empirical finding. As a marketing ploy, it seems to have worked.

This should raise larger questions about how many other publicly known figures are more fabrication than empirically based. Do these figures tend to pop up in health statistics more than in other fields? Does countering the figures with an academic study stem the tide of their usage?

 

The correct interpretation of the concept of a 500 or 1,000 year flood

A flood expert addresses five myths about floods and this includes the idea that a 500 year flood only happens once every 500 years:

Myth No. 3
A “100-year flood” is a historic, once-in-a-century disaster.

Describing floods in terms of “100-year,” “500-year” and “1,000-year” often makes people think the disaster was the most severe to occur in that time frame — as encapsulated by President Trump’s tweet calling Harvey a “once in 500 year flood!” He’s not alone. When researchers from the University of California at Berkeley surveyed residents in Stockton, Calif., about their perceived flood risk, they found that although 34 percent claimed familiarity with the term “100-year flood,” only 2.6 percent defined it correctly. The most common responses were some variation of “A major flood comes every 100 years — it’s a worst-case scenario’’ and ‘‘According to history, every 100 years or so, major flooding has occurred in the area and through documented history, they can predict or hypothesize on what to expect and plan accordingly and hopefully correct.”

In fact, the metric communicates the flood risk of a given area : A home in a 100-year flood plain has a 1 percent chance of flooding in a given year. In 2018, Ellicott City, Md., experienced its second 1,000-year flood in two years, and with Harvey, Houston faced its third 500-year flood in three years.

That risk constantly changes, because of factors such as the natural movement of rivers, the development of new parcels of land, and climate change’s influence on rainfall, snowmelt, storm surges and sea level. “Because of all the uncertainty, a flood that has a 1 percent annual risk of happening has a high water mark that is best described as a range, not a single maximum point,” according to FiveThirtyEight.

I am not surprised that the majority of respondents in the cited survey got this wrong because I have never heard it explained this way. Either way, the general idea still seems to hold: the major flooding/storm/disaster is relatively rare and the probability is low in a given year that the major problem will occur.

Of course, that does not mean that there is no risk or that residents couldn’t experience multiple occurrences within a short time period (though this is predicted to be rare). Low risk events seem to flummox people when they do actually happen. Furthermore, as noted above, conditions can change and the same storms can create more damage depending on development changes.

So if this commonly used way of discussing risk and occurrences of natural disasters is not effective, what would better communicate the idea to local residents and leaders? Would it be better to provide the percent risk of flooding each year?

Significant vs. substantive differences, urban vs. suburban snow totals edition

Meteorologist Tom Skilling discusses the difference in snowfall between urban and suburban parts of the Chicago region. In doing so, he illustrates the differences between significant and substantive significance:

Dear Tom,
Why do Chicago’s suburbs get more snow in the winter than Chicago itself?
— Matt, Palatine

Dear Matt,
I do not believe that to be the case. For example, the annual snowfall at Midway Airport is 39.3 inches (Midway being closer to the lake than O’Hare); at O’Hare International Airport, it’s 37.6 inches; at Rockford, 38.3 inches. The differences aren’t large, but they are significant nonetheless. Lake Michigan enhancement of snowfall totals and the occurrence of lake-effect snows in locations closer to the lake all argue that more snow will fall with some regularity at lakeside locations.
Please note that these are generalized statements. Individual snow events will not necessarily conform to the “more snow near the lake” phenomenon. However, averaged over a period of many years, lakeside locations receive more snow than inland locations.

Because the weather data is based on decades of data, we can be fairly confident that there is a difference in snowfall between the three locations mentioned. The location nearest the lake, Midway, receives more snow, Rockford, furthest from the lake, receives a little less snow, and O’Hare, in between though much closer to the lake than Rockford, is in the middle.

On the other hand, there is very little substantive difference between these totals. Over the course of an entire year, the spread between the averages of the three locations is only 1.7 inches total. That is not much. It is likely not noticeable to the average resident. Similarly, I can’t imagine municipalities act much differently because of less than two inches of snow spread out over a year.

This illustrates an issue that often arises in doing statistical analysis: a statistical test may show a significant difference or relationship in the population but the actual difference or relationship is hard to notice or not worth acting on. Here, the data shows real differences in snowfall across locations but the real-world effect is limited.

News story suggests 40% is “Almost Half”

A Bloomberg story looks at the rise in birth in the United States outside of marriage and has this headline:

Almost Half of U.S. Births Happen Outside Marriage, Signaling Cultural Shift

And then the story quickly gets to the data:

Forty percent of all births in the U.S. now occur outside of wedlock, up from 10 percent in 1970, according to an annual report released on Wednesday by the United Nations Population Fund (UNFPA), the largest international provider of sexual and reproductive health services. That number is even higher in the European Union.

Almost Half of U.S. Births Happen Outside Marriage, Signaling Cultural Shift

There is no doubt that this is significant trend over nearly 50 years. One expert sums this up toward the end of the story:

The traditional progression of Western life “has been reversed,” said John Santelli, a professor in population, family health and pediatrics at Columbia’s Mailman School of Public Health. “Cohabiting partners are having children before getting married. That’s a long-term trend across developing nations.”

Yet, the headline oversells the change. A move from 10% of births to 40% of births is large. But, is 40% nearly 50%? When I hear almost half, I would expect a number between 45% and 49.99%. Claiming 40% is nearly half is going a little too far.

I think the reading public would better served by either using the 40% figure or saying “Two-Fifths.” Or, perhaps the headline might speak to the 30% jump in nearly 50 years.

In the grand scheme of things, this is a minor issue. The rest of the story does a nice job presenting the data and discussing what is behind the change. But, this is a headline dominated age – you have to catch those eyes scrolling quickly on their phones – and this headline goes a bit too far.

Divorce down, marriage down, telling a full story

Recent sociological work highlights how looking past the initial findings – divorce rates are down in America – reveals a more complicated reality:

In the past 10 years, the percentage of American marriages that end in divorce has fallen, and in a new paper, the University of Maryland sociologist Philip Cohen quantified the drop-off: Between 2008 and 2016, the divorce rate declined by 18 percent overall…

The point he was making was that people with college degrees are now more likely to get married than those who have no more than a high-school education. And the key to understanding the declining divorce rate, Cherlin says, is that it is “going down some for everybody,” but “the decline has been steepest for the college graduates.”

The reason that’s the case is that college graduates tend to wait longer to get married as they focus on their career. And they tend to have the financial independence to postpone marriage until they’re more confident it will work. This has translated to lower rates of divorce: “If you’re older, you’re more mature … you probably have a better job, and those things make it less likely that you’ll get into arguments with your spouse,” Cherlin says…

Chen connects this trend to the decline of well-paying jobs for those without college degrees, which, he argues, makes it harder to form more stable relationships. Indeed, Cohen writes in his paper that marriage is “an increasingly central component of the structure of social inequality.” The state of it today is both a reflection of the opportunities unlocked by a college degree and a force that, by allowing couples to pool their incomes, itself widens economic gaps.

It would be interesting to see how many of those who might celebrate the finding that divorce rates are going down also discuss the reasons linked to financial stability, education levels, and inequality.

Take more conservative Christian churches as a possible example. Evangelical Protestants are often proudly in favor of marriage (between a man and a woman). They work hard to provide programs for families as well as classes and sermons about marriage and family life. They would generally be opposed to divorce or at least view it as less than ideal. But, having conversations about how marriage is less attainable for some Americans or the evolving idea that one needs to be financially independent before marrying might be less common. How often do topics of social class and inequality come up from the front in many congregations? Or, discussions could turn to why Americans do not make correct individual choices rather than focusing on social pressures and structures (financial independence, it is more acceptable to cohabit) that influence all Americans (including conservative Christians). Ultimately, the findings may not be that good for evangelicals: divorce is down because Americans are getting married less and cohabiting more. If they want to encourage more marriage, they would have to respond to these larger social forces at work.

If one survey option receives the most votes (18%), can the item with the least votes (2%) be declared the least favorite?

The media can have difficulty interpreting survey results. Here is one recent example involving a YouGov survey that asked about the most attractive regional accents in the United States:

Internet-based data analytics and market research firm YouGov released a study earlier this month that asked 1,216 Americans over the age of 18 about their accent preferences. The firm provided nine options, ranging from regions to well-known dialects in cities. Among other questions, YouGov asked, “Which American region/city do you think has the most attractive accent?”

The winner was clear. The Southeastern accent, bless its heart, took the winning spot, with the dialect receiving 18 percent of the vote from the study’s participants. Texas wasn’t too far behind, nabbing the second-most attractive accent at 12 percent of the vote…

The least attractive? Chicago rolls in dead last, with just 2 percent of “da” vote.

John Kass did not like the results and consulted a linguist:

I called on an expert: the eminent theoretical linguist Jerry Sadock, professor emeritus of linguistics from the University of Chicago…

“The YouGov survey that CBS based this slander on does not support the conclusion. The survey asked only what the most attractive dialect was, the winner being — get this — Texan,” Sadock wrote in an email.

“Louie Gohmert? Really? The fact that very few respondents found the Chicago accent the most attractive, does not mean that it is the least attractive,” said Sadock. “I prefer to think that would have been rated as the second most attractive accent, if the survey had asked for rankings.”

In the original YouGov survey, respondents were asked: “Which American region/city do you think has the most attractive accent?” Respondents could select one option. The Chicago accent did receive the least number of selections.

However, Sadock has a point. Respondents could only select one option. If they had the opportunity to rank them, would the Chicago accent move up as a non-favorite but still-liked accent? It could happen.

Additionally, the responses were fairly diverse across the respondents. The original “winner” Southeastern accent was only selected by 18% of those surveyed. This means that over 80% of the respondents did not select the leading response. Is it fair to call this the favorite accent of Americans when fewer than one-fifth of respondents selected it?

Communicating the nuances of survey results can be difficult. Yet, journalists and other should resist the urge to immediately identify “favorites” and “losers” in such situations where the data does not show an overwhelming favorite respondents did not have the opportunity to rate all of the possible responses.