Scientists have difficulty explaining p-values

Posted on by

Scientists regularly use p-values to evaluate their findings but apparently have difficulty explain exactly what they mean:

To be clear, everyone I spoke with at METRICS could tell me the technical definition of a p-value — the probability of getting results at least as extreme as the ones you observed, given that the null hypothesis is correct — but almost no one could translate that into something easy to understand.

It’s not their fault, said Steven Goodman, co-director of METRICS. Even after spending his “entire career” thinking about p-values, he said he could tell me the definition, “but I cannot tell you what it means, and almost nobody can.” Scientists regularly get it wrong, and so do most textbooks, he said. When Goodman speaks to large audiences of scientists, he often presents correct and incorrect definitions of the p-value, and they “very confidently” raise their hand for the wrong answer. “Almost all of them think it gives some direct information about how likely they are to be wrong, and that’s definitely not what a p-value does,” Goodman said.

We want to know if results are right, but a p-value doesn’t measure that. It can’t tell you the magnitude of an effect, the strength of the evidence or the probability that the finding was the result of chance.

So what information can you glean from a p-value? The most straightforward explanation I found came from Stuart Buck, vice president of research integrity at the Laura and John Arnold Foundation. Imagine, he said, that you have a coin that you suspect is weighted toward heads. (Your null hypothesis is then that the coin is fair.) You flip it 100 times and get more heads than tails. The p-value won’t tell you whether the coin is fair, but it will tell you the probability that you’d get at least as many heads as you did if the coin was fair. That’s it — nothing more. And that’s about as simple as I can make it, which means I’ve probably oversimplified it and will soon receive exasperated messages from statisticians telling me so.

Complicated but necessary? This can lead to fun situations when teaching statistics: students need to know enough to do the statistical work and evaluate findings (we at least need to know what to do with a calculated p-value, even if we don’t quite understand what it means) but explaining the complexity of some of these techniques wouldn’t necessarily help the learning process. In fact, the more you learn about statistics, you tend to find that the various methods and techniques have limitations even as they can help us better understand phenomena.

Solving flooding in China with “sponge cities”

Posted on by

Chinese officials are providing funds for “sponge cities” to reduce the effects of flooding:

“A sponge city is one that can hold, clean, and drain water in a natural way using an ecological approach,” says Yu, who is helping to coordinate the national project.

Traditionally, Chinese cities handled water well, Yu notes. “But in modern China, we have destroyed those natural systems of ponds, rivers, and wetlands, and replaced them with dams, levees, and tunnels, and now we are suffering from floods.”…

Reverse-engineering a city to make it more spongey requires a mental rather than physical shift, he argues. “It’s a whole new philosophy of dealing with water. It is about how we plan and design our cities in an ecological way. Not about piecemeal, manmade engineering projects. So we need to avoid this kind of trap.”

Sponge-city design could also run up against China’s centralized planning system.

It sounds like this is a major work in progress. As has been found in American cities, such as Chicago, trying to solve flooding issues after the city is a certain size is quite difficult. Are cities really willing to move residents or commercial structures to better deal with water issues? Is it only possible to make changes after a major flood convinces people? The optimal way to do this would be before the development happens as planners and others can set aside space or promote greener options.

For Thanksgiving, carve up McMansions

Posted on by

The Financial Review serves up this headline: “Architects sharpen knives to carve up McMansions.”

He says there are different ways to redeem McMansions. One could be divided into two side-by-side dwellings or even three separate townhouses. Alternatively, the division could be horizontal, with a ground-floor apartment and separate first-floor one. Fences could be knocked down to create common garden areas between dwellings.

Neustein calculates that to turn a typical seven-bedroom, three-bathroom McMansion into a split-level two-unit dwelling would cost about $350,000, or a $770 per square metre.

The procedure would involve removing the brick veneer walls and plasterboard lining, weatherproofing the newly exposed timber frame, demolishing internal walls and putting in new external walls with double-glazed windows and doors, rerouting plumbing and electricity, waterproofing decking for the new verandah space and installing a rainwater tank and sprinklers to fire-proof the timber structure.

It’s a cheaper outlay than buying a new apartment, the median price of which in Sydney was $675,000 last month, would let the existing owner stay in their home and area, and would create an asset they could rent.

This article provides a lot more details about these plans that were featured at the Chicago Architecture Biennial. Two quick thoughts:

  1. If apartments really are so expensive in the Sydney area, it seems like there is a lot of financial incentive to try something like this. Imagine a local business or institutional investor buying up some of these large McMansions and converting them into rentals with the aim of long-term profits (as opposed to the big profits made by building and selling such a big home in the first place).
  2. As the article goes on to note, it is hard to know whether people would want to rent so far out from job centers. Yet, I imagine another issue: what would the neighbors say? These large homes are probably built in neighborhoods with a number of similar homes. Renovations like these would be frowned upon as it would introduce renters (who are different kinds of people than those who own expensive large homes) and change the character of a quiet neighborhood. In the United States, such changes would have to go through municipal approval and to put it mildly, I think the neighbors would be opposed.

The effects of residential segregation on kids

Posted on by

Sociologist Carla Shedd describes some of her findings about what children in segregated neighborhoods learn about the world:

I’m looking at young people in a very similar predicament, and their vantage point is outside their window. I interviewed one kid and asked, “What do you think about police in your neighborhood?” He said, “I think they’re fair. When I look outside my window I see black gangbangers, I see people doing things they shouldn’t be doing.” Whereas the kids who move across these boundaries of race and geography, they’re saying, “Wow, the police downtown, they wave at people and give the thumbs up. They don’t do that to me. They actually go in the shops and buy stuff. These aren’t things I see in the South Shore.”

The protective piece is so key. A kid asks, “Don’t the police stop everyone?” If you shatter that and say, “No, it’s actually just in your neighborhood,” or “No, it’s people who look like you,” what does that do, if they feel like they can’t really respond against it, or change that reality?…

So when it comes to looking at someone under the guise of criminalized suspicion, younger and younger black kids are caught under that same gaze. That’s one piece where they say, “I have to learn that I can’t run and race with my friends, since I might be seen as running from the scene of a crime.” Or, “I have to be careful about how I use my body in public spaces.”

Think about all the opportunities adolescents have to figure out who they are and try different things. These kids don’t get that same freedom to do that. To try different looks, to range farther from home, to be around different types of people. That’s not a mark of their adolescent experience, and it ages them.

In my estimation, it makes them skip that step of exploration, because there are consequences to stepping outside of those boundaries, or perhaps being received in a certain way by authoritative figures in what could be free spaces, like in school or particular neighborhoods.

While the attention urban neighborhoods receive may typically focus on large or traumatic events, Shedd and a number of other scholars have revealed day to day life and its consequences.

This reminds me that one of the goals of relocation programs – like Moving to Opportunity – was to provide better opportunities for kids. Some of the research has found such moves help.

Photographing affordable housing in New York City

Posted on by

With the expense of Manhattan and a booming luxury market in NYC, one sociologist shows what affordable housing looks like:

Garbage-strewn common areas, ominous graffiti, the twitching fluorescent stairwell lighting — these are the images most often associated with public housing. Even privately owned affordable housing is often seen as something bland and tiny you settle for, not aspire to. But David Schalliol, a photographer and sociology professor in Minnesota, sees a shift toward something that goes beyond the cliché…

In New York, however, he found that there was much more to the story. “Affordable housing means so many different things in New York City,” he said, citing developments, like Co-op City in the Bronx, that helped give the city a reputation for finding innovative ways to provide decent housing to middle- and working-class families.

Mr. Schalliol surveys this landscape in a new anthology, “Affordable Housing in New York: The People, Places and Policies That Transformed a City,” from Princeton University Press…

“One of the aims of this book, this project, is not only to demonstrate the wide variety of these developments, but also the common experience within them,” he said. “But it’s where people make their homes, where they meet their friends. They don’t just come home, they’re actively producing community.”

It would be interesting to see what sort of argument Schalliol makes in this book. The photos provided with the article suggest the people in New York City’s affordable housing are just trying to live a normal life. Yet, collections of photographs can counter stereotypes and help the broader public see what affordable housing really looks like. Perhaps the images could even help people see that having affordable housing nearby will not necessarily ruin their property values and lives.

Bigger American houses limit energy efficiency

Posted on by

A new Pew report looks at how the growing size of American homes affects energy efficiency:

U.S. homes have become considerably more energy-efficient over the past four decades, according to government data. But homes also are a lot bigger than they used to be, and their growing girth wipes out nearly all the efficiency gains.

According to preliminary figures from the Department of Energy’s Office of Energy Efficiency and Renewable Energy, the average U.S. home used 101,800 British thermal units (Btu) of energy per square foot in 2012, the most recent year with available data. That’s 31% less than in 1970, after adjusting for weather effects and efficiency improvements in electricity generation…

While some homeowners do add onto their existing structures, the trend is driven largely by new construction. According to the Census Bureau, the average new single-family house completed last year was 2,657 square feet – 57% larger than four decades earlier. While the biggest new homes are being built in the South (an average of 2,711 square feet last year), home sizes have grown the most in the Northeast: a 64% increase in average new-home size over the past four decades…

What all of this means is that, after dropping sharply during the 1970s, the overall energy intensity of U.S. homes has changed little over the past three decades. Energy intensity is a metric that compares the amount of energy used against some unit of economic activity – households, in the case of the residential sector.

A logical question at the end of this is to ask what should be done in response. One line of argument would suggest Americans should cut their home size. When they build and purchase larger homes, they use more energy than they probably need to consume. (This is in addition to other arguments against building larger houses.) On the other hand, I imagine some would argue that we will continue to see gains in energy efficiency through technology and this will soon reduce energy use even in spite of larger homes. This second argument may be more appealing to many as then Americans could get even bigger homes and we get to utilize the benefits of technological progress.

Evangelicals recommend four beliefs that should identify them on surveys

Posted on by

The National Association of Evangelicals and LifeWay Research suggest evangelicals should be identified by agreeing with four beliefs:

  • The Bible is the highest authority for what I believe.
  • It is very important for me personally to encourage non-Christians to trust Jesus Christ as their Savior.
  • Jesus Christ’s death on the cross is the only sacrifice that could remove the penalty of my sin.
  • Only those who trust in Jesus Christ alone as their Savior receive God’s free gift of eternal salvation.

More on the reasons for these four:

The statements closely mirror historian David Bebbington’s classic four-point definition of evangelicalism: conversionism, activism, biblicism, and crucicentrism. But this list emphasizes belief rather than behavior, said Ed Stetzer, executive director of LifeWay Research.

“Affiliation and behavior can be measured in addition to evangelical beliefs, but this is a tool for researchers measuring the beliefs that evangelicals—as determined by the NAE—believe best define the movement,” he said.

A few quick thoughts on this:

  1. On one hand, it can be helpful for religious groups to identify what they see as unique to them. Outsiders may not pick up on these things. On the other hand, outsiders might see beliefs or other characteristics that mark evangelicals.
  2. Measuring religiosity involves a lot more than just beliefs. From later in the article:

    “Identity, belief, and behavior are three different things when it comes to being an evangelical,” McConnell said. “Some people are living out the evangelical school of thought but may not embrace the label. And the opposite is also true.”

    So this is just one piece of the puzzle. And I think sociologists (and other social scientists) have contributed quite a bit here in looking at how these particular theological views relate to other social behavior from race relations to voting to charitable activity and more.

  3. The suggestion here is that research shows the “correct” number of evangelicals identify with these four statements – identifying evangelicals in other ways seems to get to similar percentages as working with these four beliefs. Yet, I wonder how many evangelicals would name these four statements if asked what they believe. How exactly are these statements taught and passed on within evangelicalism?

Calls for a special census as Naperville continues to grow

Posted on by

There is some evidence that Naperville has grown since the 2010 Census – and a special census would be able to prove it.

This would be the first special census in Naperville since 2008 and would follow other special counts taken in 2003 and four times between 1990 and 2000 during a period of major expansion…

Some subdivisions on Naperville’s southwest side are still filling with residents and the city has approved a few new housing developments in recent years…

“If things in Springfield were normal, which they haven’t been for a long time, each additional body would have been worth $100,” Krieger said. “This year there has been a lot of discussion about reducing that amount.”

Conducting the special census could cost between $80,000 and $140,000, depending on how many census tracts the city decides to count. Officials would want to be certain a tract has seen enough new residents — based on building permits and occupancy certificates — before counting it.

It sounds like this is all about money: proof of more residents = more state dollars. Many communities could use such cash but as this story points out, there has to be enough growth to outweigh the costs of conducting the census. I don’t believe I’ve ever seen the reverse where communities conduct a census to track population loss; that would be a double cost.

Another note: population growth can translate into funding but it can also add to a community’s image. If it hasn’t already, Naperville is very close to being a built-out suburb. Yet, we generally have the idea in the United States that good communities are growing communities. A special census could suggest Naperville is still attractive.

Explaining the drop in DUI arrests and crashes in the Chicago suburbs

Posted on by

The roads in the Chicago suburbs have been safer since 2007:

DUI arrest totals last year in 79 suburbs were about half what they were in 2007, despite only a small drop in police staffing. There were 6,955 arrests last year, compared to 12,166 in 2007, according to annual state-funded surveys compiled by the Schaumburg-based Alliance Against Intoxicated Motorists.

Meanwhile, those same suburbs in six counties reported 1,555 crashes involving alcohol-impaired driving in 2007, according to Illinois Department of Transportation crash reports. By 2009, that number was down to 1,012 alcohol-impaired crashes, and it has hovered near that mark ever since, with 1,065 crashes in 2014, the reports show.

What is behind this?

“It’s the economy,” said Don Ramsell, a Wheaton-based attorney who specializes in drunken driving defenses. “It’s so obvious it’s ridiculous. Alcohol is a feature of people’s disposable income, and most people have a lot less of that these days…

Lake in the Hills Police Chief David Brey chalks up the decline in the number of arrests to “more and more people making a conscious effort to take a cab or have a designated driver.”…

Ramsell and AAIM Executive Director Rita Kreslin say lean budgets might have something to do with fewer DUI arrests. Both said police officers have told them they’re under less pressure to make DUI arrests because of the time and expense of following up in court.

Three different explanations: people have less money to spend on alcohol, drinkers have become smarter about using alternative transportation, and police departments may have been devoting less attention to this area. Getting this explanation right could be consequential as communities and police departments think about their budgets. In contrast, simply throwing out possible explanations (probably based on anecdotal evidence) may serve particular interests.

Still, good news overall for the safety of suburban roads. Now we can see whether the trend lasts and this might provide evidence for the explanations given above.

“Pollsters defend craft amid string of high-profile misses”

Posted on by

Researchers and polling organizations continue to defend their efforts:

Pollsters widely acknowledge the challenges and limitations taxing their craft. The universality of cellphones, the prevalence of the Internet and a growing reluctance among voters to respond to questions are “huge issues” confronting the field, said Ashley Koning, assistant director at Rutgers University’s Eagleton Center for Public Interest Polling…

“Not every poll,” Koning added, “is a poll worth reading.”

Scott Keeter, director of survey research at the Pew Research Center, agreed. Placing too much trust in early surveys, when few voters are paying close attention and the candidate pools are their largest, “is asking more of a poll than what it can really do.”…

Kathryn Bowman, a public opinion specialist at the American Enterprise Institute, also downplayed the importance of early primary polls, saying they have “very little predictive value at this stage of the campaign.” Still, she said, the blame is widespread, lamenting the rise of pollsters who prioritize close races to gain coverage, journalists too eager to cover those results and news consumers who flock to those types of stories.

Given the reliance on data in today’s world, particularly in political campaigns, polls are unlikely to go away. But, there will be likely be changes in the future that might include:

  1. More consumers of polls, the media and potential voters, learn what exactly polls are saying and what they are not. Since the media seems to love polls and horse races, I’m not sure much will change in that realm. But, we need great numeracy among Americans to sort through all of these numbers.
  2. Continued efforts to improve methodology when it is harder to reach people and obtain representative samples and predict who will be voting.
  3. A consolidation of efforts by researchers and poling organizations as (a) some are knocked out by a string of bad results or high-profile wrong predictions and (b) groups try to pool their resources (money, knowledge, data) to improve their accuracy. Or, perhaps (c) polling will just become a partisan effort as more objective observers realize their efforts won’t be used correctly (see #1 above).