Tag Archives: methodology

Changing the official poverty measure leads to 49.7 million Americans in poverty

Posted on by

If recommendations from social scientists are followed, the official US government measure of poverty will change and nearly 50 million Americans will be categorized as in poverty:

The number of poor people in America is 3 million higher than the official count, encompassing 1 in 6 residents due to out-of-pocket medical costs and work-related expenses, according to a revised census measure released Wednesday.

The new measure is aimed at providing a fuller picture of poverty, but does not replace the official government numbers. Put in place two years ago by the Obama administration, it generally is considered more reliable by social scientists because it factors in living expenses as well as the effects of government aid, such as food stamps and tax credits.

Administration officials have declined to say whether the new measure eventually could replace the official poverty formula, which is used to allocate federal dollars to states and localities and to determine eligibility for safety-net programs such as Medicaid.

Congress would have to agree to adopt the new measure, which generally would result in a higher poverty rate from year to year and thus higher government payouts for aid programs.

Some other interesting data in the story as well: Social Security reduces poverty for those over 65 years old quite a bit and food stamps reduce the number of Americans in poverty by over 5 million.

On one hand, it is hard to argue with calls for a more accurate measure of poverty. This would better reflect actual living situations and give the government a better tool for addressing the issue. On the other hand, this is quite the political football. Don’t poor Americans have plenty of electronics (a bad argument)? But, as Joseph Stiglitz notes in the article, how can one of the wealthiest countries in the world have 1 out of every 6 residents living below the poverty line?

Focus less on how all of Manhattan’s 120,000 blocks can be walked and spend more time with the sociological findings

Posted on by

A sociologist who has walked every block of Manhattan shares what he learned in a new book:

The result is his new book, The New York Nobody Knows: Walking 6,000 Miles in the City. The expansive sociological study relies on Helmreich’s on-the-ground research, culled from thousands of hours of observation and casual conversations with local residents, to help parse hot-button issues like immigration, assimilation, and gentrification. But more than that, the miles and miles clocked – he wore out nine pairs of shoes in his trek across the city – come through as a sort of extensive love letter to the frenetic energy and diversity of New York…

For non-New Yorkers, the time the book spends on the outer boroughs is a fairly obvious corrective for what Helmreich sees as the tourism-generated, Manhattan-centric view of New York. And for all its diversity – the book spends hundreds of pages on the immigrant communities of the city – New York comes off as an inextricably linked web of groups that constantly must interact, change, and adjust. “It’s almost as if you dropped a hundred towns in Nebraska into the middle of the city,” Helmreich says. But what sets New York apart, he adds, is that “there’s this duality to New York that you can be in these places, but you can also be in the city.” Even those who live in more isolated pockets, such as the waterfront community of Edgewater Park in the Bronx, have a sense of connectedness…

By necessity, given the size of the city, Helmreich calls his book no more than a much-needed “introductory work” to the diversity of New York City. His method is, in some ways, a throw back to a much earlier form of social criticism, when walking was curiously in vogue for the self-styled intellectuals and elites of 19th century Europe. Think of Charles Dickens’s night walks through London or the well-dressed flâneurs of Paris. And it’s one that anyone can learn from. “If I accomplish anything besides sociology,” Helmreich says, “it’s to encourage people to walk through what I call the greatest museum in the world.”

Interesting findings that could suggest how disparate communities within a larger community understand their place in the whole. Additionally, there is a lot of potential here to detail the New York that most of its residents know, not the big money Wall Street/hedge fund world or the celebrity/glamorous crowd.

However, this article goes more for the human interest angle than the actual findings of the book. While it may be interesting to detail how a single person was able to walk the whole city, it may not mean much if they weren’t very observant or didn’t find much of interest. Rather then calling this an “epic quest,” how about thinking through what this methodology leads to compared to traditional ethnographic work that calls for spending extended time with a more limited group of people? How does this compare with other studies of American streets, such as the work of Jane Jacobs looking at places like Greenwich Village and Rittenhouse Square (Philadelphia), Elijah Anderson examining street life in poor Philadelphia neighborhoods, or Mitchell Duneier analyzing how black street vendors utilize public sidewalk space in New York City? Even as New York City gets a lot of attention, this seems like a lost opportunity to highlight how a sociologist (versus a journalist or a reality TV show or an academic from another discipline) views the street-level operation of the world’s #1 global city.

Both eHarmony.com and Match.com claim to be #1 sites for marriages. Who is right?

Posted on by

After recently seeing ads from both eharmony.com and match.com claiming they are #1 in marriages, I decided to look into their claims. First, from match.com:

Research Study Overview & Objectives
In 2009 and 2010, Match.com engaged research firm Chadwick Martin Bailey to conduct three studies to provide insights into America’s dating behavior: a survey of recently married people (“Marriage Survey”), a survey of people who have used online dating (“Online Dating Survey”),
and a survey of single people and people in new committed relationships (“General Survey”).
Key Findings Marriage Survey
• 17% of couples married in the last 3 years, or 1 in 6, met each other on an online dating site. (Table 1)
• In the last year, more than twice as many marriages occurred between people who met on an online dating site than met in bars, at clubs and other social events combined. (Table 1)
• Approximately twice as many recently married couples met on Match.com than the site that ranked second. (Table 2)

The data is from 2009-2010. And from eHarmony.com:

SANTA MONICA, Calif. – June 3, 2013 – New research data released today, “Marital Satisfaction and Breakups Differ Across Online and Offline Meeting Venues” published in Proceedings of the National Academy of Sciences (PNAS) shows eHarmony ranks first in creating more online marriages than any other online site.* The study also ranks eHarmony first in its measures of marital satisfaction.* Data also shows eHarmony has the lowest rates of divorce and separation than couples who met through all other online and offline meeting places.

eHarmony Ranked #1 for Number of Marriages Created by an Online Dating Site

The largest number of marriages surveyed who met via online dating met on eHarmony (25.04%)

eHarmony Ranked #1 for Marital Satisfaction by an Online Dating Site

The happiest couples meeting through any means met on eHarmony (mean = 5.86)…

*John T. Cacioppo, Stephanie Cacioppo, Gian C. Gonzaga, Elizabeth L. Ogburn, and Tyler J. VanderWeele (2013) Marital satisfaction and break-ups differ across on-line and off-line meeting venues. Proceedings of the National Academy of Sciences (www.pnas.org/lookup/suppl/doi:10.1073/pnas.1222447110/-/DCSupplemental)

Just based on these brief descriptions from their own websites, here is which number I would trust more: eHarmony.com. Why?

1. More recent data. Data that is a few years old is eons old in Internet time. People on dating sites today likely want to know the marriage rates today.

2. More reliable place where the study is published as well as the more scientific method. It looks like match.com hired a firm to do a study for them while the eHarmony.com data comes from a respectable academic journal.

When two companies both claim to be number one, it is not necessarily the case that one is lying or that one has to be wrong. However, it does help to compare their data sources, see what their claims are based on, and then make a decision as to which number you are more likely to believe. .

The “trick or treat index” for metropolitan areas and Chicago neighborhoods

Posted on by

In a ranking sure to bring in some Internet traffic, Zillow has put together a “trick or treat index”. The top ten cities: San Francisco, Boston, Honolulu, San Jose, Seattle, Los Angeles, Chicago, Washington D.C., Portland, and Philadelphia. You can also see the top neighborhoods for these cities. Here is what goes into the index:

Zillow takes numbers seriously, even when it comes to trick or treating. Taking the most holistic approach, the Trick-or-Treat Index is calculated using four equally weighted data variables: Zillow Home Value Index, population density, Walk Score and local crime data from Relocation Essentials. Based on these variables, the Index represents cities that will provide the most candy, with the fewest walking and safety risks.

A brief and clear explanation. The index includes four equally weighted factors: the price of homes (giving some indication of the wealth in the neighborhood), density (how many people/households are available to go to for candy), walkability (can easily walk to more candy locations), and crime rates (safety while trick-or-treating). All of this presumably adds up to identifying the best places to get candy: wealthy people are likely to give better candy, there are more households within a short walk, and it is safe. But, why don’t we get the actual ratings in these four categories for the top cities?

It is probably not worth anyone doing a serious research project on this but it would be interesting to crowdsource some data from Halloween to see how this index matches up with experiences on the ground. In other words, does this index have validity? This seems like a perfect Internet project – think GasBuddy for Halloween candy.

Journalists: stop saying scientists “proved” something in studies

Posted on by

One comment after a story about a new study on innovation in American films over time reminds journalists that scientists do not “prove” things in studies.

The front page title is “Scientist Proves…”

I’m willing to bet the scientist said no such thing. Rather it was probably more along the lines of “the data gives an indication that…”

Terms in science have pretty specific meanings that differ from our day-to-day usage. “Prove” and “theory, among others, are such terms. Indeed, science tends to avoid “prove” or “proof.” To quote another article “Proof, then, is solely the realm of logic and mathematics (and whiskey).”

[end pedantry]

To go further, using the language of proof/prove tends to relay a particular meaning to the public: the scientist has shown without a doubt and that in 100% of cases that a causal relationship exists. This is not how science, natural or social, works. We tend to say outcomes are more or less likely. There can also be relationships that are not causal – correlation without causation is a common example. Similarly, a relationship can still be true even if it doesn’t apply to all or even most cases. When teaching statistics and research methods, I try to remind my students of this. Early on, I suggest we are into “proving” things but rather looking for relationships between things using methods, quantitative or qualitative, that still have some measure of error built-in. If we can’t have 100% proof, that doesn’t mean science is dead – it just means that done correctly, we can be more confident about our observations.

See an earlier post regarding how Internet commentors often fall into similar traps when responding to scientific studies.

 

Internet commenters can’t handle science because they argue by anecdote, think studies apply to 100% of cases

Posted on by

Popular Science announced this week they are not allowing comments on their stories because “comments can be bad for science”:

But even a fractious minority wields enough power to skew a reader’s perception of a story, recent research suggests. In one study led by University of Wisconsin-Madison professor Dominique Brossard, 1,183 Americans read a fake blog post on nanotechnology and revealed in survey questions how they felt about the subject (are they wary of the benefits or supportive?). Then, through a randomly assigned condition, they read either epithet- and insult-laden comments (“If you don’t see the benefits of using nanotechnology in these kinds of products, you’re an idiot” ) or civil comments. The results, as Brossard and coauthor Dietram A. Scheufele wrote in a New York Times op-ed:

Uncivil comments not only polarized readers, but they often changed a participant’s interpretation of the news story itself.
In the civil group, those who initially did or did not support the technology — whom we identified with preliminary survey questions — continued to feel the same way after reading the comments. Those exposed to rude comments, however, ended up with a much more polarized understanding of the risks connected with the technology.
Simply including an ad hominem attack in a reader comment was enough to make study participants think the downside of the reported technology was greater than they’d previously thought.

Another, similarly designed study found that just firmly worded (but not uncivil) disagreements between commenters impacted readers’ perception of science…

A politically motivated, decades-long war on expertise has eroded the popular consensus on a wide variety of scientifically validated topics. Everything, from evolution to the origins of climate change, is mistakenly up for grabs again. Scientific certainty is just another thing for two people to “debate” on television. And because comments sections tend to be a grotesque reflection of the media culture surrounding them, the cynical work of undermining bedrock scientific doctrine is now being done beneath our own stories, within a website devoted to championing science.

In addition to rude comments and ad hominem attacks leading to changed perceptions about scientific findings, here are two common misunderstandings of how science works often found in online comments (these are also common misconceptions offline):

1. Internet conversations are ripe for argument by anecdote. This happens all the time: a study is described and then the comments are full of people saying that the study doesn’t apply to them or someone they know. Providing a single counterfactual usually says very little and scientific studies are often designed to be as generalizable as they can be. Think of jokes made about global warming: just because there is one blizzard or one cold season doesn’t necessarily invalidate a general trend upward for temperatures.

2. Argument by anecdote is related to a misconception about scientific studies: the findings do not often apply to 100% of cases. Scientific findings are probabilistic, meaning there is some room for error (this does not mean science doesn’t tell us anything – it means it is hard to measure and analyze the real world – and scientists try to limit error as much as possible). Thus, scientists tend to talk in terms of relationships being more or less likely. This tends to get lost in news stories that suggest 100% causal relationships.

In other words, in order to have online conversations about science, you have to have readers who know the basics of scientific studies. I’m not sure my two points above are necessarily taught before college but I know I cover these ideas in both Statistics and Research Methods courses.

Chicago Tribune editorial against “survey mania”

Posted on by

The Chicago Tribune takes a strong stance against “survey mania.”

Question 1: Do you find that being pelted by survey requests from your bank, cable company, doctor, insurance agent, landlord, airline, phone company — and so on — is annoying and intrusive?

Question 2: Do you ignore all online and phone requests for survey responses because, well, your brief encounter with a bank teller doesn’t really warrant a 15-minute exegesis on the endearing time you spent together?

Question 3: Don’t you wish that virtually every company in America hadn’t succumbed to survey mania at the same time, so that you’d feel, well, a little more special when each request for your precious thoughts pings into your email?

Question 4: Do you wish that companies would spend a little less on surveys and a little more on customer service staff, so that callers would not be held captive by soul-sucking, brain-scorching, automated answering systems in which a chirpy-voiced robot only grudgingly ushers your call — “which is very important to us, which is still very important to us” — to a human being?

Question 5: Do you agree that blogger Greg Reinacker laid out some reasonable guidelines for companies that send surveys to customers: “Tell me how long it’s going to take. Even better, tell me exactly how many questions there will be. … Don’t ask me the same question three different ways just to see if I’m consistent. … If you really, really want me to take the survey, offer me something. I’m a sucker for free stuff. And a drawing probably won’t do it.”

Question 6: Do you think companies should be aware that a pleasant experience — a flight, a hotel stay, a cruise — can be retroactively tainted by an exhausting survey and all those nagging email reminders that you haven’t yet filled it out?

Question 7: Do you find it irritating when a salesperson tries to game the system by reminding you over and over that only an excellent rating for his or her service will suffice … before said service has been rendered to you?

Question 8: Do you agree that there are ample opportunities to put in a good word for, say, an excellent waiter or sales clerk or customer service agent (just ask to speak to his or her supervisor!), which is much more sincere than you unhappily trudging through a long multiple-choice online questionnaire?

Question 9: Are you aware that marketing professors tell us that these surveys can be vitally important for companies to improve their service and that employee bonuses and other incentives hinge on whether you rate their service highly or not? We’re dubious, too, but just in case it’s true … would you please tell our boss how great you think this editorial is? Use all the space you need.

We get it – some people think they are being asked to do too many surveys. At the same time, this hints at some larger issues with surveys:

1. Companies and organizations would love to have more data. This reminds me of part of the genius of Facebook – people voluntarily give up their data because they get something out of it (the chance to maintain relationships with people they know).

2. Some of these problems listed above could be fixed easily. Take #7. Salespeople can be too pushy in trying to get data.

3. Some things in #5 could be done while others listed there are harder. It should be common practice to tell survey takers how long the survey might take. But, asking about a topic multiple times is often important to see if people are consistent. This is called testing the validity of the data.

4. I think more consumers would like to receive more for participating in surveys. This could be in the form of incentives, everything from free or cheaper products or special opportunities. At the least, they don’t want to feel used or to feel like just another data point.

5. Survey fatigue is a growing problem. This makes collecting data more difficult for everyone, including academic researchers.

All together, I don’t think the quest for survey data is going to end soon because customer or consumer info is so valuable for businesses and organizations. But, approaching consumers for data can be done in better or worse ways. To get good data – not just some data – organizations need to offer consumers something worthwhile in return.

US News & World Report changing up its college ranking methodology

Posted on by

US News & World Report recently announced changes to how it ranks colleges:

  • The “student selectivity” portion of the methodology will count for 12.5 percent of a college’s total, not 15 percent.
  • Within the student selectivity formula, class rank will count for 25 percent, not 40 percent. The change is attributed to the increase in the proportion of high schools that do not report class rank. SAT/ACT scores, meanwhile, rise to 65 percent from 50 percent of that score. (The rest isn’t explained but has in the past been based on colleges’ acceptance rates.)
  • Graduation rate performance (a measure that attempts to reward colleges for doing better than expected with their student body) will be applied to all colleges, not just the “national” ones at the top of the rankings.
  • “Peer assessment” — one of the most widely criticized criteria, based on a survey of presidents — will be cut from 25 to 22.5 percent of the formula for evaluating regional colleges. (One of the questions U.S. News declined to answer was whether there would be any change in the weighting for national universities.)
  • Graduation and retention rates will matter more for national universities, going from 20 percent to 22.5 percent.

This would really get interesting if these changes lead to significant shake-ups in the rankings. If some colleges move up quite a bit and, perhaps more importantly, others fall (knowing that people/institutions would feel a loss harder than an equivalent gain), there could be a lot of discussion. It would probably lead to schools that drop decrying the changes while colleges that rise would praise the new system.

It is too bad we don’t get an explanation of why these changes were made. The validity of the methodology is always in question but US News could at least try to make a case.

Is Chicago’s flag “a much bigger deal than” the flags of other big cities?

Posted on by

Here is an argument for “why Chicago’s flag is a much bigger deal than any other city’s flag“:

As reporter Elliott Ramos suggested in a 2011 post for WBEZ, Chicago’s love affair with its flag seems to have taken off in the 1990s, with an influx of young adults into the city. Michael, a kickball player featured on the Chicago Flag Tattoos website, explains why he felt compelled to have the flag permanently emblazoned on his arm: “After moving to Chicago and living here for a few years, Chicago really kind of took a place in my heart, so I thought it’d be a good thing to do.”…

Symbolism aside, the flag’s simple, bold design is the reason it caught on. On his Urbanophile blog, Aaron M. Renn wrote: “In the United States, I’d have to rate Chicago far and away #1 in the use of official civic symbols (maybe the best in the world for all I know), and also note the overall high level of design quality of these objects … If you come to Chicago, you’ll notice that the city flag is ubiquitous.”

It’s enough to make you wonder: Is this a unique local thing? How do other cities’ flags stack up against Chicago’s?

Turns out, many are bland, and a few are downright appalling. Even the good flags aren’t necessarily well-known by the people of their cities.

When the North American Vexillological Association (vexillology is the study of flags) conducted a survey in 2004 ranking the nation’s best city flags, Chicago’s flag received a stellar 9.03 out of 10 possible points. But that was only good enough to land Chicago in the No. 2 spot. No. 2? Who could possibly beat us?

There is some limited evidence here: anecdotal tales that Chicagoans seem to display the flag often and the flag is rated highly by a flag group. But, there are several issues at work here. One, Chicago’s flag might be “better” than other flags. This is more of an aethestic or design consideration. This is where you want to appeal to outside, impartial groups like the North American Vexillological Association. Second, Chicagoans might like their flag or identify with it more than residents of other cities. Perhaps it indicates that Chicagoans have some decent levels of civic pride. This could be addressed by survey research. Third, Chicagoans might display the flag more often. This is probably the easiest to quantify and observational data could provide better evidence (perhaps easier to do these days with Google Street View).

Given the evidence presented in this piece, I’m not convinced any of these three options are true…

“The United States Redrawn as Fifty States with Equal Populations” leads to interesting names in the Chicago area

Posted on by

Here is a fun map/solution/art project regarding reforming the American electoral college: have all the states have equal populations.

electorally reformed US map

Here is the methodology for the map:

The map began with an algorithm that grouped counties based on proximity, urban area, and commuting patterns. The algorithm was seeded with the fifty largest cities. After that, manual changes took into account compact shapes, equal populations, metro areas divided by state lines, and drainage basins. In certain areas, divisions are based on census tract lines.

The District of Columbia is included into the state of Washington, with the Mall, major monuments and Federal buildings set off as the seat of the federal government.

The capitals of the states are existing states capitals where possible, otherwise large or central cities have been chosen. The suggested names of the new states are taken mainly from geographical features:

  • mountain ranges or peaks, or caves – Adirondack, Allegheny, Blue Ridge, Chinati, Mammoth, Mesabi, Ozark, Pocono, Rainier, Shasta, Shenandoah and Shiprock
  • rivers – Atchafalaya, Menominee, Maumee, Nodaway, Sangamon, Scioto, Susquehanna, Trinity and Willimantic
  • historical or ecological regions – Big Thicket, Firelands and Tidewater
  • bays, capes, lakes and aquifers – Casco, Tampa Bay, Canaveral, Mendocino, Ogallala, Salt Lake and Throgs Neck
  • songs – Gary, Muskogee and Temecula
  • cities – Atlanta, Chicago, Columbia, Detroit, Houston, Los Angeles, Miami, New York, Newark, Philadelphia, Phoenix and Washington
  • plants – Tule and Yerba Buena
  • people – King and Orange

The words used for names for the name are drawn from many languages, including many American Indian languages.

Interesting naming conventions. However, I don’t understand what is going on in the Chicago area. While it makes sense to name Chicago and some of the nearby suburbs “Chicago” (though I’m guessing a number of these suburbs would not want to be lumped in with Chicago), why in the world would the new state made up of the outer regions of the current Chicago area be called Gary? I’m sure people would ask why an industrial boomtown now ghost town (it isn’t quite this bad yet this is the sort of reputation Gary has), an exemplar par excellence of the Rust Belt, would lend its name to a full state. Gary has a bad reputation (which other suburbs, particularly the wealthier ones, would not want to be associated with), it is not the largest city in the area (Milwaukee, Rockford, Joliet are larger), it is located on the eastern side of the new state so isn’t exactly central, and Joliet is the named capital.

It is also interesting to see the New York and Los Angeles metropolitan regions are also split up. However, they don’t appear to be quite split on the lines of concentric rings like the Chicago area.