“Most romantic ‘L’ station” analysis with faulty generalization

Among other features, Craigslist offers a “missed connections” page where people can try to identify and track down people they ran into in public life. Based on this data, Craigslist recently released a list of the “most romantic” spots in Chicago’s CTA system:

Turns out it’s Belmont. The stop on the CTA’s Red, Brown and Purple lines won the title of Chicago’s most romantic ‘L’ station in a report the Web site released Thursday. The crown for most romantic train line went to the Red Line.

The site did a four-week study this summer of more than 250 missed connections postings (read, “potential hookups”) in Chicago and ranked stations based on a scale called the Train Romance Index Score Total — or TRIST for short.

The TRIST is calculated by dividing the number of missed connections that mention a CTA station or line by the number of riders a year who use that station or line. Then, that number is multiplied by 10 to get a whole number and rounded to two decimal places.

The romantic train line is defined as having the best odds of a passenger spotting another rider across a crowded train or platform and then posting a missed connections listing to get in touch.

The data is limited (a pretty small sample) but the generalization is the biggest issue: does this really reveal what is the most romantic spot on the CTA rail system? It is probably much more indicative of who uses Craigslist (young North-siders?).

My guess is that this was simply meant to be fun and promote Craigslist. But sometimes statistics and arguments like this take on a life of their own…

Using undergraduates in research experiments

It is common for research experiments to use undergraduates as subjects: they are a convenient and often willing sample pool for researchers. These studies then draw conclusions about human behavior based on undergraduate subjects.

In Newsweek, Sharon Begley writes about a new study that suggests American undergraduates are unlike many people in the world and therefore, it is difficult to make generalizations based on them.

Three psychology researchers have done a systematic search of experiments with subjects other than American undergrads, who made up two thirds of the subjects in all U.S. psych studies. From basics such as visual perception to behaviors and beliefs about fairness, cooperation, and the self, U.S. undergrads are totally unrepresentative, Joseph Henrich of the University of British Columbia and colleagues explain in a paper in Behavioral and Brain Sciences. They share responses with subjects from societies that are also Western, educated, industrialized, rich, and democratic (WEIRD), but not with humanity at large.

One way around such issues is to replicate studies with different people groups. The article describes some of these attempts, such as with the ultimatum game where two people have to negotiate a split of $10. When done with different people, the studies produce different results, suggesting that what we might think is “human nature” is heavily culturally dependent.

Another possible outcome of this study is that researchers may continue to use undergraduates but would have to scale back on their ability to generalize about humanity as a whole.

Finally, this study is a reminder that “typical” behavior in one culture is not guaranteed to be the same in another culture. What we may think of as givens can be quite different with people who do not share our cultural assumptions and practices.