Discovering fake randomness

In the midst of a story involving fake data generated for DailyKos by the polling firm, Research 2000, TechDirt summarizes how exactly it was discovered that Research 2000 was faking the data. Several statisticians approached Kos after seeing some irregularities in cross-tab (table) data. The summary and the original analysis on DailyKos are fascinating: even truly random data follows certain parameters. One takeaway: faking random data is a lot harder than it looks. Another takeaway (for me at least): statistics can be both useful and enjoyable.

The three issues as summarized on DailyKos:

Issue one: astronomically low odds that both male and female figures would both be even or odd numbers.

In one respect, however, the numbers for M and F do not differ: if one is even, so is the other, and likewise for odd. Given that the M and F results usually differ, knowing that say 43% of M were favorable (Fav) to Obama gives essentially no clue as to whether say 59% or say 60% of F would be. Thus knowing whether M Fav is even or odd tells us essentially nothing about whether F Fav would be even or odd.

Issue two: the margin between favorability and unfavorability ratings did not display enough variance. If the polls were truly working with random samples, there would be broader range of values.

What little variation there was in the difference of those cross-tab margins seemed to happen slowly over many weeks, not like the week-to-week random jitter expected for real statistics.

Issue three: the changes in favorability ratings from week to week were too random. In most polls like this that track week to week, the most common result is no change. Research 2000 results had too many changes from week to week – often small changes, a percent either way.

For each individual issue, the odds are quite low that each would arise with truly random data. Put all three together happening with the same data and the odds are even lower.

Besides issues regarding integrity of data collection (and it becomes clearer why many people harbor a distrust toward polls and statistics), this is a great example of statistical detective work. Too often, many of us see numbers and quickly trust them (or distrust them). In reality, it takes just a little work to dig deeper into figures to discover what exactly is being measured and how it is being measured. The “what” and “how” matter tremendously as they can radically alter the interpretation of the data. Citizens and journalists need some of these abilities to decipher all the numbers we encounter on a daily basis.

Blood gadgets

Many consumers don’t ask, and presumably often don’t care, how their newly purchased products came to be. Certain products have drawn attention, such as “blood diamonds” (accompanied by a preachy Hollywood film by the same name) or Nike shoes made by sweatshop laborers.

New York Times columnist Nicholas Kristof draws attention to another set of consumer goods: high-tech gadgets including cell phones. These devices often include hard to obtain minerals, such as tantalum which is found in Congo. There are some activists who are planning to bring attention to this by taking their argument to tech companies like Microsoft, Apple, and Intel:

A humorous new video taunting Apple and PC computers alike goes online this weekend on YouTube, with hopes that it will go viral. Put together by a group of Hollywood actors, it’s a spoof on the famous “I’m a Mac”/”I’m a PC” ad and suggests that both are sometimes built from conflict minerals.

“Guess we have some things in common after all,” Mac admits.

Stay tuned. A strong-enough consumer/activist push will likely lead to these companies pledging to use responsible materials.

Also: one wonders how this decades-long situation in mineral-rich Congo might inform decision-making regarding recent finds of vast amounts of valuable minerals in Afghanistan.

54 years ago: Federal interstates are born

On June 29, 1956, President Eisenhower signed the Federal Aid Highway Act of 1956. This legislation, though immediately about infrastructure, had a tremendous impact on American life. Many of the interstate highways of today were built with this money.

These roads have produced a number of changes:

-Suburbanization. People could now easily travel from suburbs to the city center. By the 1960s, many businesses were also locating headquarters along suburban highway exits.

-The American love of the car. This already existed before Federal Interstates but it was enhanced by these well-maintained roads. Now, the average American could drive farther and more safely. From this point on, money for public transportation would always be limited compared to funds for roads.

-Shipping. Many goods today are carried by trucks. Cheap roads coupled with cheap gasoline helps keep Wal-Marts and McDonald’s stocked and cheap.

-Urban renewal. A number of big city neighborhoods were bulldozed to make way for new highways. Recently, some cities have reversed these trends by removing highways and establishing parks and public spaces. Two notable and beautiful examples: the Big Dig in Boston and the Embarcadero in San Francisco.

-Aesthetics. Many of these roads are about brute efficiency: moving the largest number of people in the shortest amount of time. To many, these highways scar the landscape. But they can often take on a beauty of their own, particularly in complicated interchanges.

-Small town life all but disappeared. With the rise of suburbs and highways rerouting traffic around small communities, rural populations dwindled.

-A fast-food approach to life. Not only does food have to be obtained quickly so one can get back on the road, signs need to be larger to be legible at 65 MPH, cars need to be larger to survive the occasional highway accident, travelers need built-in DVD players to be entertained, and so on.

Prior to the signing of this act, local governments and states had begun to cobble together a highway system. The City of Chicago had been planning for a local highway system for years but did not begin construction until after World War II. Pennsylvania had a turnpike (now I-76) and Indiana, Ohio, and Illinois had started roads that would create an interstate toll road. Robert Moses had begun a system in New York City.

But this law helped build and codify a system that is still going strong today.

Winning vs. a country’s culture

Brazilian coach Carlos Caetano Bledorn “Dunga” Verri has had a successful World Cup run thus far: four matches and four pretty easy wins. For many national coaches, this would lead to general praise from the media and fans.

But not in Brazil. Dunga has been playing with a more defensive-minded system, particularly compared to the attacking-with-flair Brazilian teams of past decades. A quick description of the battle Dunga has been fighting:

Then there were the fans, who almost always favor the spectacular and revel in the nation’s tradition of breathtaking open-field play.

Brazil has always been about offense, offense, offense. It has the deepest pool of talent in which to select a team. Its players pride themselves on creativity.

Even some of the  country’s former stars, such as Carlos Alberto, captain of Brazil’s 1970 team, have blasted Dunga:

“I am not confident in this group because our national team do not play Brazilian football…I’m talking about movement and use of the ball. We have good defenders, but the midfielders: if you ask Brazilian kids, who are our midfielders, they shrug their shoulders.”

So if Brazil wins a sixth World Cup title, what then? Will the country celebrate in the same way or will it be considered a less-than-great title? Sports fans can be an interesting lot, particularly when they are used to winning.

“The Triumphant Decline of the WASP”

A NY Times opinion piece from Harvard law professor Noah Feldman makes this argument: “The decline of the Protestant elite is actually its greatest triumph.” Feldman explores the changes in the Supreme Court (the appointment of Kagan would make it 6 Catholics, 3 Jews) and Princeton (“As late as 1958, the year of the “dirty bicker” in which Jews were conspicuously excluded from its eating clubs, Princeton could fairly have been seen as a redoubt of all-male Protestant privilege).

So what changed? Feldman provides some reasons: “the anti-aristocratic ideals of the Constitution,” education was an important defining trait for WASPs so opening up universities was a big step, and the American value of fair play. The result:

Together, these social beliefs in equality undercut the impulse toward exclusive privilege that every successful group indulges on occasion. A handful of exceptions for admission to societies, clubs and colleges — trivial in and of themselves — helped break down barriers more broadly. This was not just a case of an elite looking outside itself for rejuvenation: the inclusiveness of the last 50 years has been the product of sincerely held ideals put into action.

These may be accurate reasons. But they seem to ignore the historical context: something happened in the 1960s that changed institutions like Ivy League schools and led to a very different looking Supreme Court. In that decade, the Civil Rights Movement plus an explosion in higher education for the burgeoning US population plus higher rates of immigration from non-European locales plus cultural change (rock ‘n’ roll, television, more open questioning of authority, etc.), changed, or at least began to change, the socioeconomic status of WASPs.

Measuring celebrity

Forbes has released its annual list of the 100 most powerful celebrities. See Forbes’ website for a full portal that includes profiles of some of the celebrities and the full rankings (including rankings on subcategories). Topping the list is Oprah followed by Beyonce and James Cameron.

Buried at the bottom of the story is the methodology by which Forbes developed its list (the methodology is mentioned in this reposted story at Yahoo):

The Celebrity 100 is a measure of power based on money and fame. Earnings estimates, which include income from films, television shows, endorsements, books, and other entertainment ventures, are calculated between June 2009 and June 2010. Figures were rounded off where appropriate. Additional sources include Billboard, Pollstar, Adams Media Research, The Nielsen Company, and SNL Kagan. Fame is calculated using web hits on Google, Blog Search, TV/radio mentions on LexisNexis, overall press mentions on Factiva, and the number of times a celebrity’s image appeared on the cover of 25 consumer magazines. Social rank is calculated using metrics like Facebook friends and fans as well as Twitter followers.

I would be very interested in knowing the weights applied to each of these measures and broader categories (such as social rank). Take Lady Gaga for example: she is new to the list this year, does not have the media empires like some of the others on the list (Oprah’s big money advantage comes from an involvement in a multitude of media outlets), and yet benefits from a #1 ranking in the social rankings.

After a quick glance, money appears most important here. Perhaps having money prompts more media (of all kinds) mentions. Or perhaps the media mentions help build the money which then leads to a reinforcing cycle. Regardless, just having money may be a sign that you are a true celebrity. We as Americans may like our celebrities because they host a TV show or can do amazing things with a golf ball or can direct exciting movies, but just having money seems pretty interesting in itself.

Modern careers more amenable to women?

Hanna Rosin writes in the July/August 2010 issue of The Atlantic about the rise of women in many career fields and the consequences for society. Rosin argues that in addition to women holding “a majority of the nation’s jobs,” dominating higher education, and having a majority in 13 of the 15 job categories predicted to grow the most in the next ten years, more and more jobs today seem suited to women and men have not yet adapted:

The postindustrial economy is indifferent to men’s size and strength. The attributes that are most valuable today—social intelligence, open communication, the ability to sit still and focus—are, at a minimum, not predominantly male. In fact, the opposite may be true.

The list of growing jobs is heavy on nurturing professions, in which women, ironically, seem to benefit from old stereotypes and habits.

Some of this has been more visible lately with the effects of the recent economic trouble, dubbed by some a “man-cession” or “he-pression” due to a disproportionate loss of jobs in male-dominated fields. The loss of manufacturing and manual labor jobs in the last five decades has been severe and men, unlike women, have not yet jumped on the higher education bandwagon.

Privacy is possible online?

A journalist laments the end of JournoList – but also seems to think that privacy is possible in today’s online world. Really? This is something that most 20 year old Facebook users know: if you don’t want something to become known online, don’t ever post it online. Even among groups that trust each other, as the journalists on JournoList did, the Internet is one of the least private places I can imagine.

Stories for Afghanistan

Wired writes of a new Department of Defense program to develop “interactive stories” to find common ground with Afghan residents.

The “Negotiate Across Cultures” project will kick off with a two-day workshop… “participants will use a ‘Wizard of Oz’ approach to illustrate how each approach, if implemented, would operate on selected negotiation problems within specific socio-cultural environments. In the ‘Wizard of Oz’ approach, one or two human representatives of each participant team will listen to a problem description and manually act out the operation and structure of the tool using props they bring to the event.”

Using stories in this way is not uncommon. In recent decades, governments, activists, and social scientists have successfully used media, such as radio and television shows, to teach certain messages in areas like public health.

Where Hitler is not reviled

Interesting AP story regarding a Bollywood film about Hitler titled Dear Friend Hitler. From the piece:

[In India], Hitler is not viewed as the personification of evil, but with an attitude of morally ambiguous fascination. He is seen as a management guru – akin to Machiavelli or Sun Tzu – by business students, and an object of wonder by people craving order amid the chaos of India.

A sociologist, Ashsish Nandy, gives several reasons for this:

For some readers, modern India is a country in chaos and, there is a “certain admiration” for Hitler and his extreme authoritarianism.

There is also India’s colonial inheritance when “every enemy of Britain was a friend of India and at least potentially a good person,” he says, adding that among today’s young readers “there is kind a vague sense that it’s about a person who gave a tough time to the Brits.”

Much of this is likely to look strange to Westerners where Hitler is often invoked as the “epitome of evil.” But the two reasons given by Nandy may have some merit. The British colonial legacy, often negative even more than six decades later, is still a strong cultural factor. Current “chaos” also invokes hope that any leader, in any form, might bring order.