Tag Archives: generalizability

The educational level of immigrants in America

Posted on by

A new report suggests that there are more immigrants with college degrees than immigrants without high school diplomas:

“There’s more high-skilled (immigrants) than people believe,” said Audrey Singer, senior fellow with the Metropolitan Policy Program at the Brookings Institution and co-author of the report, which contends that the economic contribution of immigrants has been overshadowed by the rancorous debate over illegal immigration.

Singer and Matthew Hall, a sociologist at the University of Illinois-Chicago, analyzed census data for the nation’s 100 largest metropolitan areas and found that 30 percent of working-age immigrants had at least a bachelor’s degree, compared with 28 percent who lack a high school diploma.

The article suggests that the report is intended to influence the national immigration debate, presumably by suggesting that many immigrants are an asset to the country.

But it would be helpful here to compare these figures for immigrants to the statistics for American adults overall to know whether these figures are impressive or not. Here are the 2010 educational attainment figures for Americans 18 and older of all races: 27.28% have a bachelor’s degree or higher while 13.71% have less than a high school degree. It looks like the figures for immigrants are more polarized compared to the general population with a higher percentage, about 2-3% more, having a college degree while a much higher percentage, about double, having less than a high school diploma. (Figures for Americans 25 and older change a little: 29.93% have a college degree or greater while 12.86% have less than a high school degree.)

The value, then, in the figures about immigrants are probably in the field of public perceptions, particularly the statistic of immigrants with a college degree which matches up well with comparisons to Americans 18+ and 25+ years old.

(The article doesn’t address this and I don’t know if the report does either: does it matter that the figures for immigrants are drawn from the 100 largest metropolitan areas? Would the figures be different if looking at all immigrants?)

Poll figures on how the Rapture would have affected the Republican presidential field

Posted on by

Even as the news cycle winds down on Harold Camping and his prediction about the Rapture, Public Policy Polling (PPP) digs through some data to determine how the Rapture would have affected the field of Republican presidential candidates:

First off- no one really believed the Rapture was going to happen last weekend, or at least they won’t admit it. Just 2% of voters say they thought that was coming on Saturday to 98% who say they did not. It’s really close to impossible to ask a question on a poll that only 2% of people say yes to. A national poll we did in September 2009 found that 10% of voters thought Barack Obama was the Anti-Christ, or at least said they thought so. That 2% number is remarkably low.

11% of voters though think the Rapture will occur in their lifetimes, even if it didn’t happen last weekend. 66% think it will not happen and 23% are unsure. If the true believers who think the Rapture will happen in their lifetime are correct- and they’re the ones who had the strongest enough faith to get taken up into heaven- then that’s going to be worth a 2-5 point boost to Obama’s reelection prospects. That’s because while only 6% of independents and 10% of Democrats think the Rapture will happen during their lifetime, 16% of Republicans do. We always talk about demographic change helping Democrats with the rise of the Hispanic vote, but if the Rapture occurs it would be an even more immediate boost to Democratic electoral prospects.

Obama’s lead over Romney is 7 points with all voters, but if you take out the ones who think the Rapture will occur in their lifetime his advantage increases to 9 points. That’s because the Rapture voters support Romney by a 49-35 margin. Against Gingrich Obama’s 14 point lead overall becomes a 17 point one if you take out take the ‘Rapturers’ because they support Gingrich 50-37. And Obama’s 17 point lead over Palin becomes a 22 point spread without those voters because they support Palin 54-37.

Palin is the only person we tested on this poll who is actually popular with people who think the Rapture is going to happen. She has a 53/38 favorability with them, compared to 33/41 for Romney, 26/48 for Gingrich, and a 31/58 approval for Obama. Palin’s problem is that her favorability with everyone who doesn’t think the Rapture will happen is 27/66.

What a great way to combine two of the media’s recent fascinations. I would guess PPP put this poll together solely to take advantage of this news cycle. Should we conclude that Democrats should have wished the Rapture to actually happen to improve their political chances?

Of course, all of this data should be taken with a grain of salt as only 2% of the voters believed the Rapture was going to happen this past weekend and 11% believe it will happen in their lifetimes. These small numbers are out of a total sample of 600 people, meaning that about 12 people thought the Rapture would happen on Saturday and about 66 thought it would happen while they are alive. And this is all with a margin of error of plus or minus 4 percent, suggesting all of these numbers could be really, really small and not generalizable.

Do polls/surveys like these help contribute to giving all polls/surveys a bad reputation?

Getting better data on how students use laptops in class: spy on them

Posted on by

Professors like to talk about how students use laptops in the classroom. Two recent studies shed some new light on this issue and they are unique in how they obtained the data: they spied on students.

Still, there is one notable consistency that spans the literature on laptops in class: most researchers obtained their data by surveying students and professors.

The authors of two recent studies of laptops and classroom learning decided that relying on student and professor testimony would not do. They decided instead to spy on students.

In one study, a St. John’s University law professor hired research assistants to peek over students’ shoulders from the back of the lecture hall. In the other, a pair of University of Vermont business professors used computer spyware to monitor their students’ browsing activities during lectures.

The authors of both papers acknowledged that their respective studies had plenty of flaws (including possibly understating the extent of non-class use). But they also suggested that neither sweeping bans nor unalloyed permissions reflect the nuances of how laptops affect student behavior in class. And by contrasting data collected through surveys with data obtained through more sophisticated means, the Vermont professors also show why professors should be skeptical of previous studies that rely on self-reporting from students — which is to say, most of them.

While these studies might be useful for dealing with the growing use of laptops in classrooms, discussing the data itself would be interesting. A few questions come to mind:

1. What discussions took place with an IRB? It seems that this might have been a problem in the study using spyware on student computers and this was reflected in the generalizability of the data with just 46% of students agreeing to have the spyware on their computer. The other study also could run into issues if students were identifiable. (Just a thought: could a professor insist on spyware being on student computers if the students insisted on having a laptop in class?)

2. These studies get at the disparities between self-reported data and other forms of data collection. I would guess that students would underestimate their distractable laptop use on self-reported surveys because they would suspect that this is the answer that they should give (social desirability bias). But it could also reveal things about how cognizant computer/Internet users are about how many windows and applications they actually cycle through.

3. Both of these studies are on a relatively small scale: one had 45 students, the other had a little more than 1,000 but the data was “less precise” since it involved TAs sitting in the back monitoring students. Expanding the Vermont study and linking laptop use to outcomes on a larger scale is even better: move beyond just talking about the classroom experience and look at its impact on learning outcomes. Why doesn’t someone do this on a larger scale and in multiple settings? Would it be too difficult to get past some of the IRB issues?

In looking at the comments about this story, it seems like having better data on this topic would go a long ways to moving the discussion beyond anecdotal evidence.

National Association of Home Builders survey on homes in 2015: smaller, more green

Posted on by

The National Association of Home Builders (NAHB) recently published findings of a survey about what “builders, designers, architects, manufacturers, and marketing specialists” think homes will be like in 2015. Two results from this survey were reported elsewhere:

The McMansions of the boom era are quickly losing their style.The NAHB reports that the builders they “surveyed expect homes to average 2,152 square feet in 2015, 10 percent smaller than the average size of single-family homes started in the first three quarters of 2010. To save on square footage, the living room is high on the endangered list – 52 percent of builders expect it to be merged with other spaces in the home by 2015 and 30 percent said it will vanish entirely.”

Also a heavy influence on the housing front are green and eco-friendly features. The NAHB reports that “in addition to floor plan changes, 68 percent of builders surveyed say that homes in 2015 will also include more green features and technology, including low-E windows; engineered wood beams, joists or tresses; water-efficient features such as dual-flush toilets or low-flow faucets; and an Energy Star rating for the whole house.”

These two changes by 2015 were the leaders by far: 74% said smaller single-family homes were most probable or probable and 68% said it was most probable or probable that “green” features would increase in homes. This news is not too surprising: the square footage of the average new American home dropped recently and more eco-friendly homes are on the way (read about LEED certified homes here). What is interesting is that these conclusions are from members of the home building industry who likely are responding to what they think the market desires.

(Going back to the original NAHB report, something else caught my eye. Here is a short description of the methodology behind this survey:

NAHB’s The New Home in 2015 survey was sent electronically to 3,019 builders, designers, architects, manufacturers, and marketing specialists. The sample was stratified by region of the country (to be proportional to housing starts in each of the four Census regions) and, among builders, by their number of units started.

A total of 238 responses were received, of which 30 percent came from single-family builders, 19 percent from architects, 26 percent from designers, 7 percent from manufacturers, and 18 percent from “other” building industry professionals.

On one hand, the stratification of the survey is good to try to get results proportional to builders and areas of the country where building starts are taking place. On the other hand, the response rate to this electronic survey is 7.9%. With such a low response rate, how do we know that these findings are representative of the home building industry at large?)

Study about drunk fans has a limited sample

Posted on by

A recently released study suggests that 8 percent of fans leave sporting events drunk. This may be an interesting finding – but the newspaper description of the sample suggests there may be issues:

University of Minnesota researchers tested the blood alcohol content of 362 people to see how much folks drink when they go to professional baseball and football games. In their study, released Tuesday, they determined that 40 percent of the participants had some alcohol in their system and 8 percent were drunk, meaning their blood alcohol content was .08 or higher.

“Given the number of attendees at these sporting events, we can be talking about thousands of people leaving a professional sporting event who are legally intoxicated,” lead author Darin Erickson said. The study did not address what percentage, if any, of those fans intended to drive.

To collect the data, research staff waited outside 13 Major League Baseball and three National Football League games and randomly approached fans as they left. Those who consented took a breath test and answered questions about when, where and how much they drank on game day.

So the researchers waited outside 16 sporting events. Across these 16 events, the researchers performed voluntary tests on 362 people. This averages out to 22.625 fans per event.

Let’s say the events average at least 30,000 fans – not an unreasonable expectation for MLB and NFL games. If they tested about 23 fans at each event, that is less than 1 percent of each fans at each game. How could these findings be considered generalizable? First, you would need to test more fans. Second, could there be something different about the fans who were willing to volunteer for this test after a game?

Another report on this study bumps the sample number up a bit to 382 people. This doesn’t change the averages too much. Also, this may be the first study to examine the particular phenomenon of drinking at sporting events. However, the sample still seems to be too small even as the research study is going to be published in Alcoholism: Clinical & Experimental Research.

What may look like a decent survey is lacking generalizability, military officer edition

Posted on by

The latest issue of Atlantic has an interesting article discussing why a number of US military officers are leaving the military. The argument: the military is too bureaucratic and doesn’t practice meritocracy so the brightest and more entrepreneurial officers leave for other fields.

All of this is interesting but I was struck by the data used for the article. Here is how the author describes the surveys he conducted and draws conclusions from:

In a recent survey I conducted of 250 West Point graduates (sent to the classes of 1989, 1991, 1995, 2000, 2001, and 2004), an astonishing 93 percent believed that half or more of “the best officers leave the military early rather than serving a full career.” By design, I left the definitions of best and early up to the respondents. I conducted the survey from late August to mid-September, reaching graduates through their class scribes (who manage e-mail lists for periodic newsletters). This ensured that the sample included veterans as well as active-duty officers. Among active- duty respondents, 82 percent believed that half or more of the best are leaving. Only 30 percent of the full panel agreed that the military personnel system “does a good job promoting the right officers to General,” and a mere 7 percent agreed that it “does a good job retaining the best leaders.”

This sort of paragraph is very helpful and is toward the front of the story. And the numbers look overwhelming, particularly the first cited figure about 93% believing the best officers leave early.

But there is an issue here: the generalizability of this data. The article suggests surveys were conducted with 250 officers spread across six graduating classes (presumably to help control for time effects). But does this represent West Point graduates on the whole? Does this even represent each graduating class? If one looks at the class page for the graduating class of 2004, there were almost 1,200 entering students. Even if a decent amount leave before graduating, this is a lot more than the 40 or so that would have been surveyed if we had equal representation out of the six graduating classes (250 total surveys divided by six graduate classes).

This does not necessarily mean that these survey results and their interpretation are necessarily wrong. But it should cast doubt: does this survey really speak for all West Point graduates or even more broadly, military officers as a whole? While conducting some sort of survey is better than simply working with anecdotes one hears from officers veterans, this survey could still be improved so that the results could be generalized to all officers. We need a larger N of officers to survey in order to have results that we could really trust.

Another possible consequence of the foreclosure crises: a lack of trust of financial institutions

Posted on by

In recent decades, a number of sociologists have written about a necessary feature of human interaction: trust between the individuals or groups involved. Two sociologists discuss this in the Huffington Post:

While also feeling shame and embarrassment, even personal failure, for having allowed themselves to be taken in, these families are also aware of the exploitation they have experienced at the hands of their “trusted” financial advisors. That mistrust threatens the recovery some believe has begun in recent months.

As the British sociologist Anthony Giddens has noted, in complex societies where each individual cannot become expert in all the institutional contexts in which they must operate, trust is essential for people to negotiate the various realms, including financial institutions, in which they operate. People must feel secure in the trust networks they establish in order to survive and prosper, and for society itself to advance.

In a series of in-depth interviews nationwide with 22 adults who are at risk of foreclosure (they were either behind in their mortgage payments at some point in the past two years or, in two instances, had already lost their homes due to foreclosure) all respondents expressed both anger and personal responsibility. The interviews lasted between 30 and 90 minutes. In no question with any respondent was the word “trust” used. But in every case but one, the respondents explicitly referred to the mistrust they now have for anyone associated with the mortgage lending industry in particular or financial services generally.

From this short excerpt, it is hard to get an idea of how representative these 22 respondents are and how we can know whether their opinions reflect those of Americans at large. But if this is generalizable information, it suggests it could take a long for customers to approach the mortgage industry in the same way. At the same time, many Americans don’t really have many other options when shopping for a home: a mortgage is a necessity. So how could the mortgage industry once again gain the trust of consumers – special programs, special efforts, more government regulation?

According to the postscript at the end of the article, a longer argument from these two sociologists will be published soon in Social Science Quarterly.

WEIRD (Western, education, industrialized, rich, democratic) people may indeed be weird

Posted on by

A new article in Brain and Behavioral Sciences makes a thought-provoking cross-cultural conclusion about WEIRD people:

The article, titled “The weirdest people in the world?”, appears in the current issue of the journal Brain and Behavioral Sciences. Dr. Henrich and co-authors Steven Heine and Ara Norenzayan argue that life-long members of societies that are Western, educated, industrialized, rich, democratic — people who are WEIRD — see the world in ways that are alien from the rest of the human family. The UBC trio have come to the controversial conclusion that, say, the Machiguenga are not psychological outliers among humanity. We are…

WEIRD people, the UBC researchers argue, have unusual ideas of fairness, are more individualistic and less conformist than other people. In many of these respects, Americans are the most “extreme” Westerners, especially young ones. And educated Americans are even more extremely WEIRD than uneducated ones…

One of the consequences of this argument that is pointed out by the authors is that WEIRD people are then a bad population for studies and experiments because the results may not be generalizable.

I wonder how average Westerners and Americans in particular would react after reading this argument. Perhaps it might fit in with some of the ideas regarding “American exceptionalism” – though whether this is good or bad could be debated.
Regardless, if other researchers agree with these conclusions, it suggests that social science studies about humanity need to be expanded across the globe. The era of the undergraduate research subject might then be over.