Tag Archives: methodology

More details of unethical US medical experiments in Guatemala in the 1940s

Posted on by

Research methods courses tend to cover the same classic examples of unethical studies. With more details emerging from a government panel, the US medical experiments undertaken in Guatemala during the 1940s could join this list.

From 1946-48, the U.S. Public Health Service and the Pan American Sanitary Bureau worked with several Guatemalan government agencies to do medical research — paid for by the U.S. government — that involved deliberately exposing people to sexually transmitted diseases…

The research came up with no useful medical information, according to some experts. It was hidden for decades but came to light last year, after a Wellesley College medical historian discovered records among the papers of Dr. John Cutler, who led the experiments…

During that time, other researchers were also using people as human guinea pigs, in some cases infecting them with illnesses. Studies weren’t as regulated then, and the planning-on-the-fly feel of Cutler’s work was not unique, some experts have noted.

But panel members concluded that the Guatemala research was bad even by the standards of the time. They compared the work to a 1943 experiment by Cutler and others in which prison inmates were infected with gonorrhea in Terre Haute, Ind. The inmates were volunteers who were told what was involved in the study and gave their consent. The Guatemalan participants — or many of them — received no such explanations and did not give informed consent, the commission said.

Ugh – a study that gives both researchers and Americans a bad name. It is also a good reminder of why we need IRBs.

While the article suggests President Obama apologized to the Guatemalan president, is anything else going to be done to try to make up for this? I also wonder how this is viewed in Central America: yet more details about the intrusiveness of Americans over the last century?

(See my original post on this here.)

Measuring colleges by their service to community and country

Posted on by

Many publications want to get into the college rankings business and Washington Monthly released their own take today. The difference? They emphasize how the college gives back to society:

The Monthly’s list aims to be a corrective to the annual ranking of colleges published by U.S. News World & Report–the industry-standard roster that typically leads with well-endowed Ivy League schools that turn away the vast majority of applicants.

Instead, the Monthly ranks schools using three main categories: how many low-income students the college enrolls, how much community and national service a given college’s students engage in, and the volume of groundbreaking research the university produces (in part measured by how many undergraduates go on to get PhDs). To paraphrase the long-ago dictum of President John F. Kennedy, the Monthly is seeking, in essence, to ask not so much what colleges can to for themselves as what they can be doing for their country.

By that measure, only one Ivy cracked the top 10–Harvard. The University of California system dominated, with six of California state schools among the top 30 national universities. Texas A&M, which is ranked 63rd by U.S. News, shot into the top 20 in part because of how many of its students participate in ROTC. Meanwhile, Washington University in St. Louis plunged in these rankings to 112 from U.S. News’ 13, because only 6 percent of its student body qualifies for federal Pell grants, an indication that Washington’s students come almost entirely from upper- and middle-class backgrounds.

The U.S. News & World Report “relies on crude and easily manipulated measures of wealth, exclusivity, and prestige for its rankings,” Washington Monthly editor Paul Glastris wrote. The U.S. News’ rankings take into account freshmen retention rate, admissions’ selectivity, high school counselors’ opinions of the school, faculty salary, per-pupil spending and the rate of alumni giving, among other things.

While the editor suggests these new rankings are not as influenced by status and wealth, I wonder if the measures really get away from these entirely. It takes resources to enroll low-income students, provide resources ground-breaking research, and perhaps extra time for students to be able to be engaged in community and national service. On the other hand, colleges make decisions about how to spend their money and could choose to put their resources into these particular areas.

I’m sure there will be questions about methodology: how did they measure impactful research? How much should ROTC count for and how did they measure community engagement?

New rankings also give more schools an opportunity to claim that they are at the top. For example, Northwestern College in Iowa now trumpets on their main page that “Washington Monthly ranks NWC third in the nation.” Read past the headline and you find that it is third within baccalaureate colleges. On the other side, will schools like Washington University in St. Louis even acknowledge these new rankings since they don’t look so good?

Changes in the Census Bureau’s data collection between 2000 and 2010

Posted on by

While certain parts of the dicennial Census have remained the same, a sociologist outlines some data collection changes between 2000 and 2010:

In 2000 the short form included only eight questions for the householder and six for every person living in the household. These were the same basic questions that would be asked in 2010 about sex, age, relationship to the householder, race and Hispanic status.

About one in six households received the “long form,” which included some 53 questions. In addition to the basic questions in every census form, this survey also contained questions on a whole panoply of characteristics, including housing, moving in the last five years, employment, detailed income sources, military service, disability status, ancestry, place of birth and education.

Until 2010, the long form and short form were distributed in tandem, but then the government split them. Beginning in 2005, the Census Bureau began to collect data for the American Community Survey, which is very similar to the old long form. That survey gets responses from about 2 million households and residents of group quarters (prison, dormitory, institution) every year, and tracks the same sort of data that was produced by the long form sample. The survey takes place all year and interviews, where necessary, are conducted by permanent staff — not the temporary workers who interview for decennial census.

The Census Bureau releases three sets of data each year from the community survey: the one-year, the three-year and the five-year files. The one-year file is released for areas of at least 65,000 in population, the three-year file for areas of at least 20,000 and the five-year file for all of the areas for which the long form was released (block-groups, tracts and higher). The census releases data only for certain locations with larger populations at one and three-year intervals. This is because, since the ACS is a sample, its reliability depends on sample size.

So on Dec. 14, 2010, the Census Bureau released the first five-year file from the American Community Survey, including all the data that used to be released from the census long form. This totaled 32,000 columns with 675,000 rows of data. These data are comparable to what was compiled from the 2000 Census and allow one to answer many questions at the neighborhood level, where data are needed beyond the few questions in the 2010 census.

In summary, two things have changed:

1. The Census has been moving toward more frequent data collection for a while. Some of the stuff that used to be asked every 10 years is now being asked more frequently, providing more up-to-date information.

2. Beveridge goes on to explain a couple of issues with the American Community Survey data, one good and one bad. On the bad side: although it may be more frequent, the sample size is not as large as a decennial census, meaning that we can’t be as certain about the results. On the good side: the Census will no longer have to rely on large teams of temporary staff every ten years but will have trained professionals collecting data year after year. In the coming years, it will be interesting to see how researchers treat these new figures.

Overall, this is a reminder that official figures don’t simply happen. They are the product of particular methodologies and definitions and when these protocols change, so can the data.

Additionally, the data is easily accessible (here are the 2010 figures). I’m sure it will continue to get easier to access and analyze the data. This is good for democracy as it is relatively easy for citizens to see the current status and changes in their nation. I wish more people could or would use this data frequently to better understand their surroundings.

Sort this out: poll of 39 economists suggests “30% chance of recession”

Posted on by

Polling economists about whether the country is headed for a recession does not seem to be the best way to make predictions:

The 39 economists polled Aug. 3-11 put the chance of another downturn at 30% — twice as high as three months ago, according to their median estimates. That means another shock to the fragile economy — such as more stock market declines or a worsening of the European debt crisis — could push the nation over the edge.

Yet even if the USA avoids a recession, as economists still expect, they see economic growth muddling along at about 2.5% the next year, down from 3.1% in April’s survey. The economy must grow well above 3% to significantly cut unemployment…

The gloomier forecast is a stunning reversal. Just weeks ago, economists were calling for a strong rebound in the second half of the year, based on falling gasoline prices giving consumers more to spend on other things and car sales taking off as auto supply disruptions after Japan’s earthquake faded. In fact, July retail sales showed their best gain in four months.

But that was before European debt woes spread, the government cut its growth estimates for the first half of 2011 to less than 1%, and Standard & Poor’s lowered the USA’s credit rating after the showdown over the debt ceiling.

Here is what I find strange about this:

1. The headline meant to grab our attention focuses on the 30% statistic. Is this a good or bad figure? It is less than 50% (meaning there are less equal odds) but it is also double the prediction of predictions three months ago. Based on a 3 in 10 chance of a recession, how would the country and individual change their actions?

2. This comes from a poll of 39 economists. One, this isn’t that many. Two, how do we know that these economists know what they are talking about? How successful have their predictions been in the past? I see the advantages of “crowd-sourcing,” consulting a number of estimates to get an aggregate figure, but the sample could be larger and we don’t know whether these economists will be right. (Even if they are not right, perhaps it gives us some indication about what “leading economists” think and this could matter as well).

3. How much of this is based on real data versus perceptions of the economy? The article suggests this is a “stunning reversal” of earlier predictions and then cites some data that seems to be worse. These figures don’t determine everything. I wonder what it would take for economists to predict a recession – which numbers would have to be worse and how bad would they have to get?

4. Will anyone ever come back and look at whether these economists got it right?

In the end, I’m not sure this really tells us anything. I suspect it is these sorts of statistics and headlines that push people to throw up their hands altogether about statistics.

Leaving bookstores alone in the London riots

Posted on by

Some people have noticed that the rioters/looters in London have ignored the bookstores:

While the rioters in England this week have looted shops selling shoes, clothes, computers, and plasma televisions, they’ve curiously bypassed one particular piece of merchandise: books. The Economist observes that while rioters have a centuries-old history of book burning, “books are losing out to high-end jeans and Apple-made gadgets” in London, with the Waterstone’s bookstore chain emerging unscathed and the WH Smith chain reporting only one incident (some stores closed as a precaution). In explaining that the store would probably stay open during the unrest, one Waterstone’s employee even felt comfortable enough to issue a dare to the rioters: “If they steal some books, they might actually learn something.” The exception to the rule is the gay bookstore Gay’s the Word, which had its front window smashed and its shopfront splattered with eggs (notably, no goods were stolen). “Our impression is that there are certain people who have an issue with a visible gay business and are using the excuse of chaos to cause anti-gay damage,” an assistant manager told PinkPaper…

So where does that leave us on the question of why the rioters refrained from looting and burning bookstores? The most likely explanation appears to be that the rioters were more interested in high-end clothing and electronics than books, for economic and personal reasons. But a Guardian article yesterday suggests the rioters may have been more principled about what they stole and what they didn’t than one might think.

Interesting. The image many people might have is that the rioters act indiscriminately, breaking and smashing things at will out of anger. But these different possible explanations suggest rioters follow some sort of logic. Yes, their actions fall outside the normal bounds of civil behavior but they are acting upon not-too-unreasonable logic (go for the high-end electronic goods). This reminds me of the work of Sudhir Venkatesh who suggests that gangs follow logical paths even though their actions may seem chaotic or unclear.

Perhaps I only think this as a sociologist or as someone who teaches research methods but it seems like these ideas about bookstores could be tested. Researchers could take different groups of people, perhaps split by socioeconomic status, and then give them a variety of free items that they could take including electronics, clothing, and books. Then, researchers could see what people would take and then could question them about their choices afterward. Of course, some of the same information could be obtained by asking this is a question or series of questions on a large-scale survey. These sorts of options could help provide some insights into where books fall among other desirable consumer goods. If I had to hazard a guess, I imagine American teenagers or emerging adults (18-29 years old) would also put books toward the bottom of the list of things they would desire.

60% of British teenagers, 37% of adults “highly addicted” to their smartphones

Posted on by

A recent British study found that many teenagers are “highly addicted” to their smartphones:

Britons’ appetite for Facebook and social networks on the go is driving a huge demand for smartphones – with 60% of teenagers describing themselves as “highly addicted” to their device – according to new research by the media regulator, Ofcom…

The study, published on Thursday, also shows that smartphones have begun to intrude on our most private moments, with 47% of teenagers admitting to using their device in the toilet. Only 22% of adults confessed to the same habit. Unsurprisingly, mobile-addicted teens are more likely than adults to be distracted by their phones over dinner and in the cinema – and more would answer their phone if it woke them up…

Of the new generation of smartphone users, 60% of teenagers classed themselves as “highly addicted” to their device, compared to 37% of adults.

Ofcom surveyed 2,073 adults and 521 children and teenagers in March this year. The regulator defines teenagers as aged between 12 and 15, with adults 16-years-old and above.

Perhaps these results are not that surprising but it leads to several thoughts about addiction:

1. Since this is self-reported, couldn’t the percentage of teenagers and adults who are “highly addicted” actually be higher? If asked, how many people would admit to being “highly addicted” to things that they were actually addicted to?

2. That this many people were willing to say that they are “highly addicted” suggests that this addiction is probably considered to be normal behavior. If everyone or most people are actually addicted to using their smartphones, doesn’t this turn into a norm rather than an addiction in the eyes of the public? In twenty years, when these teenagers are the ones running these surveys, they may not use the same language or terms to describe phone/mobile device/computer use.

Forbes’ college rankings signals possible trend of looking at alumni earnings and status

Posted on by

The college rankings business is a lucrative one and there are a number of different players with a number of different measures. Forbes recently released its 2011 rankings and they have a particular angle that seems aimed at unseating the rankings of US News & World Report:

Our annual ranking of the 650 best undergraduate institutions focuses on the things that matter the most to students: quality of teaching, great career prospects, graduation rates and low levels of debt. Unlike other lists, we pointedly ignore ephemeral measures such as school “reputation” and ill-conceived metrics that reward wasteful spending. We try and evaluate the college purchase as a consumer would: Is it worth spending as much as a quarter of a million dollars for this degree? The rankings are prepared exclusively for Forbes by the Center for College Affordability and Productivity, a Washington, D.C. think tank founded by Ohio University economist Richard Vedder.

With phrases like “ephemeral measures” and “ill-conceived metrics,” Forbes claims to have a better methodology. This new approach helps fill a particular niche in the college rankings market: those looking for the “biggest bang for your educational buck.”

In their rankings, 30% of the final score is based on “Post-Graduate Success.” This is comprised of three values: “Listings of Alumni in Who’s Who in America” (10%), “Salary of Alumni from payscale.com” (15%), and “Alumni in Forbes/CCAP Corporate Officers List” (5%). These may be good measures (Forbes goes to some effort to defend them) but I think there is a larger issue at play here: are these good measures by which to evaluate a college degree and experience? Is a college degree simply about obtaining a certain income and status?

At this point, many rankings and assessment tools rely on the experiences of students while they are in school. But, with an increasing price for a college degree and a growing interest in showing that college students do learn important skills and content in college, I think we’ll see more measures of and a greater emphasis placed on post-graduation information. This push will probably come from both outsiders, Forbes, parents and students, the government, etc., and college insiders. This could be good and bad. On the good side, it could help schools tailor their offerings and training to what students need to succeed in the adult world. On the bad side, if value or bang-for-your-buck becomes the overriding concern, college and particular degrees simply become paths to higher or lower-income outcomes. This could particularly harm liberal arts schools or non-professional majors.

In the coming years, perhaps Forbes will steal some of the market away from US News with the financial angle. But this push is not without consequences for everyone involved.

(Here is another methodological concern: 17.5% of a school’s total score is based on ratings from RateMyProfessors.com. Forbes suggests it cannot be manipulated by schools and is uniform across schools but this is a pretty high percentage.)

(Related: a new report rates colleges by debt per degree. A quick explanation:

Its authors say they aim to give a more complete picture of higher education — rather than judging by graduation rates alone or by default rates alone — by dividing the total amount of money undergraduates borrow at a college by the number of degrees it awards.

We’ll see if this catches on.)

Pew again asks for one-word survey responses regarding budget negotiations

Posted on by

I highlighted this survey technique in April but here it is again: Pew asked Americans to provide a one-word response to Congress’ debt negotiations.

Asked for single-word characterizations of the budget negotiations, the top words in the poll — conducted in the days before an apparent deal was struck — were “ridiculous,” “disgusting” and “stupid.” Overall, nearly three-quarters of Americans offered a negative word; just 2 percent had anything nice to say.

“Ridiculous” was the most frequently mentioned word among Democrats, Republicans and independents alike. It was also No. 1 in an April poll about the just-averted government shutdown. In the new poll, the top 27 words are negative ones, with “frustrating,” “poor,” “terrible,” “disappointing,“ “childish,” “messy” and “joke” rounding out the top 10.

And then we are presented a word cloud.

On the whole, I think this technique can suggest that Americans have generally unfavorable responses. But the reliance on particular terms is better for headlines than it is for collecting data. What would happen if public responses were split more evenly: what words/responses would then be used to summarize the data? The Washington Post headline (and Pew Research as well) can now use forceful and emotional words like “ridiculous” and “disgusting” rather than the more accurate numerical figures than about “three-quarters of Americans offered a negative word.” Why not also include an ordinal question (strongly disapprove to strongly approve) about American’s general opinion of debt negotiations in order to corroborate this open ended question?

This is a possibly interesting technique in order to take advantage of open ended questions without allowing respondents to give possibly lengthy responses. Open ended questions can produce a lot of data: there were over 330 responses in this survey alone. I’ll be interested to see if other organizations adopt this approach.

Sociologists on the “chore wars”

Posted on by

Time magazine’s latest cover story on “chore wars” features two competing explanations from sociologists:

The assumption that working women had become the Clydesdales of contemporary marriage can be traced back to the publication of Arlie Russell Hochschild’s The Second Shift in 1989. In the 1970s, Hochschild was a sociologist with two young children who was trying to get tenure at Berkeley, where she saw her male colleagues unencumbered by demands at home and was inspired to write about the working women’s double day. “It came from my own anguish, my own conflict,” she says…

Hochschild came up with that number by averaging data collected in the 1960s, spotlighting what is now clearly the product of a culture in transition, a lag between women’s entry into the workforce and the great domestic shakeout in which working women cut back on housework, often by outsourcing, and men reduced office hours and chipped in more at home. Yet Hochschild’s interpretation of that statistical blip in the 1960s came to define the plight of women in the 1990s and 2000s. The Second Shift was a huge crossover hit and sparked a huge surge of academic writing on the inequalities of the household…

One American sociologist, Suzanne Bianchi, stood on the sidelines of the why-men-aren’t-doing-more debate for many years. From 1978 to 1994, she was a demographer and statistician at the U.S. Census Bureau working with large represtnative samples that shed light on long-term changes at thepopulation level. Bianchi was looking at almost everything but housework – education, earnings, changes in employment – so she became aware of the pitfalls of focusing only on the domestic sphere. “Maybe men really were all jerks and not doing their fair share, or maybe they were allocating their time to other things. By isolating housework from other kinds of work, you lost track of the fact that families need money as well as time,” she says. “I began to get interested in what we really know. We think men don’t do anything, but is that right? Are we systematically missing what they do do?”…

Bianchi and her colleagues analyzed time-diary data from 2003 to ’05 and found that among couples in which both partners work full time, men’s greater hours of paid work counterbalanced women’s greater hours of unpaid work. A second shift, where it still existed, was most evident in dual-earner couples with children under the age of 6, but it was a difference of five hours more of combined paid and unpaid work for women a week, not 15. “That didn’t mean that The Second Shift was completely wrong, just that it was misleading,” says Bianchi, who published her analysis in 2009. “Another thing that got missed was that women shed housework when they’re employed full time, but they hold on to a lot of child care, and that’s a big piece of why The Second Shift resonates so much.”

The article suggests that the gap between the work of men and women has closed and there needs to a more nuanced explanation about the subject. It seems like the larger conversation would also be enhanced with more data rather individuals relying on personal anecdotes. For the forthcoming edition of The Second Shift (January 2011), will Hochschild also include updated/new data?

Two further issues:

1. The argument in this story works because of a methodological concern: men’s paid work counts in their total. If we look at just the figures for work within the home, there is still a decent gap between men and women. On one hand, we could consider all kinds of work to be equal (and work at a paid job certainly has its own stress and advantages). On the other hand, if an earlier goal was for men and women to equally share work at home, it hasn’t happened.

2. Where do we go with this data? The article suggests the arguments of The Second Shift resonate with newer generations. Will this article convince anyone that men are doing more work (or more equal work) or will it simply reinforce existing divides?

God gets good job approval ratings in recent survey

Posted on by

I’m not sure why Public Policy Polling decided to recently include these questions in a mid-July survey (just to make comparisons with current politicians and Rupert Murdoch?) but here is how Americans rate God’s job performance:

While many polls have asked what Americans’ beliefs are about God, there has been little measurement of voters’ evaluation of its performance. It turns out, if God exists, voters would give God a strong 52-9 approval rating. This is hardly a surprise considering the vast majority of the country believes in an infallible deity, but some of the crosstabs are quite interesting.

There is a considerable age divide on God’s approval with those 18-29 approving 67-18 compared to a 40-6 approval rating among those over 65. What jumps out from this divide is not just that young voters are more likely to be critical of the job performance of the omnipotent figure, but that they are considerably more likely to voice their opinion. Only 15% of those 18-29 said they were unsure whether they approved of God, while 54% of those over 65 said they were unsure. This could indicate that the youth is much more comfortable answering silly questions about religion while the elderly feel a question on God’s approval is taking religion too lightly. There is also an ideological divide over God’s performance. Those who identify as very liberal approve of God 54-18, while those who identify as very conservative are almost uniform in their approval, 61-4.

God also performs well on some of the issues it could be responsible for if it exists. God scores its best rating on its handling of creating the universe. The big bang may be messy, but most voters must feel it gets the job done as they give God a 71-5 rating on the issue. As for the animal kingdom, if God exists it may have been off its game when it evolved up the giraffe’s laryngeal nerve, but perhaps the elegant Monarch butterfly makes up for it as voters give God a 56-11 rating on its handling of animals. As one would expect, God’s worst ratings are on its handling of natural disasters; however, Americans may feel the occasional earthquake or hurricane builds character as voters give God strong marks, rating it 50-13.

These figures seem pretty high but they are rather vague questions. If you are going to ask these questions, why not add a few more: do you feel God has treated you fairly lately? Do you think God has blessed the United States? Which political party do you think God would side with more? Do you feel that God approves of your job performance in life?

Maybe these figures do indicate some generational and political gaps. There are also some differences by race as African-Americans are more likely to approve of God’s performance. Or perhaps these figures don’t tell us much of anything. PPP could start asking this question more regularly and track the data over time.

It would be interesting to follow these questions by asking respondents why they gave the rating they did. Were respondents afraid to give God negative ratings?

Also, how come there is no indication of how many people didn’t answer this question since it starts with “If God exists”? Cross-tabs give us percentages but we don’t have Ns for the categories or cells.