Tag Archives: methodology

Determining the best colleges…using RateMyProfessor.com?

Posted on by

Forbes recent published another installment of their rankings of the best colleges in America. One of the question that arises with such a list is the methodology behind the rankings. To their credit, Forbes provides a lengthy explanation.

Even as the ranking is supposedly from the point of view of students, I initially had some questions about one of the major criteria which accounts for 17.5% of the score for a college: using student evaluations of professors at RateMyProfessor.com. At first, this sounded crazy to me – how representative is the data from RateMyProfessors.com and does it accurately reflect what is going on in the classroom?

Forbes sums up why they used this data:

In spite of some drawbacks of student evaluations of teaching, they apparently have value for the 86% of schools that have some sort of internal evaluation system. RMP ratings give similar results to these systems. Moreover, they are a measure of consumer preferences, which is what is critically important in rational consumer choice. When combined with the significant advantages of being uniform across different schools, not being subject to easy manipulation by schools, and being publicly available, RMP data is a preferred data source for information on student evaluations of teaching–it is the largest single uniform data set we know of student perceptions of the quality of their instruction.

To recap why these used data from RateMyProfessors.com:

1. RMP ratings are similar to evaluation scores gathered by colleges. There is some scholarly research to back this up.

2. RMP ratings are “a measure of consumer preference.” This is data generated voluntarily by students. If Forbes wants the students’ perspective, this website offers it. (Though it is still a question whether it is a representative measure – but point #1 may take care of that.)

3. RMP ratings are perhaps the only data source to answer the question of what students experience in the classroom. It may not be perfect data but it can be used as an approximation.

Overall, Forbes logic makes some sense: RateMyProfessor.com offers a unique dataset that when cleaned up (and they describe how they weighted and standardized the scores) offers some insights into the classroom experience.

However, I’m still leery of giving 17.5% of the total score over to RateMyProfessor.com evaluations. Perhaps the scholarly literature will continue to examine this website and determine the value of its ratings. And you can see that Forbes is tweaking their measurements: the 2009 methodology explanation has some differences and the RateMyProfessor.com score then counted for 25% of the total score (compared to 17.5% in the 2010 edition).

Using undergraduates in research experiments

Posted on by

It is common for research experiments to use undergraduates as subjects: they are a convenient and often willing sample pool for researchers. These studies then draw conclusions about human behavior based on undergraduate subjects.

In Newsweek, Sharon Begley writes about a new study that suggests American undergraduates are unlike many people in the world and therefore, it is difficult to make generalizations based on them.

Three psychology researchers have done a systematic search of experiments with subjects other than American undergrads, who made up two thirds of the subjects in all U.S. psych studies. From basics such as visual perception to behaviors and beliefs about fairness, cooperation, and the self, U.S. undergrads are totally unrepresentative, Joseph Henrich of the University of British Columbia and colleagues explain in a paper in Behavioral and Brain Sciences. They share responses with subjects from societies that are also Western, educated, industrialized, rich, and democratic (WEIRD), but not with humanity at large.

One way around such issues is to replicate studies with different people groups. The article describes some of these attempts, such as with the ultimatum game where two people have to negotiate a split of $10. When done with different people, the studies produce different results, suggesting that what we might think is “human nature” is heavily culturally dependent.

Another possible outcome of this study is that researchers may continue to use undergraduates but would have to scale back on their ability to generalize about humanity as a whole.

Finally, this study is a reminder that “typical” behavior in one culture is not guaranteed to be the same in another culture. What we may think of as givens can be quite different with people who do not share our cultural assumptions and practices.

Debate over food portions in Last Supper paintings

Posted on by

ARTnews reports on a debate concerning a study that was published earlier this year in the International Journal of Obesity. The study from Brian and Craig Wansink examined depictions of the food at the Last Supper in artwork dating back to the sixth century. Their conclusions: “the food portions became increasingly generous over time, with the main dish expanding by 69 percent, the bread portions by 23 percent, and the plates swelling in size by 66 percent.” The study hit the news wires in March; read reporting from the New York Times here.The implication in some of the news coverage was that food portions have increased over time, contributing to issues like obesity.

According to ARTnews, some art historians have taken issue with the study. Some of the issues listed in the article:

1. Is the Last Supper the best meal to examine?

2. Is the growing importance of still-life art over this time period more responsible for the growing size of plates?

3. Is there a growing amount of food because the cuisine of European cultures expanded over time?

4. Is this an appropriate methodology for measuring something like food portions?

An interesting study and an interesting debate over what it means.

Ongoing issue of measuring online audiences

Posted on by

If you were examining Hulu.com’s online audience figures from the last few months, you would find some fluctuation: 43.5 million viewers in May and then 24 million viewers in June. What happened? Did something radically change with the website? Are people abandoning the practice of watching television online?

No, the main change is that ComScore changed its methodology for measuring who used the website. According to the Los Angeles Times:

The three dominant measurement firms — ComScore, Nielsen and Quantcast — have been working since 2007 with an independent media auditing group to make improvements so the Web data they report don’t have a fun-house quality, in which the same site’s traffic can look emaciated or bulging, depending on the viewer’s angle.

These firms have used different measurements over time including panels of users (like Nielsen uses for television and radio) and embedded tags in videos and websites to track viewership. These numbers matter more than ever for advertisers as they will spend around $25 billion in online advertising in the United States in 2010.

As in many cases, knowing the means of measurement matters tremendously for interpreting statistics.

Defining the Washington elite

Posted on by

Politico is reporting today on an online poll where they compare opinions of the “Washington elite” vs. other Americans. The main news seems to be the divergent opinions between the two groups but the means of measurement is intriguing as well. To qualify as a Washington elite:

[R]espondents must live within the D.C. metro area, earn more than $75,000 per year, have at least a college degree and be involved in the political process or work on key political issues or policy decisions.

Another point of interest: only 227 Washington elite are in the poll. This is a fairly small group for a typical poll to use for analysis. The margin of error for the Washington elites is 6.53%.

Find the story and full polling results here.