Making the case for reputational rankings

A statistician argues that the National Research Council’s study of doctoral programs released earlier this week should have included reputational rankings:

Mr. Stigler says that it was a mistake for the NRC to so thoroughly abandon the reputational measures it used in its previous doctoral studies, in 1982 and 1995. Reputational surveys are widely criticized, he says, but they do provide a check on certain kinds of qualitative measures. When the new NRC counts faculty publication rates, it does not offer any information about whether scholars in the field believe those publications are any good. (That’s especially true in humanities fields, where the NRC report does not include citation counts.)

“Everybody involved in this was trying hard, and with good intentions and high integrity,” Mr. Stigler says. “But once they decided to rule out reputation, they cut off what I consider to be the most useful measure from all past surveys.”

In an e-mail message to The Chronicle this week, Mr. Ostriker declined to reply to Mr. Stigler’s specific statistical criticisms. But he pointed out that the National Academies explicitly instructed his committee not to use reputational measures.

I was curious about this when I looked at the list of sociology doctoral programs. Perhaps several of the schools that were lower than I expected, such as the University of California – Berkeley, were lower because of this.

Stigler defends reputational measures but I’ve seen others argue that they prohibit “true” rankings within fields because certain schools retain a reputation even without the necessary output (research, good grad students, etc.). This particular discussion is part of a larger one where it will need to be decided whether reputational rankings should be used or not.

NRC ratings of doctoral programs

The National Research Council has released its long-awaited report that measures and compares doctoral programs in a number of disciplines. Check out the interactive tool at the Chronicle of Higher Education that allows users to compare programs on a variety of the 21 criteria.

Some interesting features of the data:

1. The surveys were conducted in 2006-2007 so the information is somewhat dated. This would be particularly true in departments with productive new or departed faculty.

2. The NRC doesn’t assign ordinal ranks to schools but instead now gives ranges for each program. This seems like a sound decision that helps suggests what schools are like/near each other without having to introduce what may be artificial distinctions by saying one school is #4 while another is #6.

Just quickly looking through some of the S-rankings in sociology, some schools seem to be quite a bit off compared to other rankings over the years.