A statistician argues that the National Research Council’s study of doctoral programs released earlier this week should have included reputational rankings:

Mr. Stigler says that it was a mistake for the NRC to so thoroughly abandon the reputational measures it used in its previous doctoral studies, in 1982 and 1995. Reputational surveys are widely criticized, he says, but they do provide a check on certain kinds of qualitative measures. When the new NRC counts faculty publication rates, it does not offer any information about whether scholars in the field believe those publications are any good. (That’s especially true in humanities fields, where the NRC report does not include citation counts.)

“Everybody involved in this was trying hard, and with good intentions and high integrity,” Mr. Stigler says. “But once they decided to rule out reputation, they cut off what I consider to be the most useful measure from all past surveys.”

In an e-mail message to The Chronicle this week, Mr. Ostriker declined to reply to Mr. Stigler’s specific statistical criticisms. But he pointed out that the National Academies explicitly instructed his committee not to use reputational measures.

I was curious about this when I looked at the list of sociology doctoral programs. Perhaps several of the schools that were lower than I expected, such as the University of California – Berkeley, were lower because of this.

Stigler defends reputational measures but I’ve seen others argue that they prohibit “true” rankings within fields because certain schools retain a reputation even without the necessary output (research, good grad students, etc.). This particular discussion is part of a larger one where it will need to be decided whether reputational rankings should be used or not.