The value of using multiple coders

A well-known psychologist from Harvard is in trouble for allegedly reporting false data from laboratory studies. How the allegations surfaced is illustrative of why researchers should have more than just one person looking at data. As reported in the Chronicle of Higher Education, here is what happened after the psychologist and a graduate student coded an experiment involving rhesus monkeys:

According to the document that was provided to The Chronicle, the experiment in question was coded by Mr. Hauser and a research assistant in his laboratory. A second research assistant was asked by Mr. Hauser to analyze the results. When the second research assistant analyzed the first research assistant’s codes, he found that the monkeys didn’t seem to notice the change in pattern. In fact, they looked at the speaker more often when the pattern was the same. In other words, the experiment was a bust.

But Mr. Hauser’s coding showed something else entirely: He found that the monkeys did notice the change in pattern—and, according to his numbers, the results were statistically significant. If his coding was right, the experiment was a big success.

The second research assistant was bothered by the discrepancy. How could two researchers watching the same videotapes arrive at such different conclusions? He suggested to Mr. Hauser that a third researcher should code the results. In an e-mail message to Mr. Hauser, a copy of which was provided to The Chronicle, the research assistant who analyzed the numbers explained his concern. “I don’t feel comfortable analyzing results/publishing data with that kind of skew until we can verify that with a third coder,” he wrote.

A graduate student agreed with the research assistant and joined him in pressing Mr. Hauser to allow the results to be checked, the document given to The Chronicle indicates. But Mr. Hauser resisted, repeatedly arguing against having a third researcher code the videotapes and writing that they should simply go with the data as he had already coded it. After several back-and-forths, it became plain that the professor was annoyed.

These discrepancies in the data led to indications that something similar had happened in other experiments.

Having multiple coders is good for several reasons:

1. Helping to eliminate or catch problems such as these where someone might be tempted to falsify data.

2. To help interpret ambiguous situations.

3. To demonstrate to the broader research community that the results are more than just one person’s conclusions. (This should also be aided by the review process as other researchers look over the work.)